Routes defined in clish/webUI are missing-checkpoint-gaia

error
checkpoint
best-practices
gaia
Routes defined in clish/webUI are missing-checkpoint-gaia
0

#1

Routes defined in clish/webUI are missing-checkpoint-gaia

Vendor: checkpoint

OS: gaia

Description:
Sometimes the routes that are defined in the Check Point Web UI or through clish may not be fully applied to the operating system layer. If this happens, Indeni will alert.

Remediation Steps:
A workaround to get it to work can be to restart the routeD daemon by running “cpstop;cpstart” or restarting the device. However since this should not happen a case can also be opened with your technical support provider. In the case of devices in a cluster it is possible that the issue happens only for one of the nodes and a failover to the other node could lessen the impact of the issue.

How does this work?
Retrieve Linux kernel routes using the Linux “netstat” command, and then the Check Point configured routes from Gaia’s /config/active file. Then compare two route sets to make sure they are the same.

Why is this important?
If a static route is configured via Clish or WebUI, sometimes the system does not write the route into the Linux kernel routing table. To make sure all routes have been written, we compare the actual kernel routes with those configured in Check Point.

Without Indeni how would you find this?
An administrator could login and manually list routes from both commands, and then compare it. However, often there are a many routes configured; combine this with the difference in output format (for example subnet), and it can be a very cumbersome task.

chkp-gaia-os-routes-novsx

#! META
name: chkp-gaia-os-routes-novsx
description: Check if configured routes in GAIA configuration are the same as set in the kernel, as well as record static and connected routes.
type: monitoring
monitoring_interval: 10 minute
requires:
    vendor: "checkpoint"
    os.name: "gaia"
    or:
        -
            vsx: 
                neq: "true"
        -
            mds: "true"
    asg:
        neq: "true"

#! COMMENTS
routes-missing-kernel:
    why: |
        If a static route is configured via Clish or WebUI, sometimes the system does not write the route into the Linux kernel routing table. To make sure all routes have been written, we compare the actual kernel routes with those configured in Check Point.
    how: |
        Retrieve Linux kernel routes using the Linux "netstat" command, and then the Check Point configured routes from Gaia's /config/active file. Then compare two route sets to make sure they are the same.
    without-indeni: |
        An administrator could login and manually list routes from both commands, and then compare it. However, often there are a many routes configured; combine this with the difference in output format (for example subnet), and it can be a very cumbersome task.
    can-with-snmp: false
    can-with-syslog: false
    vendor-provided-management: |
        Listing routes from kernel is only available from the command line interface. Listing configured routes is also available from the WebUI.

static-routing-table:
    skip-documentation: true

connected-networks-table:
    skip-documentation: true

#! REMOTE::SSH
${nice-path} -n 15 grep "route" /config/active ; ${nice-path} -n 15 netstat -rn; ${nice-path} -n 15 ifconfig -a

#! PARSER::AWK

function decimalToBinary(N) {
	r = ""						# initialize result to empty (not 0)
	paddingData = ""
	while (N != 0) {			# as long as number still has a value
		r = ((N%2)?"1":"0") r	# prepend the modulos2 to the result
		N = int(N/2)			# shift right (integer division by 2)
	}

	# Need to pad with zeroes if less than 8 bits
	count = r

	# Count how many digits in count.
	count = gsub(/[0-1]/, "", count)

	if(count < 8) {
		# How many to pad?
		padding = 8 - count
		
		# Start padding
		for(i = 1; i <= padding; i++) {
			paddingData = paddingData 0
		}
		r = paddingData r 
	}

	# Return result
	return r
}

# Function to calculate 8 bits to decimal
function binaryToDecimal(binary) {

	# Reset variables.
	delete bitArr
	totalValue = ""

	# Bit values.
	bitValueArr[1] = 128
	bitValueArr[2] = 64
	bitValueArr[3] = 32
	bitValueArr[4] = 16
	bitValueArr[5] = 8
	bitValueArr[6] = 4
	bitValueArr[7] = 2
	bitValueArr[8] = 1

	# Record each binary value into an array.
	for(i = 1; i <= 8; i++) {
		bitArr[i] = substr(binary, i, 1)
	}

	# For each binary, calculate value.
	for(id in bitArr) {
		if (bitArr[id] == 1) {
			value = bitValueArr[id]
		}
		totalValue = totalValue + value

		value = 0
	}

	# Return result
	return totalValue
}

BEGIN {
	netMaskToCIDR["0.0.0.0"] = 0
	netMaskToCIDR["128.0.0.0"] = 1
	netMaskToCIDR["192.0.0.0"] = 2
	netMaskToCIDR["224.0.0.0"] = 3
	netMaskToCIDR["240.0.0.0"] = 4
	netMaskToCIDR["248.0.0.0"] = 5
	netMaskToCIDR["252.0.0.0"] = 6
	netMaskToCIDR["254.0.0.0"] = 7
	netMaskToCIDR["255.0.0.0"] = 8
	netMaskToCIDR["255.128.0.0"] = 9
	netMaskToCIDR["255.192.0.0"] = 10
	netMaskToCIDR["255.224.0.0"] = 11
	netMaskToCIDR["255.240.0.0"] = 12
	netMaskToCIDR["255.248.0.0"] = 13
	netMaskToCIDR["255.252.0.0"] = 14
	netMaskToCIDR["255.254.0.0"] = 15
	netMaskToCIDR["255.255.0.0"] = 16
	netMaskToCIDR["255.255.128.0"] = 17
	netMaskToCIDR["255.255.192.0"] = 18
	netMaskToCIDR["255.255.224.0"] = 19
	netMaskToCIDR["255.255.240.0"] = 20
	netMaskToCIDR["255.255.248.0"] = 21
	netMaskToCIDR["255.255.252.0"] = 22
	netMaskToCIDR["255.255.254.0"] = 23
	netMaskToCIDR["255.255.255.0"] = 24
	netMaskToCIDR["255.255.255.128"] = 25
	netMaskToCIDR["255.255.255.192"] = 26
	netMaskToCIDR["255.255.255.224"] = 27
	netMaskToCIDR["255.255.255.240"] = 28
	netMaskToCIDR["255.255.255.248"] = 29
	netMaskToCIDR["255.255.255.252"] = 30
	netMaskToCIDR["255.255.255.254"] = 31
	netMaskToCIDR["255.255.255.255"] = 32
}

# Store information about a normal route configured in Clish.
#routed:instance:default:static:network:10.10.100.0:masklen:24:gateway:address:10.11.2.50 t
#routed:instance:default:static:network:100.10.10.0:masklen:24:gateway t
/routed:instance:default:static:network:.+:masklen:[0-9]+:gateway/ {
	split($1, configSplitArr, ":")
	
	if (configSplitArr[11] ~ /(?:[0-9]{1,3}\.){3}[0-9]{1,3}/) {
		nextHop = configSplitArr[11]
	}

	if (nextHop) {
		destinationNetwork = configSplitArr[6]
		subnetPrefix = configSplitArr[8]
		
		clishRouteArr[destinationNetwork "/" subnetPrefix] = ""

		iStaticRoute++
		staticRoutes[iStaticRoute, "network"] = destinationNetwork
		staticRoutes[iStaticRoute, "mask"] = subnetPrefix
		staticRoutes[iStaticRoute, "next-hop"] = nextHop
	}
	nextHop = ""

	next
}

# Store information about a default gateway configured in Clish.
#routed:instance:default:static:default:gateway:address:10.10.6.1 t
/routed:instance:default:static:default:gateway:address/ {
	split($1, configSplitArr, ":")
	
	if (configSplitArr[8] ~ /(?:[0-9]{1,3}\.){3}[0-9]{1,3}/) {
		nextHop = configSplitArr[8]
	}
	
	if (nextHop) {
		clishRouteArr["0.0.0.0/0"] = ""

		iStaticRoute++
		staticRoutes[iStaticRoute, "network"] = "0.0.0.0"
		staticRoutes[iStaticRoute, "mask"] = "0"
		staticRoutes[iStaticRoute, "next-hop"] = nextHop
	}
	nextHop = ""

	next
}

# Store information about routes configured in the OS
#10.11.2.0       0.0.0.0         255.255.255.0   U         0 0          0 eth1
/^(?:[0-9]{1,3}\.){3}[0-9]{1,3}/ {
	destination = $1
	subnetMask = $3
	subnetPrefix = netMaskToCIDR[subnetMask]
	flags = $4
	gateway = $2

	if (destination == "0.0.0.0") {
		subnetPrefix = "0"
	}
	
	netstatRouteArr[destination "/" subnetPrefix] = ""

	next
}


#routed:instance:default:ospf2:area:0.0.0.1 t
/^routed:instance:default:ospf/ {

	# If OSPF is enabled, disable comparing static and OS routes, as routes are learned dynamically.
	compareRoutesDisable = 1

	next
}

#            inet addr:192.168.194.41  Bcast:192.168.194.255  Mask:255.255.255.0
/^\s+inet addr:/ {
	ipAddress = $2
	subnetMask = $4

	
	#addr:192.168.194.41
	gsub(/addr:/, "", ipAddress)

	#Mask:255.255.255.0
	gsub(/Mask:/, "", subnetMask)

	subnetPrefix = netMaskToCIDR[subnetMask]

	# Calculate the subnet address.
	# Get subnet address by filling the host portion of the address with zeroes.
	# Example: 172.16.35.123/20

	# Split in octets.
	#172.16.35.123
	split(ipAddress, ipAddressArr, ".")

	# Translating all octets to binary (no dots between octets)
	for(octet in ipAddressArr) {
		octetBinary = decimalToBinary(ipAddressArr[octet])

		ipBinary = ipBinary octetBinary
	}

	# 10101100000100000010001101111011

	# Remove everything except first 20 bits.
	#10101100000100000010
	keepBits = substr(ipBinary, 1, subnetPrefix)
	
	
	subnetIpBinary = keepBits

	#32-20=12
	addBinary = 32 - subnetPrefix

	# Fill out remaining bits as zeroes until reaching 32 bits.
	for (i = 1; i <= addBinary; i++) {
		subnetIpBinary = subnetIpBinary 0
	}

	#subnetIpBinary = 10101100000100000010000000000000

	# Split binary into octets
	octetArr[1] = substr(subnetIpBinary, 1, 8)
	octetArr[2] = substr(subnetIpBinary, 9, 8)
	octetArr[3] = substr(subnetIpBinary, 17, 8)
	octetArr[4] = substr(subnetIpBinary, 25, 8)

	# Convert från binary to decimal, and combine octets to form the subnet IP address.
	subnetIp = binaryToDecimal(octetArr[1]) "." binaryToDecimal(octetArr[2]) "." binaryToDecimal(octetArr[3]) "." binaryToDecimal(octetArr[4])

	# If the subnet mask is empty, like it is for localhost, ignore.
	if (subnetMask != "") {
		iDirectRoute++
		directRoutes[iDirectRoute, "network"] = subnetIp
		directRoutes[iDirectRoute, "mask"] = subnetPrefix
	}


	# Reset variables
	ipBinary = ""

	next
}

END {
	
	# Do not run if the host uses OSPF
	if (compareRoutesDisable != 1) {
		# For each route in clishRoute, make sure we have it in netstatRouteArr
		for (id in clishRouteArr) {
			if (! (id in netstatRouteArr)) {
				iClishRoutemiss++
				routeMissing[iClishRoutemiss, "missing-route"] = id
			}
		}

		metricTags["name"] = "novsx"
		writeComplexMetricObjectArray("routes-missing-kernel", metricTags, routeMissing)
	}

	writeComplexMetricObjectArrayWithLiveConfig("static-routing-table", null, staticRoutes, "Static routes")
	writeComplexMetricObjectArrayWithLiveConfig("connected-networks-table", null, directRoutes, "Directly Connected Networks")
}

chkp_firewall_routes_missing_novsx

package com.indeni.server.rules.library.templatebased.checkpoint

import com.indeni.ruleengine.expressions.conditions.{Equals, Not}
import com.indeni.ruleengine.expressions.data.SnapshotExpression
import com.indeni.server.rules.RuleContext
import com.indeni.server.common.data.conditions.{Equals => DataEquals}
import com.indeni.server.rules.library.RuleHelper
import com.indeni.server.rules.library.templates.MultiSnapshotValueCheckTemplateRule

case class chkp_firewall_routes_missing_novsx() extends MultiSnapshotValueCheckTemplateRule(
  ruleName = "chkp_firewall_routes_missing_novsx",
  ruleFriendlyName = "Check Point Firewalls (Non-VSX): Routes defined in clish/webUI are missing",
  ruleDescription = "Sometimes the routes that are defined in the Check Point Web UI or through clish may not be fully applied to the operating system layer. If this happens, Indeni will alert.",
  metricName = "routes-missing-kernel",
  applicableMetricTag = "name",
  alertItemsHeader = "Routes missing",
  alertDescription = "The configured routes have not been correctly applied to the Gaia OS. This means that some of the routes configured do not currently work.",
  baseRemediationText = "A workaround to get it to work can be to restart the routeD daemon by running \"cpstop;cpstart\" or restarting the device. However since this should not happen a case can also be opened with your technical support provider. In the case of devices in a cluster it is possible that the issue happens only for one of the nodes and a failover to the other node could lessen the impact of the issue.",
  metaCondition = !DataEquals("vsx", "true"),
  complexCondition = Not(Equals(RuleHelper.createEmptyComplexArrayConstantExpression(), SnapshotExpression("routes-missing-kernel").asMulti().mostRecent().value().noneable))
)()