Routes defined in clish/webUI are missing-checkpoint-gaia

Routes defined in clish/webUI are missing-checkpoint-gaia
0

Routes defined in clish/webUI are missing-checkpoint-gaia

Vendor: checkpoint

OS: gaia

Description:
Sometimes the routes that are defined in the Check Point Web UI or through clish may not be fully applied to the operating system layer. If this happens, Indeni will alert.

Remediation Steps:
A workaround to get it to work can be to restart the routeD daemon by running “cpstop;cpstart” or restarting the device. However since this should not happen a case can also be opened with your technical support provider. In the case of devices in a cluster it is possible that the issue happens only for one of the nodes and a failover to the other node could lessen the impact of the issue.

How does this work?
Retrieve Linux kernel routes using the Linux “netstat” command, and then the Check Point configured routes from Gaia’s /config/active file. Then compare two route sets to make sure they are the same.

Why is this important?
If a static route is configured via Clish or WebUI, sometimes the system does not write the route into the Linux kernel routing table. To make sure all routes have been written, we compare the actual kernel routes with those configured in Check Point.

Without Indeni how would you find this?
An administrator could login and manually list routes from both commands, and then compare it. However, often there are a many routes configured; combine this with the difference in output format (for example subnet), and it can be a very cumbersome task.

chkp-gaia-os-routes-novsx

#! META
name: chkp-gaia-os-routes-novsx
description: Check if configured routes in GAIA configuration are the same as set in the kernel, as well as record static and connected routes.
type: monitoring
monitoring_interval: 10 minute
requires:
    vendor: "checkpoint"
    os.name: "gaia"
    or:
        -
            vsx:
                neq: "true"
        -
            mds: "true"
    asg:
        neq: "true"

#! COMMENTS
routes-missing-kernel:
    why: |
        If a static route is configured via Clish or WebUI, there is no guarantee that those routes will be successfully propagated the Linux kernel routing table. To ensure that all routes are applied correctly it is recommended to compare the actual routes with the configured ones.
    how: |
        Actual routes are retrieved using the built-in "netstat" command, and the configured routes from the gaia configuration database /config/active. The two are then compared to make sure they are the same.
    without-indeni: |
        An administrator could login and manually list routes from both commands, and then compare the output. However, often there are a lot of routes configured; combine this with the difference in output (for example subnet), and comparison can be a cumbersome task.
    can-with-snmp: false
    can-with-syslog: false
    vendor-provided-management: |
        Listing routes from kernel is only available from the command line interface. Listing configured routes is also available from the WebUI.

static-routing-table:
    skip-documentation: true

connected-networks-table:
    skip-documentation: true

#! REMOTE::SSH
${nice-path} -n 15 grep "route" /config/active ; ${nice-path} -n 15 netstat -rn; ${nice-path} -n 15 ifconfig -a

#! PARSER::AWK

function decimalToBinary(N) {
    r = ""                      # initialize result to empty (not 0)
    padding_data = ""
    while (N != 0) {            # as long as number still has a value
        r = ((N%2)?"1":"0") r   # prepend the modulos2 to the result
        N = int(N/2)            # shift right (integer division by 2)
    }

    # Need to pad with zeroes if less than 8 bits
    count = r

    # Count how many digits in count.
    count = gsub(/[0-1]/, "", count)

    if (count < 8) {
        # How many to pad?
        padding = 8 - count

        # Start padding
        for(i = 1; i <= padding; i++) {
            padding_data = padding_data 0
        }
        r = padding_data r
    }

    # Return result
    return r
}

# Function to calculate 8 bits to decimal
function binaryToDecimal(binary) {

    # Reset variables.
    delete bit_arr
    total_value = ""

    # Bit values.
    bit_value_arr[1] = 128
    bit_value_arr[2] = 64
    bit_value_arr[3] = 32
    bit_value_arr[4] = 16
    bit_value_arr[5] = 8
    bit_value_arr[6] = 4
    bit_value_arr[7] = 2
    bit_value_arr[8] = 1

    # Record each binary value into an array.
    for(i = 1; i <= 8; i++) {
        bit_arr[i] = substr(binary, i, 1)
    }

    # For each binary, calculate value.
    for(id in bit_arr) {
        if (bit_arr[id] == 1) {
            value = bit_value_arr[id]
        }
        total_value = total_value + value

        value = 0
    }

    return total_value
}

BEGIN {
    net_mask_to_CIDR["0.0.0.0"] = 0
    net_mask_to_CIDR["128.0.0.0"] = 1
    net_mask_to_CIDR["192.0.0.0"] = 2
    net_mask_to_CIDR["224.0.0.0"] = 3
    net_mask_to_CIDR["240.0.0.0"] = 4
    net_mask_to_CIDR["248.0.0.0"] = 5
    net_mask_to_CIDR["252.0.0.0"] = 6
    net_mask_to_CIDR["254.0.0.0"] = 7
    net_mask_to_CIDR["255.0.0.0"] = 8
    net_mask_to_CIDR["255.128.0.0"] = 9
    net_mask_to_CIDR["255.192.0.0"] = 10
    net_mask_to_CIDR["255.224.0.0"] = 11
    net_mask_to_CIDR["255.240.0.0"] = 12
    net_mask_to_CIDR["255.248.0.0"] = 13
    net_mask_to_CIDR["255.252.0.0"] = 14
    net_mask_to_CIDR["255.254.0.0"] = 15
    net_mask_to_CIDR["255.255.0.0"] = 16
    net_mask_to_CIDR["255.255.128.0"] = 17
    net_mask_to_CIDR["255.255.192.0"] = 18
    net_mask_to_CIDR["255.255.224.0"] = 19
    net_mask_to_CIDR["255.255.240.0"] = 20
    net_mask_to_CIDR["255.255.248.0"] = 21
    net_mask_to_CIDR["255.255.252.0"] = 22
    net_mask_to_CIDR["255.255.254.0"] = 23
    net_mask_to_CIDR["255.255.255.0"] = 24
    net_mask_to_CIDR["255.255.255.128"] = 25
    net_mask_to_CIDR["255.255.255.192"] = 26
    net_mask_to_CIDR["255.255.255.224"] = 27
    net_mask_to_CIDR["255.255.255.240"] = 28
    net_mask_to_CIDR["255.255.255.248"] = 29
    net_mask_to_CIDR["255.255.255.252"] = 30
    net_mask_to_CIDR["255.255.255.254"] = 31
    net_mask_to_CIDR["255.255.255.255"] = 32
}

# Skip comments. These shouldn't cause any problems. This is just defensive programming.
#routed:instance:default:static:network:10.127.0.0:masklen:16:comment xxxxx
/routed:instance:default:static:network:.*:comment / {
     next
}

# Store information about a normal route configured in Clish.
#routed:instance:default:static:network:10.10.100.0:masklen:24:gateway:address:10.11.2.50 t
#routed:instance:default:static:network:100.10.10.0:masklen:24:gateway t
/routed:instance:default:static:network:.+:masklen:[0-9]+:gateway/ {
    split($1, split_line, ":")

    if (split_line[11] ~ /[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/) {
        gateway = split_line[11]
        destination_network = split_line[6]
        cidr = split_line[8]

        static_route = destination_network "/" cidr ":" gateway

        # Checkpoint's /config/active file can contain many 'route patterns', and it's hard to predict all of them,
        # so we protect here against any possible 'duplicate routes' we might read from that file. See IKP-2117.
        if (! (static_route in dedup_routes) ) {
            static_routes_for_compare[static_route] = ""

            static_count++
            static_routes[static_count, "network"] = destination_network
            static_routes[static_count, "mask"] = cidr
            static_routes[static_count, "next-hop"] = gateway
            dedup_routes[static_route] = ""
        }
    }

    next
}

# Store information about a default gateway configured in Clish.
#routed:instance:default:static:default:gateway:address:10.10.6.1 t
/routed:instance:default:static:default:gateway:address/ {
    split($1, split_line, ":")

    if (split_line[8] ~ /[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/) {
        gateway = split_line[8]

        default_route = "0.0.0.0/0:" gateway

        # Checkpoint's /config/active file can contain many 'route patterns', and it's hard to predict all of them,
        # so we protect here against any possible 'duplicate routes' we might read from that file. See IKP-2117.
        if (! (default_route in dedup_routes) ) {
            static_routes_for_compare[default_route] = ""

            static_count++
            static_routes[static_count, "network"] = "0.0.0.0"
            static_routes[static_count, "mask"] = "0"
            static_routes[static_count, "next-hop"] = gateway
            dedup_routes[default_route] = ""
         }
    }

    next
}

# Store information about routes configured in the OS
#10.11.2.0       0.0.0.0         255.255.255.0   U         0 0          0 eth1
/^(?:[0-9]{1,3}\.){3}[0-9]{1,3}/ {
    destination = $1
    subnet_mask = $3
    cidr = net_mask_to_CIDR[subnet_mask]
    flags = $4
    gateway = $2

    if (destination == "0.0.0.0") {
        cidr = "0"
    }

    netstat_routes_for_compare[destination "/" cidr ":" gateway] = ""

    next
}


#routed:instance:default:ospf2:area:0.0.0.1 t
/^routed:instance:default:ospf/ {

    # If OSPF is enabled, disable comparing static and OS routes, as routes are learned dynamically.
    disable_compare_routes = 1

    next
}

#            inet addr:192.168.194.41  Bcast:192.168.194.255  Mask:255.255.255.0
/^\s+inet addr:/ {
    ip_address = $2
    subnet_mask = $4


    #addr:192.168.194.41
    sub(/addr:/, "", ip_address)

    #Mask:255.255.255.0
    sub(/Mask:/, "", subnet_mask)

    cidr = net_mask_to_CIDR[subnet_mask]

    # Calculate the subnet address.
    # Get subnet address by filling the host portion of the address with zeroes.
    # Example: 172.16.35.123/20

    # Split in octets.
    #172.16.35.123
    split(ip_address, ip_addresses, ".")

    # Translating all octets to binary (no dots between octets)
    for(octet in ip_addresses) {
        octet_binary = decimalToBinary(ip_addresses[octet])

        ip_binary = ip_binary octet_binary
    }

    # 10101100000100000010001101111011

    # Remove everything except first 20 bits.
    #10101100000100000010
    subnet_ip_binary = substr(ip_binary, 1, cidr)

    #32-20=12
    add_binary = 32 - cidr

    # Fill out remaining bits as zeroes until reaching 32 bits.
    for (i = 1; i <= add_binary; i++) {
        subnet_ip_binary = subnet_ip_binary 0
    }

    #subnet_ip_binary = 10101100000100000010000000000000

    # Split binary into octets
    octets[1] = substr(subnet_ip_binary, 1, 8)
    octets[2] = substr(subnet_ip_binary, 9, 8)
    octets[3] = substr(subnet_ip_binary, 17, 8)
    octets[4] = substr(subnet_ip_binary, 25, 8)

    # Convert från binary to decimal, and combine octets to form the subnet IP address.
    subnet_ip = binaryToDecimal(octets[1]) "." binaryToDecimal(octets[2]) "." binaryToDecimal(octets[3]) "." binaryToDecimal(octets[4])

    # If the subnet mask is empty, like it is for localhost, ignore.
    if (subnet_mask != "" && cidr != 32) {
        direct_count++
        direct_routes[direct_count, "network"] = subnet_ip
        direct_routes[direct_count, "mask"] = cidr
    }


    # Reset variables
    ip_binary = ""

    next
}

END {

    # This is how to initialize an empty array in AWK. Create this here so that, if there are no missing routes for
    # a given VS, we write an empty array as the metric. See also below.
    split("", missing_routes)

    # Do not run if the host uses OSPF
    if (disable_compare_routes != 1) {
        # For each route in clishRoute, make sure we have it in netstat_routes_for_compare
        for (id in static_routes_for_compare) {
            if (! (id in netstat_routes_for_compare)) {
                route_misses++
                missing_routes[route_misses, "missing-route"] = id
            }
        }

        metric_tags["name"] = "novsx"
    }

    # Always write the metric, even if it's empty, so that it will always eventually resolve.
    writeComplexMetricObjectArray("routes-missing-kernel", metric_tags, missing_routes)

    writeComplexMetricObjectArrayWithLiveConfig("static-routing-table", null, static_routes, "Static routes")
    writeComplexMetricObjectArrayWithLiveConfig("connected-networks-table", null, direct_routes, "Directly Connected Networks")
}

chkp_firewall_routes_missing_vsx

package com.indeni.server.rules.library.templatebased.checkpoint

import com.indeni.ruleengine.expressions.conditions.{Equals, Not}
import com.indeni.ruleengine.expressions.data.SnapshotExpression
import com.indeni.server.rules.{RuleCategory, RuleContext}
import com.indeni.server.common.data.conditions.{Equals => DataEquals}
import com.indeni.server.rules.library.RuleHelper
import com.indeni.server.rules.library.templates.MultiSnapshotValueCheckTemplateRule

case class chkp_firewall_routes_missing_vsx() extends MultiSnapshotValueCheckTemplateRule(
  ruleName = "chkp_firewall_routes_missing_vsx",
  ruleFriendlyName = "Check Point Firewalls (VSX): Routes defined in clish/webUI are missing",
  ruleDescription = "Sometimes the routes that are defined in the Check Point Web UI or through clish may not be fully applied to the operating system layer. If this happens, Indeni will alert.",
  metricName = "routes-missing-kernel",
  applicableMetricTag = "vs.name",
  alertItemsHeader = "Routes missing",
  alertDescription = "The configured routes have not been correctly applied to the Gaia OS. This means that some of the routes configured do not currently work.",
  baseRemediationText = "A workaround to get it to work can be to restart the routeD daemon by running \"cpstop;cpstart\" or restarting the device. However since this should not happen a case can also be opened with your technical support provider. In the case of devices in a cluster it is possible that the issue happens only for one of the nodes and a failover to the other node could lessen the impact of the issue.",
  complexCondition = Not(Equals(RuleHelper.createEmptyComplexArrayConstantExpression(), SnapshotExpression("routes-missing-kernel").asMulti().mostRecent().value().noneable)),
  ruleCategories = Set(RuleCategory.CustomerBestPractices)
)()