RX packets overrun-checkpoint-ipso

error
health-checks
ipso
checkpoint
RX packets overrun-checkpoint-ipso
0

#1

RX packets overrun-checkpoint-ipso

Vendor: checkpoint

OS: ipso

Description:
Indeni tracks the number of packets that had issues and alerts if the ratio is too high.

Remediation Steps:
Packet overruns usually occur when there are too many packets being inserted into the port’s memory buffer, faster than the rate at which the kernel is able to process them.

How does this work?
How many packets received by the interface is retrieved by running “netstat -idb”.

Why is this important?
It is useful to know how many packets have been received by the interface.

Without Indeni how would you find this?
An administrator could login and manually check interface configuration, or use SNMP.

chkp-ipso-interfaces-novsx

#! META
name: chkp-ipso-interfaces-novsx
description: Get interface information
type: monitoring
monitoring_interval: 1 minute
requires:
    vendor: "checkpoint"
    and:
        -
            os.name: "ipso"
        -
            or:
                -
                    vsx: 
                        neq: "true"
                -
                    mds: "true"

#! COMMENTS
network-interface-state:
    why: |
        Interfaces in the "down" state could result in downtime or reduced redundancy.
    how: |
        The state of the interface is retrieved by running "ifconfig -a".
    without-indeni: |
        An administrator could login and manually check interface status, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-admin-state:
    why: |
        If the interface is disabled, then it is okay for it to be down. If the interface is enabled however, it should be up.
    how: |
        The state of the interface is retrieved by running "ifconfig -a".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-speed:
    why: |
        If the interface speed is set to a low value, this could mean auto-negotiation is not working correctly and the interface does not utilize the full bandwidth available.
    how: |
        The speed of the interface is retrieved by running "ifconfig -a".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-duplex:
    why: |
        If the interface has half-duplex setting, this will reduce throughput, and should be investigated.
    how: |
        The duplex of the interface is retrieved by running "ifconfig -a".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-ipv4-address:
    why: |
        To be able to search for IP addresses in indeni, this data needs to be stored.
    how: |
        The IP address of the interface is retrieved by running "ifconfig -a".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP, WebUI or SmartDashboard.

network-interface-ipv4-subnet:
    why: |
        To be able to search for IP addresses in indeni, this data needs to be stored.
    how: |
        The subnet of the interface is retrieved by running "ifconfig -a".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP, WebUI or SmartDashboard.

network-interface-type:
    why: |
        The type of interface can be useful for administrators.
    how: |
        The type of the interface is retrieved by running "ifconfig -a".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface or SNMP.

network-interface-mtu:
    why: |
        The MTU sometimes needs to be adjusted. Storing this gives an administrator an easy way to view the MTU from a large number of devices, as well as identifying incorrectly set MTU.
    how: |
        The MTU of the interface is retrieved by running "ifconfig -a".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-mac:
    why: |
        To be able to search for MAC addresses in indeni, this data needs to be stored.
    how: |
        The MAC address of the interface is retrieved by running "ifconfig -a".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-description:
    why: |
        The description is an important way to identify interfaces.
    how: |
        Retrive the information by parsing the IPSO database in /config/active.
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-tx-bytes:
    why: |
        It is useful to know how much data has been transmitted by the interface.
    how: |
        How many bytes sent by the interface is retrieved by running "netstat -idb".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-rx-bytes:
    why: |
        It is useful to know how much data has been received by the interface.
    how: |
        How many bytes received by the interface is retrieved by running "netstat -idb".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-tx-packets:
    why: |
        It is useful to know how many packets have been transmitted by the interface.
    how: |
        How many packets sent by the interface is retrieved by running "netstat -idb".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-rx-packets:
    why: |
        It is useful to know how many packets have been received by the interface.
    how: |
        How many packets received by the interface is retrieved by running "netstat -idb".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-tx-errors:
    why: |
        Transmit errors on an interface could indicate a problem.
    how: |
        The amount of transmit errors for the interface is retrieved by running "netstat -idb".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interface-rx-dropped:
    why: |
        Dropped packets on an interface could indicate a problem and potential traffic loss.
    how: |
        The amount of receive drops for the interface is retrieved by running "netstat -idb".
    without-indeni: |
        An administrator could login and manually check interface configuration, or use SNMP.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface, SNMP or WebUI.

network-interfaces:
    skip-documentation: true

network-interface-:
    skip-documentation: true

#! REMOTE::SSH
${nice-path} -n 15 ifconfig -a
${nice-path} -n 15 grep "interface" /config/active
${nice-path} -n 15 netstat -idb

#! PARSER::AWK

function ignoreInterface(_name) {
    if (_name ~ /^(loop|soverf|stof|tun).*/) {
        return 1
    } else {
        return 0
    }
}

############
# Script explanation: We should avoid running clish commands due to the excessive logs in /var/log/messages that creates. So ifconfig and parsing /config/active instead.
###########

BEGIN {
    bitMaskToSubnet["32"] = "255.255.255.255"
    bitMaskToSubnet["31"] = "255.255.255.254"
    bitMaskToSubnet["30"] = "255.255.255.252"
    bitMaskToSubnet["29"] = "255.255.255.248"
    bitMaskToSubnet["28"] = "255.255.255.240"
    bitMaskToSubnet["27"] = "255.255.255.224"
    bitMaskToSubnet["26"] = "255.255.255.192"
    bitMaskToSubnet["25"] = "255.255.255.128"
    bitMaskToSubnet["24"] = "255.255.255.0"
    bitMaskToSubnet["23"] = "255.255.254.0"
    bitMaskToSubnet["22"] = "255.255.252.0"
    bitMaskToSubnet["21"] = "255.255.248.0"
    bitMaskToSubnet["20"] = "255.255.240.0"
    bitMaskToSubnet["19"] = "255.255.224.0"
    bitMaskToSubnet["18"] = "255.255.192.0"
    bitMaskToSubnet["17"] = "255.255.128.0"
    bitMaskToSubnet["16"] = "255.255.0.0"
    bitMaskToSubnet["15"] = "255.254.0.0"
    bitMaskToSubnet["14"] = "255.252.0.0"
    bitMaskToSubnet["13"] = "255.248.0.0"
    bitMaskToSubnet["12"] = "255.240.0.0"
    bitMaskToSubnet["11"] = "255.224.0.0"
    bitMaskToSubnet["10"] = "255.192.0.0"
    bitMaskToSubnet["9"] = "255.128.0.0"
    bitMaskToSubnet["8"] = "255.0.0.0"
    bitMaskToSubnet["7"] = "254.0.0.0"
    bitMaskToSubnet["6"] = "252.0.0.0"
    bitMaskToSubnet["5"] = "248.0.0.0"
    bitMaskToSubnet["4"] = "240.0.0.0"
    bitMaskToSubnet["3"] = "224.0.0.0"
    bitMaskToSubnet["2"] = "192.0.0.0"
    bitMaskToSubnet["1"] = "128.0.0.0"
    bitMaskToSubnet["0"] = "0.0.0.0"
}

#loop0c0:  lname loop0c0 flags=57<UP,PHYS_AVAIL,LINK_AVAIL,LOOPBACK,MULTICAST>
#eth1c0:  lname eth1c0 flags=e0<BROADCAST,MULTICAST,AUTOLINK>
/^.+:\s+/ {
    interfaceName = $1
    #eth1c0:
    sub(/:/, "", interfaceName)

    if (!ignoreInterface(interfaceName)) {
        statTags["name"] = interfaceName

        type = interfaceName
        #eth1c0
        sub(/[0-9].*/, "", type)
        writeComplexMetricString("network-interface-type", statTags, type)
        interfaces[interfaceName, "driver"] = type
    } else {
        interfaceName = ""
    }
}

#eth4c0:  lname eth4c0 flags=e4<UP,BROADCAST,MULTICAST,AUTOLINK>
#eth2c0:  lname eth2c0 flags=e7<UP,PHYS_AVAIL,LINK_AVAIL,BROADCAST,MULTICAST,AUTOLINK>
/lname.*flags=/ {
    if (interfaceName != "") {
        if ($0 ~ /PHYS_AVAIL/) {
            linkstate = 1
        } else {
            linkstate = 0
        }
        writeDoubleMetricWithLiveConfig("network-interface-state", statTags, "gauge", "60", linkstate, "Network Interfaces - Up/Down", "state", "name")

        if ($0 ~ /[^A-Z]UP[^A-Z]/) {
            adminState = 1
        } else {
            adminState = 0
        }
        writeDoubleMetricWithLiveConfig("network-interface-admin-state", statTags, "gauge", "60", adminState, "Network Interfaces - Enabled/Disabed", "state", "name")
    }
}

#         ether 00:a0:8e:b2:22:37 speed 10M half duplex
/[0-9a-fA-F]{2}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}/ {
    if (interfaceName != "") {
        match($0, /[0-9a-fA-F]{2}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}/)

        macaddress = substr($0, RSTART, RLENGTH)
        writeComplexMetricString("network-interface-mac", statTags, macaddress)
    }
}

#         inet mtu 1500 172.16.20.30/24 broadcast 172.16.20.255
/ inet / {
    if (interfaceName != "") {
        ipAndBitMask = $4
        split(ipAndBitMask, ipArray, "/")
        ip = ipArray[1]
        bitMask = ipArray[2]
        subnet = bitMaskToSubnet[bitMask]

        ipTags["name"] = interfaceName
        ipTags["im.identity-tags"] = "name"
        ipTags["im.dstype.displaytype"] = "string"

        writeComplexMetricStringWithLiveConfig("network-interface-ipv4-address", ipTags, ip, "Network Interfaces - IPv4 Address")
        writeComplexMetricStringWithLiveConfig("network-interface-ipv4-subnet", ipTags, subnet, "Network Interfaces - IPv4 Netmask")
    }
}

#        inet6 mtu 63000 ::1 --> ::1
/^\s.+mtu\s/ {
    if (interfaceName != "") {
        mtu = $0
        #        inet6 mtu 63000 ::1 --> ::1
        sub(/.* mtu /, "", mtu)
        #mtu 63000 ::1 --> ::1
        sub(/ .*/, "", mtu)

        interfaces[interfaceName, "mtu"] = mtu
        writeComplexMetricString("network-interface-mtu", statTags, mtu)
    }
}

#         ether 00:a0:8e:b2:22:37 speed 10M half duplex
/ speed .* duplex/ {
    if (interfaceName != "") {
        speed = $0
        #         ether 00:a0:8e:b2:22:37 speed 10M half duplex
        sub(/.* speed /, "", speed)
        #10M half duplex
        sub(/ .*/, "", speed)

        # speed metric need special tags for alert purpose
        speed_tags["name"] = interfaceName
        speed_tags["alert-item-port-speed"] = interfaceName "-" speed
        writeComplexMetricString("network-interface-speed", speed_tags, speed)

        duplex = $0
        if ($0 ~ /full duplex/) {
            writeComplexMetricString("network-interface-duplex", statTags, "full")
        } else if ($0 ~ /half duplex/) {
            writeComplexMetricString("network-interface-duplex", statTags, "half")
        }
    }
}

#interface:eth2c0:comments my\ comments
/interface:[a-z0-9]+:comments/ {
    if (interfaceName != "") {
        # Get interface name
        split($0, lineParts, ":")
        statTags["name"] = lineParts[2]

        # If the comment is in two words or more, a backslash is inserted. We need to remove it
        commentLine = $0
        gsub(/interface:[a-z0-9]+:comments/, "", commentLine)
        gsub(/(\\)/, "", commentLine)
        writeComplexMetricString("network-interface-description", statTags, trim(commentLine))
    }
}

#Name         Mtu   Network     Address             Ipkts Ierrs     Ibytes    Opkts Oerrs     Obytes  Coll Drop
#eth2         16018 <Link>      0:a0:8e:b2:22:35    39706   440    5481041    28917     0   10158270  1898   0
/^Name\s+Mtu\s+Network\s+/ {
    if (interfaceName != "") {
        inNetstat = "true"
        getColumns(trim($0), "[ \t]+", columns)
    }
}

#eth2         16018 <Link>      0:a0:8e:b2:22:35    39706   440    5481041    28917     0   10158270  1898   0
/[0-9]/ {
    if (inNetstat == "true") {
        name = getColData(trim($0), columns, "Name")
        statTags["name"] = name

        if (!ignoreInterface(name)) {
            writeDoubleMetricWithLiveConfig("network-interface-rx-packets", statTags, "counter", "60", getColData(trim($0), columns, "Ipkts"), "Network Interfaces - RX Packets", "number", "name")

            writeDoubleMetricWithLiveConfig("network-interface-rx-errors", statTags, "counter", "60", getColData(trim($0), columns, "Ierrs"), "Network Interfaces - RX Errors", "number", "name")

            writeDoubleMetricWithLiveConfig("network-interface-rx-bytes", statTags, "counter", "60", getColData(trim($0), columns, "Ibytes"), "Network Interfaces - RX Bytes", "bytes", "name")

            writeDoubleMetricWithLiveConfig("network-interface-rx-dropped", statTags, "counter", "60", getColData(trim($0), columns, "Drop"), "Network Interfaces - RX Dropped", "number", "name")

            writeDoubleMetricWithLiveConfig("network-interface-tx-packets", statTags, "counter", "60", getColData(trim($0), columns, "Opkts"), "Network Interfaces - TX Packets", "number", "name")

            writeDoubleMetricWithLiveConfig("network-interface-tx-errors", statTags, "counter", "60", getColData(trim($0), columns, "Oerrs"), "Network Interfaces - TX Errors", "number", "name")

            writeDoubleMetricWithLiveConfig("network-interface-tx-bytes", statTags, "counter", "60", getColData(trim($0), columns, "Obytes"), "Network Interfaces - TX Bytes", "bytes", "name")

            writeDoubleMetricWithLiveConfig("network-interface-tx-collisions", statTags, "counter", "60", getColData(trim($0), columns, "Coll"), "Network Interfaces - TX Collisions", "number", "name")
        }
    }
}


END {
    writeComplexMetricObjectArray("network-interfaces", null, interfaces)
}

cross_vendor_rx_overrun

package com.indeni.server.rules.library.templatebased.crossvendor

import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library.ConditionalRemediationSteps
import com.indeni.server.rules.library.templates.NearingCapacityWithItemsTemplateRule

/**
  *
  */
case class CrossVendorRxOverrun() extends NearingCapacityWithItemsTemplateRule(
  ruleName = "cross_vendor_rx_overrun",
  ruleFriendlyName = "All Devices: RX packets overrun",
  ruleDescription = "Indeni tracks the number of packets that had issues and alerts if the ratio is too high.",
  usageMetricName = "network-interface-rx-overruns",
  limitMetricName = "network-interface-rx-packets",
  applicableMetricTag = "name",
  threshold = 0.5,
  minimumValueToAlert = 100.0, // We don't want to alert if the number of error packets is really low
  alertDescription = "Some network interfaces and ports are experiencing a high overrun rate. Review the ports below.",
  alertItemDescriptionFormat = "%.0f packets overrun out of a total of %.0f received.",
  baseRemediationText = "Packet overruns usually occur when there are too many packets being inserted into the port's memory buffer, faster than the rate at which the kernel is able to process them.",
  alertItemsHeader = "Affected Ports")(
  ConditionalRemediationSteps.OS_NXOS ->
    """|
       |In a small number of cases, the overrun counter may be incremented because of a software defect. However, in the majority of cases, it indicates that the receiving capability of the interface was exceeded. Nothing can be done on the device that reports overruns. If possible, the rate that frames are coming should be controlled at the remote end of the connection.
       |1. Run the "show interface" command to review the interface overrun & underun counters and the bitrate. Consider to configure the "load-interval 30" interface sub command to improve the accuracy of the interface measurements.
       |2. If the number of overruns is high, the hardware should be upgraded.
       |
       |In case of high bandwidth utilization:
       |1. Run the "show interface" command to review the interface counters and the bitrate. Consider to configure the "load-interval 30" interface sub command to improve the accuracy of the interface measurements.
       |2. If the interface bitrate is too high and close to the upper bandwidth limit consider to use multiple links with the port-channel technology or upgrade the bandwidth of the link.
       |3. Consider to implement QoS in case of high bandwidth utilization.""".stripMargin,
  ConditionalRemediationSteps.VENDOR_JUNIPER ->
    """|The issue can not be resolved on the device that reports overruns. The rate that frames are coming should be controlled at the remote end of the connection.
       |1. Run the “show interface extensive” command to review the interface overrun counters. 

       |2. If the number of overruns is high, consider upgrading the hardware.
       |3. Review the following article on Juniper tech support site: <a target="_blank" href="https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/show-interfaces-security.html#jd0e1772">Operational Commands</a>""".stripMargin,
  ConditionalRemediationSteps.VENDOR_FORTINET ->
    """
       |1. Run "diag hardware deviceinfo nic <interface>" command to display a list of hardware related error names and values. Review  the next link for more details: http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-toubleshooting-54/troubleshooting_tools.htm
       |2. Run command "fnsysctl cat /proc/net/dev" to get a summary of the interface statistics.
       |3. Check for speed and duplex mismatch in the interface settings on both sides of a cable, and check for a damaged cable. Review the next link for more info: http://kb.fortinet.com/kb/documentLink.do?externalID=10653""".stripMargin
)