RX packets overrun-paloaltonetworks-panos

error
health-checks
panos
paloaltonetworks
RX packets overrun-paloaltonetworks-panos
0

#1

RX packets overrun-paloaltonetworks-panos

Vendor: paloaltonetworks

OS: panos

Description:
Indeni tracks the number of packets that had issues and alerts if the ratio is too high.

Remediation Steps:
Packet overruns usually occur when there are too many packets being inserted into the port’s memory buffer, faster than the rate at which the kernel is able to process them.

How does this work?
This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for the number of packets received through the interface.

Why is this important?
Tracking the number of packets flowing through each network interface is important to identify potential issues, spikes in traffic, etc.

Without Indeni how would you find this?
The traffic statistics of network interfaces can be manually reviewed through the CLI.

panos-show-interface

#! META
name: panos-show-interface
description: Fetch interface information and stats
type: monitoring
monitoring_interval: 5 minute
requires:
    vendor: paloaltonetworks
    os.name: panos
    product: firewall

#! COMMENTS
network-interface-admin-state:
    why: |
        If a network interface is set to be up (what's known as "admin up") but is actually down (a cable is not connected, the device on the other side is down, etc.) it is important to know.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for interfaces that are set to be up, but are actually down.
    without-indeni: |
        An administrator would normally use SNMP or write a script to poll the status and details of network interfaces and alert accordingly.
    can-with-snmp: true
    can-with-syslog: true
network-interface-speed:
    why: |
        Generally, these days network interfaces are set at 1Gbps or more. Sometimes, due to a configuration or device issue, an interface can be set below that (to 100mbps or even 10mbps). As that is usually _not_ the intended behavior, it is important to track the speed of all network interfaces.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for the actual runtime speed of each interface.
    without-indeni: |
        The status of network interfaces is visible through the web interface. If a configured interface is down, it will appear with a red indicator.
    can-with-snmp: true
    can-with-syslog: true
network-interface-bandwidth-mbps:
    why: |
        This is similar to network-interface-speed but returns double value for alerting purpose.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for the actual runtime speed of each interface.
    without-indeni: |
        The status of network interfaces is visible through the web interface. If a configured interface is down, it will appear with a red indicator.
    can-with-snmp: true
    can-with-syslog: true
network-interface-duplex:
    why: |
        Generally, these days network interfaces are set at full duplex. Sometimes, due to a configuration or device issue, an interface can be set to half duplex. As that is usually _not_ the intended behavior, it is important to track the duplex setting of all network interfaces.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for the actual runtime duplex of each interface.
    without-indeni: |
        The duplex setting of network interfaces can be manually reviewed through the CLI.
    can-with-snmp: true
    can-with-syslog: true
network-interface-tx-packets:
    why: |
        Tracking the number of packets flowing through each network interface is important to identify potential issues, spikes in traffic, etc.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for the number of packets transmitted through the interface.
    without-indeni: |
        The traffic statistics of network interfaces can be manually reviewed through the CLI.
    can-with-snmp: true
    can-with-syslog: true
network-interface-tx-bytes:
    why: |
        Tracking the amount of data flowing through each network interface is important to identify potential issues, spikes in traffic, etc.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for the number of bytes transmitted through the interface.
    without-indeni: |
        The traffic statistics of network interfaces can be manually reviewed through the CLI.
    can-with-snmp: true
    can-with-syslog: true
network-interface-rx-packets:
    why: |
        Tracking the number of packets flowing through each network interface is important to identify potential issues, spikes in traffic, etc.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for the number of packets received through the interface.
    without-indeni: |
        The traffic statistics of network interfaces can be manually reviewed through the CLI.
    can-with-snmp: true
    can-with-syslog: true
network-interface-rx-bytes:
    why: |
        Tracking the amount of data flowing through each network interface is important to identify potential issues, spikes in traffic, etc.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for the number of bytes received through the interface.
    without-indeni: |
        The traffic statistics of network interfaces can be manually reviewed through the CLI.
    can-with-snmp: true
    can-with-syslog: true
network-interface-rx-dropped:
    why: |
        If incoming packets are being dropped on a network interface, it is important to be aware of it. This may be due to a high load on the firewall, or another capacity issue.
    how: |
        This script logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for the number of packets dropped on an interface.
    without-indeni: |
        The traffic statistics of network interfaces can be manually reviewed through the CLI.
    can-with-snmp: true
    can-with-syslog: true
network-interface-state:
    why: |
        If a network interface is set to be up (what's known as "admin up") but is actually down (a cable is not connected, the device on the other side is down, etc.) it is important to know.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it looks for interfaces that are set to be up, but are actually down.
    without-indeni: |
        An administrator would normally use SNMP or write a script to poll the status and details of network interfaces and alert accordingly.
    can-with-snmp: true
    can-with-syslog: true
network-interface-mtu:
    why: |
        The MTU of an interface may be inadvertently set to a low value. It's important to know if this happens and fix it.
    how: |
        This alert logs into the Palo Alto Networks firewall through SSH and retrieves the status of all network interfaces. In that output, it retrieves the MTU values.
    without-indeni: |
        Alerting on low MTU values normally requires scripting to be done by the administrator.
    can-with-snmp: true
    can-with-syslog: true
network-interface-mac:
    skip-documentation: true
network-interface-rx-errors:
    skip-documentation: true
network-interfaces:
    skip-documentation: true

#! REMOTE::HTTP
url: /api?type=op&cmd=<show><interface>all</interface></show>&key=${api-key}
protocol: HTTPS

#! PARSER::XML
_vars:
    root: /response/result
_dynamic_vars:
    _dynamic:
        nic:
            _text: "${root}/ifnet/entry[contains(name, '/')]/name"

#! REMOTE::HTTP
url: /api?type=op&cmd=<show><interface>${nic}</interface></show>&key=${api-key}
protocol: HTTPS

#! PARSER::XML
_vars:
    root: /response/result
_metrics:
    -
        _tags:
            "im.name":
                _constant: "network-interface-state"
            "name":
                _text: "${root}/ifnet/name"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Interface state"
            "im.dstype.displayType":
                _constant: "state"
            "im.identity-tags":
                _constant: "name"
        _temp:
            "state":
                _text: "${root}/hw/state"
        _transform:
            _value.double: |
                {
                    if (temp("state") == "up") {
                        print "1"
                    } else {
                        print "0"
                    }
                }
    -
        _tags:
            "im.name":
                _constant: "network-interface-admin-state"
            "name":
                _text: "${root}/ifnet/name"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Interface admin states"
            "im.dstype.displayType":
                _constant: "state"
            "im.identity-tags":
                _constant: "name"
        _temp:
            "state_c":
                _text: "${root}/hw/state_c"
        _transform:
            _value.double: |
                {
                    if (temp("state_c") == "up" || temp("state_c") == "auto") {
                        print "1"
                    } else {
                        print "0"
                    }
                }
    -
        _tags:
            "im.name":
                _constant: "network-interface-duplex"
            "name":
                _text: "${root}/ifnet/name"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Interface duplex settings"
            "im.identity-tags":
                _constant: "name"
        _value.complex:
            value:
                _text: "${root}/hw[not(duplex = 'unknown')]/duplex"
    -
        _tags:
            "im.name":
                _constant: "network-interface-speed"
            "name":
                _text: "${root}/ifnet/name"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Interface speeds"
            "im.identity-tags":
                _constant: "name"
        _temp:
            "speed":
                _text: ${root}/hw[not(speed = 'unknown')]/speed
        _transform:
            _value.complex:
                value: |
                    {
                        print temp("speed") "M"
                    }
    -
        _tags:
            "im.name":
                _constant: "network-interface-bandwidth-mbps"
            "name":
                _text: "${root}/ifnet/name"
        _transform:
            _tags:
                "name": |
                    {
                        print dynamic("nic")
                    }
        _value.double:
            _text: ${root}/hw[not(speed = 'unknown')]/speed
    -
        _tags:
            "im.name":
                _constant: "network-interface-mtu"
            "name":
                _text: "${root}/ifnet/name"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Interface MTU settings"
            "im.dstype.displayType":
                _constant: "number"
            "im.identity-tags":
                _constant: "name"
        _value.complex:
            value:
                _text: "${root}/ifnet/mtu"
    -
        _tags:
            "im.name":
                _constant: "network-interface-mac"
            "name":
                _text: "${root}/ifnet/name"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Interface mac addresses"
            "im.identity-tags":
                _constant: "name"
        _value.complex:
            value:
                _text: "${root}/hw/mac"
    -
        _tags:
            "im.name":
                _constant: "network-interface-tx-bytes"
            "name":
                _text: "${root}/ifnet/name"
            "im.dsType":
                _constant: "counter"
        _value.double:
            _text: "${root}/ifnet/counters/hw/entry/obytes"
    -
        _tags:
            "im.name":
                _constant: "network-interface-rx-bytes"
            "name":
                _text: "${root}/ifnet/name"
            "im.dsType":
                _constant: "counter"
        _value.double:
            _text: "${root}/ifnet/counters/hw/entry/ibytes"
    -
        _tags:
            "im.name":
                _constant: "network-interface-tx-packets"
            "name":
                _text: "${root}/ifnet/name"
            "im.dsType":
                _constant: "counter"
        _value.double:
            _text: "${root}/ifnet/counters/hw/entry/opackets"
    -
        _tags:
            "im.name":
                _constant: "network-interface-rx-packets"
            "name":
                _text: "${root}/ifnet/name"
            "im.dsType":
                _constant: "counter"
        _value.double:
            _text: "${root}/ifnet/counters/hw/entry/ipackets"
    -
        _tags:
            "im.name":
                _constant: "network-interface-rx-errors"
            "name":
                _text: "${root}/ifnet/name"
            "live-config":
                _constant: "true"
            "im.dsType":
                _constant: "counter"
            "display-name":
                _constant: "Interface RX errors"
            "im.dstype.displayType":
                _constant: "number"
            "im.identity-tags":
                _constant: "name"
        _value.double:
            _text: "${root}/ifnet/counters/hw/entry/ierrors"
    -
        _tags:
            "im.name":
                _constant: "network-interface-rx-dropped"
            "name":
                _text: "${root}/ifnet/name"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Interface rx dropped"
            "im.dsType":
                _constant: "counter"
            "im.dstype.displayType":
                _constant: "number"
            "im.identity-tags":
                _constant: "name"
        _value.double:
            _text: "${root}/ifnet/counters/hw/entry/idrops"
    -
        _tags:
            "im.name":
                _constant: "network-interfaces"
        _value.complex:
            "name":
                _text: "${root}/ifnet/name"
        _value: complex-array

cross_vendor_rx_overrun

package com.indeni.server.rules.library.templatebased.crossvendor

import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library.{ConditionalRemediationSteps, NearingCapacityWithItemsTemplateRule}

/**
  *
  */
case class CrossVendorRxOverrun() extends NearingCapacityWithItemsTemplateRule(
  ruleName = "cross_vendor_rx_overrun",
  ruleFriendlyName = "All Devices: RX packets overrun",
  ruleDescription = "Indeni tracks the number of packets that had issues and alerts if the ratio is too high.",
  usageMetricName = "network-interface-rx-overruns",
  limitMetricName = "network-interface-rx-packets",
  applicableMetricTag = "name",
  threshold = 0.5,
  minimumValueToAlert = 100.0, // We don't want to alert if the number of error packets is really low
  alertDescription = "Some network interfaces and ports are experiencing a high overrun rate. Review the ports below.",
  alertItemDescriptionFormat = "%.0f packets overrun out of a total of %.0f received.",
  baseRemediationText = "Packet overruns usually occur when there are too many packets being inserted into the port's memory buffer, faster than the rate at which the kernel is able to process them.",
  alertItemsHeader = "Affected Ports")(
  ConditionalRemediationSteps.OS_NXOS ->
    """|
       |In a small number of cases, the overrun counter may be incremented because of a software defect. However, in the majority of cases, it indicates that the receiving capability of the interface was exceeded. Nothing can be done on the device that reports overruns. If possible, the rate that frames are coming should be controlled at the remote end of the connection.
       |1. Run the "show interface" command to review the interface overrun & underun counters and the bitrate. Consider to configure the "load-interval 30" interface sub command to improve the accuracy of the interface measurements.
       |2. If the number of overruns is high, the hardware should be upgraded.
       |
       |In case of high bandwidth utilization:
       |1. Run the "show interface" command to review the interface counters and the bitrate. Consider to configure the "load-interval 30" interface sub command to improve the accuracy of the interface measurements.
       |2. If the interface bitrate is too high and close to the upper bandwidth limit consider to use multiple links with the port-channel technology or upgrade the bandwidth of the link.
       |3. Consider to implement QoS in case of high bandwidth utilization.""".stripMargin,
  ConditionalRemediationSteps.VENDOR_JUNIPER ->
    """|The issue can not be resolved on the device that reports overruns. The rate that frames are coming should be controlled at the remote end of the connection.
       |1. Run the “show interface extensive” command to review the interface overrun counters. 

       |2. If the number of overruns is high, consider upgrading the hardware.
       |3. Review the following article on Juniper tech support site: <a target="_blank" href="https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/show-interfaces-security.html#jd0e1772">Operational Commands</a>""".stripMargin,
  ConditionalRemediationSteps.VENDOR_FORTINET ->
    """
       |1. Run "diag hardware deviceinfo nic <interface>" command to display a list of hardware related error names and values. Review  the next link for more details: http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-toubleshooting-54/troubleshooting_tools.htm
       |2. Run command "fnsysctl cat /proc/net/dev" to get a summary of the interface statistics.
       |3. Check for speed and duplex mismatch in the interface settings on both sides of a cable, and check for a damaged cable. Review the next link for more info: http://kb.fortinet.com/kb/documentLink.do?externalID=10653""".stripMargin
)