Interface nearing maximum Tx throughput-checkpoint-ipso

error
health-checks
checkpoint
ipso
Interface nearing maximum Tx throughput-checkpoint-ipso
0

#1

Interface nearing maximum Tx throughput-checkpoint-ipso

Vendor: checkpoint

OS: ipso

Description:
The interface is close to its maximum advertised throughput limit. Intermittent connectivity issues may occur such as network performance degradation or the inability to reach resources. This could be due to poor capacity management or user activity such as video streaming or file transfers. This could also be due to unexpected and suspicious activity.

Remediation Steps:
Follow the vendor specific remediation steps below.
Check point’s CLI cpview or SmartView Monitor tool can be used to view network top talkers such as source, destination, and protocol(s).
|Based on this information, it can be determined as to whether selected traffic should be blocked or dropped. If the throughput threshold reached is expected, consider adding more interfaces or modifying any applicable packet shaping.

How does this work?
Indeni will record the total bytes transmitted, wait a pre-determined amount of time and then record it again. By comparing before and after a value for how many bytes sent during the period of time can be determined.

Why is this important?
If the throughput reaches the limit, packets will be dropped.

Without Indeni how would you find this?
An administrator could login and manually check this from the command line interface.

chkp-ipso-throughput-alert

#! META
name: chkp-ipso-throughput-alert
description: Check the current throughput for recieve and transmit for interfaces.
type: monitoring
monitoring_interval: 1 minutes
requires:
    vendor: checkpoint
    os.name: ipso

#! COMMENTS
network-interface-tx-util-percentage:
    why: |
        If the throughput reaches the limit, packets will be dropped.
    how: |
        Indeni will record the total bytes transmitted, wait a pre-determined amount of time and then record it again. By comparing before and after a value for how many bytes sent during the period of time can be determined.
    without-indeni: |
        An administrator could login and manually check this from the command line interface.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface or SmartView Monitor.

network-interface-rx-util-percentage:
    why: |
        If the throughput reaches the limit, packets will be dropped.
    how: |
        Indeni will record the total bytes received, wait a pre-determined amount of time and then record it again. By comparing before and after a value for how many bytes sent during the period of time can be determined.
    without-indeni: |
        An administrator could login and manually check this from the command line interface.
    can-with-snmp: true
    can-with-syslog: false
    vendor-provided-management: |
        This is only accessible from the command line interface or SmartView Monitor.

#! REMOTE::SSH
${nice-path} -n 15 ifconfig -a ; netstat -idb ; sleep 10 ; netstat -idb 

#! PARSER::AWK

##################################
#
#	Determine interface name
#	
##################################

# eth1c0:  lname eth1c0 flags=e0<BROADCAST,MULTICAST,AUTOLINK>
/^[a-z0-9]+: / {
    interfaceName = $1
    sub(/:/, "", interfaceName)

    # netstat uses interface names with the "c0" (the second identifier), so we need to drop it
    if (interfaceName ~ /[A-Za-z]+[0-9]+[A-Za-z]+[0-9]+/) {
    	sub(/[A-Za-z]+[0-9]+$/, "", interfaceName)
    }
}

#eth1         16018 <Link>      0:a0:8e:b2:22:34        0     0          0        0     0          0     0   0
/^[a-z0-9]+ .*[0-9]+.*:/ {
	interfaceName = $1
}

##################################
#
#	Record bytes sent/received
#	
##################################

# Name         Mtu   Network     Address             Ipkts Ierrs     Ibytes    Opkts Oerrs     Obytes  Coll Drop
# eth2         16018 <Link>      0:a0:8e:b2:22:35    39706   440    5481041    28917     0   10158270  1898   0
/Name.*Mtu.*Network/ {
    inNetstat = "true"
    getColumns(trim($0), "[ \t]+", columns)
}

# eth2         16018 <Link>      0:a0:8e:b2:22:35    39706   440    5481041    28917     0   10158270  1898   0
/^[a-z0-9]+ .*[0-9]+.*:/ {
	bytesRx = getColData(trim($0), columns, "Ibytes")
	bytesTx = getColData(trim($0), columns, "Obytes")
	
	if (interfaceName in bytesTxArr) { # if the array bytesTxArr already contains data for the interface
		# Calculate diff
		# How many bytes sent first run - how many bytes sent second run
		# Divided by 10 - To get into per second as first and second run was 10 seconds apart
		# Times 8 - convert from byte to bit
		# Divided into 1000 - Get into kilobit
		bytesTxArr[interfaceName] = (((bytesTx - bytesTxArr[interfaceName]) / 10) * 8) / 1000  # Result kilobit per second
	} else {
		bytesTxArr[interfaceName] = bytesTx
	}
	
	if (interfaceName in bytesRxArr) { # if the array bytesRxArr already contains data for the interface
		# Calculate diff
		# How many bytes sent first run - how many bytes sent second run
		# Divided by 10 - To get into per second as first and second run was 10 seconds apart
		# Times 8 - convert from byte to bit
		# Divided into 1000 - Get into kilobit
		bytesRxArr[interfaceName] = (((bytesRx - bytesRxArr[interfaceName]) / 10) * 8) / 1000 # Result kilobit per second
	} else {
		bytesRxArr[interfaceName] = bytesRx
	}
}

##################################
#
#	Determine speed of interface
#	
##################################

#         ether 00:a0:8e:b2:22:37 speed 10M half duplex
/ speed .* duplex/ {
    speed = $0
    sub(/.* speed /, "", speed)
    sub(/ .*/, "", speed)

    if (speed ~ /M/) {
    	speedUnit = "M"
    }

    speedData = speed
    sub(/[A-Z]/, "", speedData)

	# Have not seen prefix on interface speed be anything else than Mega so far, but this would make it easy to add others if needed in the future.
	if (speedUnit == "M") {
		speedKbitArr[interfaceName] = speedData * 1000
	}
}

END {

	##################################
	#
	#	Remove interfaces that do not 
	#	have a known speed
	#	
	##################################
	for (interface in bytesTxArr) {
		if (!(interface in speedKbitArr)) {
			delete bytesTxArr[interface]
		}	
	}
	
	for (interface in bytesRxArr) {
		if (!(interface in speedKbitArr)) {
			delete bytesRxArr[interface]
		}	
	}
	
	##################################
	#
	#	Calculate percentage interface 
	#	usage and write metric data
	#	
	##################################	

	for (interface in bytesTxArr) {
		interfaceTags["name"] = interface
		percentageTxUsed[interface] = (bytesTxArr[interface] / speedKbitArr[interface]) * 100
		writeDoubleMetricWithLiveConfig("network-interface-tx-util-percentage", interfaceTags, "gauge", "60", percentageTxUsed[interface], "Network Interfaces - Throughput Transmit", "percentage", "name")
	}
	
	for (interface in bytesRxArr) {
		interfaceTags["name"] = interface
		percentageRxUsed[interface] = (bytesRxArr[interface] / speedKbitArr[interface]) * 100
		writeDoubleMetricWithLiveConfig("network-interface-rx-util-percentage", interfaceTags, "gauge", "60", percentageRxUsed[interface], "Network Interfaces - Throughput Receive", "percentage", "name")
	}
}

cross_vendor_interface_tx_utilization

package com.indeni.server.rules.library.templatebased.crossvendor

import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library.{ConditionalRemediationSteps, NearingCapacityWithItemsTemplateRule}

/**
  * Contributed by Indeni_PJ
  */
case class crossVendorInterfaceTXUtilization(context: RuleContext) extends NearingCapacityWithItemsTemplateRule(context,
  ruleName = "cross_vendor_interface_tx_utilization",
  ruleFriendlyName = "All Devices: Interface nearing maximum Tx throughput",
  ruleDescription = "The interface is close to its maximum advertised throughput limit. Intermittent connectivity issues may occur such as network performance degradation or the inability to reach resources. This could be due to poor capacity management or user activity such as video streaming or file transfers. This could also be due to unexpected and suspicious activity.",
  usageMetricName = "network-interface-tx-util-percentage",
  applicableMetricTag = "name",
  threshold = 80.0,
  alertItemsHeader = "Affected Interfaces",
  alertItemDescriptionFormat  = "The interface is sending at %.0f%% of it's capacity.",
  alertDescription = "Interfaces are nearing their maximum throughput. Connectivity issues may occur due to this. This could be due to poor capacity management, user activity such as video streaming or file transfers. This could also be due to unexpected activity.",
  baseRemediationText = "Follow the vendor specific remediation steps below.")(
  ConditionalRemediationSteps.VENDOR_CP -> """Check point’s CLI cpview or SmartView Monitor tool can be used to view network top talkers such as source, destination, and protocol(s). 
                                              |Based on this information, it can be determined as to whether selected traffic should be blocked or dropped. If the throughput threshold reached is expected, consider adding more interfaces or modifying any applicable packet shaping.""".stripMargin,
  ConditionalRemediationSteps.VENDOR_PANOS -> """Track down network top talkers such as source, destination, and application(s). This can be done by logging into the WebUI and viewing network activity on the ACC tab and Monitoring tab to view logs to look for patterns.
                                                |Based on this information, it can be determined as to whether selected traffic should be blocked or dropped. If the throughput utilization is expected or common in the environment, consider adding more interfaces or modifying any applicable packet shaping.""".stripMargin)