Hardware element down-paloaltonetworks-panos

error
health-checks
panos
paloaltonetworks
Hardware element down-paloaltonetworks-panos
0

#1

Hardware element down-paloaltonetworks-panos

Vendor: paloaltonetworks

OS: panos

Description:
Alert if any hardware elements are not operating correctly.

Remediation Steps:
Troubleshoot the hardware element as soon as possible.

How does this work?
This script leverages the PAN-OS’s XML API to glean hardware status of specific components (e.g. fans, power supply, temperature). The API call is for “show system environmentals”. Any reported hardware element issue or out of range value (e.g. temperature) will trigger an alert.

Why is this important?
Network devices that lose functionality of core components are at risk for complete system failure. It is important to immediately address any symptoms leading up to a failure. This includes increased thermal levels and loss of redundant components.

Without Indeni how would you find this?
An administrator could physically view the LED lights for alarm status. Also, running the command “show system environmentals” at the CLI will show status of hardware components.

panos-show-system-environmentals

#! META
name: panos-show-system-environmentals
description: fetch the status of the fan, power and temperature
type: monitoring
monitoring_interval: 10 minute
requires:
    vendor: paloaltonetworks
    os.name: panos

#! COMMENTS
hardware-element-status:
    why: |
        Network devices that lose functionality of core components are at risk for complete system failure. It is important to immediately address any symptoms leading up to a failure. This includes increased thermal levels and loss of redundant components.
    how: |
        This script leverages the PAN-OS's XML API to glean hardware status of specific components (e.g. fans, power supply, temperature). The API call is for "show system environmentals". Any reported hardware element issue or out of range value (e.g. temperature) will trigger an alert.
    without-indeni: |
        An administrator could physically view the LED lights for alarm status. Also, running the command "show system environmentals" at the CLI will show status of hardware components.
    can-with-snmp: true
    can-with-syslog: true
    vendor-provided-management: |
        PAN-OS appliances typically have a LED status light indicating alarms. Panorama will show if hardware status is in alarm.
power-supply-inserted:
    why: |
        Network devices that do not have redundant power supplies are at risk for a complete service outage should there be a loss of power to the device. It is best practice to install multiple PSU's with separate power sources whenever possible. 
    how: |
        This script leverages the PAN-OS's XML API to detect that all available PSU slots are in use. Detection of an empty PSU slot will trigger an alert. 
    without-indeni: |
        An administrator would need to SSH into CLI on the device and run the command "show system environmentals" to clearly identify if a PSU is installed in each available slot. 
    can-with-snmp: true
    can-with-syslog: false
 

#! REMOTE::HTTP
url: /api?type=op&cmd=<show><system><environmentals></environmentals></system></show>&key=${api-key}
protocol: HTTPS

#! PARSER::XML
_vars:
    root: /response/result
_optional_metrics:
    -
        _groups:
            ${root}/thermal/*/entry:
                _temp:
                    alarm:
                        _text: alarm
                _tags:
                    "name":
                        _text: "description"
                    "im.name":
                        _constant: "hardware-element-status"
                    "live-config":
                       _constant: "true"
                    "display-name":
                        _constant: "Hardware Elements State"
                    "im.dstype.displayType":
                        _constant: "state"
                    "im.identity-tags":
                        _constant: "name"
        _transform:
            _value.double: |
                {
                    if (temp("alarm") == "True") { print "0.0" } else { print "1.0" }
                }
    -
        _groups:
            ${root}/thermal/*/entry:
                _tags:
                    "name":
                        _text: "description"
                    "im.name":
                        _constant: "temperature-sensor-current"
                    "live-config":
                       _constant: "true"
                    "display-name":
                        _constant: "Temperature - Current"
                    "im.dstype.displayType":
                        _constant: "number"
                    "im.identity-tags":
                        _constant: "name"
                _value.double:
                    _text: "DegreesC"
    -
        _groups:
            ${root}/thermal/*/entry:
                _tags:
                    "name":
                        _text: "description"
                    "im.name":
                        _constant: "temperature-sensor-max"
                    "live-config":
                       _constant: "true"
                    "display-name":
                        _constant: "Temperature - Max Limit"
                    "im.dstype.displayType":
                        _constant: "number"
                    "im.identity-tags":
                        _constant: "name"
                _value.double:
                    _text: "max"
    -
        _groups:
            ${root}/fan/*/entry:
                _temp:
                    alarm:
                        _text: alarm
                _tags:
                    "name":
                        _text: "description"
                    "im.name":
                        _constant: "hardware-element-status"
                    "live-config":
                       _constant: "true"
                    "display-name":
                        _constant: "Hardware Elements State"
                    "im.dstype.displayType":
                        _constant: "state"
                    "im.identity-tags":
                        _constant: "name"
        _transform:
            _value.double: |
                {
                    if (temp("alarm") == "True") { print "0.0" } else { print "1.0" }
                }
    -
        _groups:
            ${root}/power/*/entry:
                _temp:
                    alarm:
                        _text: alarm
                _tags:
                    "name":
                        _text: "description"
                    "im.name":
                        _constant: "hardware-element-status"
                    "live-config":
                       _constant: "true"
                    "display-name":
                        _constant: "Hardware Elements State"
                    "im.dstype.displayType":
                        _constant: "state"
                    "im.identity-tags":
                        _constant: "name"
        _transform:
            _value.double: |
                {
                    if (temp("alarm") == "True") { print "0.0" } else { print "1.0" }
                }
    -
        _groups:
            ${root}/power-supply/*/entry:
                _temp:
                    alarm:
                        _text: alarm
                _tags:
                    "name":
                        _text: "description"
                    "im.name":
                        _constant: "hardware-element-status"
                    "live-config":
                       _constant: "true"
                    "display-name":
                        _constant: "Hardware Elements State"
                    "im.dstype.displayType":
                        _constant: "state"
                    "im.identity-tags":
                        _constant: "name"
        _transform:
            _value.double: |
                {
                    if (temp("alarm") == "True") { print "0.0" } else { print "1.0" }
                }
    -
        _groups:
            ${root}/power-supply/*/entry:
                _temp:
                    inserted:
                        _text: Inserted
                _tags:
                    "name":
                        _text: "description"
                    "im.name":
                        _constant: "power-supply-inserted"
                    "live-config":
                       _constant: "true"
                    "display-name":
                       _constant: "PSU Detected"
                    "im.dstype.displayType":
                        _constant: "state"
                    "im.identity-tags":
                        _constant: "name"
        _transform:
            _value.double: |
                {
                    if (temp("inserted") == "True") { print "1.0" } else { print "0.0" }
                }

cross_vendor_hardware_element_status

package com.indeni.server.rules.library.templatebased.crossvendor

import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library.ConditionalRemediationSteps
import com.indeni.server.rules.library.templates.StateDownTemplateRule

/**
  *
  */
case class cross_vendor_hardware_element_status() extends StateDownTemplateRule(
  ruleName = "cross_vendor_hardware_element_status",
  ruleFriendlyName = "All Devices: Hardware element down",
  ruleDescription = "Alert if any hardware elements are not operating correctly.",
  metricName = "hardware-element-status",
  applicableMetricTag = "name",
  alertItemsHeader = "Hardware Elements Affected",
  alertDescription = "The hardware elements listed below are not operating correctly.",
  baseRemediationText = "Troubleshoot the hardware element as soon as possible.")(
  ConditionalRemediationSteps.OS_NXOS ->
    """|While the port may be in up status, the link quality might be degraded and is not between the threshold levels. Check the following to troubleshoot this issue.
       |1.	Run the “show interface transceiver detailed” NX-OS command to display information about the transceivers connected to a specific interface. Besides, this NX-OS command output provides information about the Cisco SFP Product ID (PID). NOTE: In case that have been used 3rd party SFPs it is possible to get an Indeni alert because the current light signal is different than the recommended min/max thresholds defined by Cisco.
       |2.	Use the “show interface transceiver calibrations” NX-OS command to display calibration information for the transceiver interfaces.
       |3.	Consider to enable DOM (if supported). Digital Optical Monitoring or DOM is an industry wide standard, intended to define a SFP to access real-time operating parameters such as Tx power, Rx power etc. More details can be found below: https://www.cisco.com/c/en/us/td/docs/interfaces_modules/transceiver_modules/compatibility/matrix/DOM_matrix.html
       |4.	Cisco has published official specifications (Rx, Tx power level etc) per transceiver category and can be found at the following link:
        https://www.cisco.com/c/en/us/products/interfaces-modules/transceiver-modules/index.""".stripMargin,
  ConditionalRemediationSteps.VENDOR_FORTINET ->
    """
      |1. Login via ssh to the Fortinet firewall and run the FortiOS command "exec sensor list" to review the status of the hardware components and temperature
      |>>> thresholds. When the flag to the command output is set to 0, the component is working correctly and when flag is set to 1, the component has a problem.
      |>>> The FortiOS command "execute sensor detail" will show extra information such as the low/high thresholds. More details can be found here:
      |>>> http://kb.fortinet.com/kb/viewContent.do?externalId=FD36793&sliceId=1
      |2. Consider running the fotrinet hardware diagnostics commands. While they do not detect all hardware malfunctions, tests for the most common hardware
      |>>> problems are performed. More details can be found here:
      |- http://kb.fortinet.com/kb/viewContent.do?externalId=FD39581&sliceId=1
      |- http://kb.fortinet.com/kb/documentLink.do?externalID=FD34745
      |3. It is recommended that any failed fan or power supply unit should be replaced immediately.
      |4. The cooling system for the devices should be installed to avoid overheat.
      |5. If the problem persists, contact Fortinet Technical support at https://support.fortinet.com/ for further assistance.""".stripMargin.replaceAll("\n>>>", "")
)