Hardware element down-f5-False

error
health-checks
false
f5
Hardware element down-f5-False
0

#1

Hardware element down-f5-False

Vendor: f5

OS: False

Description:
Alert if any hardware elements are not operating correctly.

Remediation Steps:
Troubleshoot the hardware element as soon as possible.

How does this work?
This alert logs into the device through SSH and uses TMSH to retrieve the health of any configured RAID arrays.

Why is this important?
A critical aspect to track on a given device is the health of the hardware components. A power supply which stopped working or a dead fan can spell trouble down the line.

Without Indeni how would you find this?
An administrator could check this metric manually by logging into the device through TMSH and executing the command “show sys raid array”. Please note that only some platforms uses RAID and for those that doesn’t this command would not show any output.

f5-tmsh-show-sys-raid-array

#! META
name: f5-tmsh-show-sys-raid-array
description: Get the status of configured arrays, if any
type: monitoring
monitoring_interval: 10 minutes
requires:
    vendor: "f5"
    product: "load-balancer"
    linux-based: "true"
    shell: "bash"

#! COMMENTS
hardware-element-status:
    why: |
        A critical aspect to track on a given device is the health of the hardware components. A power supply which stopped working or a dead fan can spell trouble down the line.
    how: |
        This alert logs into the device through SSH and uses TMSH to retrieve the health of any configured RAID arrays.
    without-indeni: |
        An administrator could check this metric manually by logging into the device through TMSH and executing the command "show sys raid array". Please note that only some platforms uses RAID and for those that doesn't this command would not show any output.
    can-with-snmp: true
    can-with-syslog: false
#! REMOTE::SSH
tmsh -q show sys raid array

#! PARSER::AWK

# Example: 
#1 WD-WCAT18586780 HD2 yes failed
#2 WD-WCAT1E733419 HD1 yes ok
# or if theres a RAID with only 1 drive active:
#1 WD-WCAT18586780 HD2 yes ok

/^[0-9]+\s/ {
    # This will check if Array member is set to yes, if so it will add it to a counter that we use afterwards
    if ($4 == "yes") {
        array_member++
    }
    if ($NF != "ok" || raid_status == 0) {
        raid_status = 0
    } else {
        raid_status = 1
    }
}

END {
    # If there is a raid configured, and one drive is missing/dead and tmsh still says status is ok, it will alert
    if ( (array_member == 1 ) && ( raid_status = 0 ) ) {
        raid_status = 0
    }


    if (match(/(0|1)/, raid_status)) {
        raidTags["name"] = "RAID"
        writeDoubleMetric("hardware-element-status", raidTags, "state", 600, raid_status)
    }
}

cross_vendor_hardware_element_status

package com.indeni.server.rules.library.templatebased.crossvendor

import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library.ConditionalRemediationSteps
import com.indeni.server.rules.library.templates.StateDownTemplateRule

/**
  *
  */
case class cross_vendor_hardware_element_status() extends StateDownTemplateRule(
  ruleName = "cross_vendor_hardware_element_status",
  ruleFriendlyName = "All Devices: Hardware element down",
  ruleDescription = "Alert if any hardware elements are not operating correctly.",
  metricName = "hardware-element-status",
  applicableMetricTag = "name",
  alertItemsHeader = "Hardware Elements Affected",
  alertDescription = "The hardware elements listed below are not operating correctly.",
  baseRemediationText = "Troubleshoot the hardware element as soon as possible.")(
  ConditionalRemediationSteps.OS_NXOS ->
    """|While the port may be in up status, the link quality might be degraded and is not between the threshold levels. Check the following to troubleshoot this issue.
       |1.	Run the “show interface transceiver detailed” NX-OS command to display information about the transceivers connected to a specific interface. Besides, this NX-OS command output provides information about the Cisco SFP Product ID (PID). NOTE: In case that have been used 3rd party SFPs it is possible to get an Indeni alert because the current light signal is different than the recommended min/max thresholds defined by Cisco.
       |2.	Use the “show interface transceiver calibrations” NX-OS command to display calibration information for the transceiver interfaces.
       |3.	Consider to enable DOM (if supported). Digital Optical Monitoring or DOM is an industry wide standard, intended to define a SFP to access real-time operating parameters such as Tx power, Rx power etc. More details can be found below: https://www.cisco.com/c/en/us/td/docs/interfaces_modules/transceiver_modules/compatibility/matrix/DOM_matrix.html
       |4.	Cisco has published official specifications (Rx, Tx power level etc) per transceiver category and can be found at the following link:
        https://www.cisco.com/c/en/us/products/interfaces-modules/transceiver-modules/index.""".stripMargin,
  ConditionalRemediationSteps.VENDOR_FORTINET ->
    """
      |1. Login via ssh to the Fortinet firewall and run the FortiOS command "exec sensor list" to review the status of the hardware components and temperature
      |>>> thresholds. When the flag to the command output is set to 0, the component is working correctly and when flag is set to 1, the component has a problem.
      |>>> The FortiOS command "execute sensor detail" will show extra information such as the low/high thresholds. More details can be found here:
      |>>> http://kb.fortinet.com/kb/viewContent.do?externalId=FD36793&sliceId=1
      |2. Consider running the fotrinet hardware diagnostics commands. While they do not detect all hardware malfunctions, tests for the most common hardware
      |>>> problems are performed. More details can be found here:
      |- http://kb.fortinet.com/kb/viewContent.do?externalId=FD39581&sliceId=1
      |- http://kb.fortinet.com/kb/documentLink.do?externalID=FD34745
      |3. It is recommended that any failed fan or power supply unit should be replaced immediately.
      |4. The cooling system for the devices should be installed to avoid overheat.
      |5. If the problem persists, contact Fortinet Technical support at https://support.fortinet.com/ for further assistance.""".stripMargin.replaceAll("\n>>>", "")
)