Device restarted (uptime low)-juniper-junos

health-checks
critical
junos
juniper
Device restarted (uptime low)-juniper-junos
0

#1

Device restarted (uptime low)-juniper-junos

Vendor: juniper

OS: junos

Description:
Indeni will alert when a device has restarted.

Remediation Steps:
Determine why the device was restarted.

How does this work?
This script logs into the Juniper JUNOS-based device using SSH and retrieves the output of the “show system uptime” command. The output includes the device’s uptime as well as additional information.

Why is this important?
Capture the uptime of the device. If the uptime is lower than the previous sample, the device must have reloaded.

Without Indeni how would you find this?
It is possible to poll this data through SNMP or capture a syslog/trap event of a device booting up.

junos-show-system-uptime

#! META
name: junos-show-system-uptime
description: Fetches system uptime
type: monitoring
monitoring_interval: 5 minute
requires:
    vendor: "juniper"
    os.name: "junos"
    high-availability:
        neq: "true"

#! COMMENTS
uptime-milliseconds:
    why: |
        Capture the uptime of the device. If the uptime is lower than the previous sample, the device must have reloaded.
    how: |
        This script logs into the Juniper JUNOS-based device using SSH and retrieves the output of the "show system uptime" command. The output includes the device's uptime as well as additional information.
    without-indeni: |
        It is possible to poll this data through SNMP or capture a syslog/trap event of a device booting up.
    can-with-snmp: true
    can-with-syslog: true

current-datetime:
    why: |
        Capture the current date and time of the device. Device current date and time should never be more than 24 hours away from date and time of the device polling the data, otherwise date and time are not correctly set on device.
    how: |
        This script logs into the Juniper JUNOS-based device using SSH and retrieves the current time using the output of the "show system uptime" command. The output includes the device's current date and time as well as configured time zone.
    without-indeni: |
        It is possible to poll this data through SNMP.
    can-with-snmp: true
    can-with-syslog: false

timezone:
    why: |
        Capture the current time zone of the device. The time zone information is useful for display purposes.
    how: |
        This script logs into the Juniper JUNOS-based device using SSH and retrieves the configured time zone using the output of the "show system uptime" command. The output includes the device's current date and time as well as configured time zone.
    without-indeni: |
        An administrator may write a script to pull this data from cluster members and compare it.
    can-with-snmp: false
    can-with-syslog: false

#! REMOTE::SSH
show system uptime | display xml

#! PARSER::XML
_vars:
    root: /rpc-reply//system-uptime-information[1]
_metrics:
    -
        _tags:
            "im.name":
                _constant: "uptime-milliseconds"
            "live-config":
                _constant: "true"
            "im.dstype.displayType":
                _constant: "duration"
            "display-name":
                _constant: "Uptime"
        _temp:
            "uptimeSeconds":
                _attribute:
                    _name: "junos:seconds"
                    _path: "${root}/uptime-information/up-time"
        _transform:
            _value.double: |
                {
                    uptime = temp("uptimeSeconds") * 1000
                    print uptime
                }
    -
        _tags:
            "im.name":
                _constant: "current-datetime"
            "live-config":
                _constant: "true"
            "im.dstype.displayType":
                _constant: "date"
            "display-name":
                _constant: "Current date/time"
        _value.double:
            _attribute:
                _name: "junos:seconds"
                _path: "${root}/uptime-information/date-time"
    -
        _tags:
            "im.name":
                _constant: "timezone"
            "live-config":
                _constant: "true"
            "display-name":
                _constant: "Time Zone"
        _temp:
            dateTime:
                _text: "${root}/current-time/date-time"
        _transform:
            _value.complex:
                value: |
                    {
                        currentTime = temp("dateTime")
                        print substr(currentTime, length(currentTime) - 3, length(currentTime))
                    }

cross_vendor_uptime_low

package com.indeni.server.rules.library.templatebased.crossvendor

import com.indeni.apidata.time.TimeSpan
import com.indeni.apidata.time.TimeSpan.TimePeriod
import com.indeni.server.common.data.conditions.Equals
import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library._
import com.indeni.server.sensor.models.managementprocess.alerts.dto.AlertSeverity

case class cross_vendor_uptime_low(context: RuleContext) extends TimeIntervalThresholdOnDoubleMetricTemplateRule(context,
  ruleName = "cross_vendor_uptime_low",
  ruleFriendlyName = "All Devices (Non-VSX): Device restarted (uptime low)",
  ruleDescription = "Indeni will alert when a device has restarted.",
  severity = AlertSeverity.CRITICAL,
  metricName = "uptime-milliseconds",
  threshold = TimeSpan.fromMinutes(60),
  metricUnits = TimePeriod.MILLISECOND,
  thresholdDirection = ThresholdDirection.BELOW,
  alertDescriptionFormat = "The current uptime is %.0f seconds which seems to indicate the device has restarted.",
  alertDescriptionValueUnits = TimePeriod.SECOND,
  baseRemediationText = "Determine why the device was restarted.",
  metaCondition = !Equals("vsx", "true")
)(
  ConditionalRemediationSteps.OS_NXOS ->
    """|
       |1. Use the "show version" or "show system reset-reason" NX-OS commands to display the reason for the reload.
       |2. Use the "show cores" command to determine if a core file was recorded during the unexpected reboot.
       |3. Run the "show process log" command to display the processes and if a core was created.
       |4. With the show logging command, review the events that happened close to the time of reboot.""".stripMargin,
  ConditionalRemediationSteps.VENDOR_FORTINET ->
    """
      |1. Watch the system reboot time.
      |2. Review the log messages and focus on error messages that were generated at least 5 minutes prior to system reboot, especially before unexpected system reboot.
      |3. Verify the status of the scheduled restart command to making  sure it's an irregular restart
      |   - config sys global
      |   - get | grep restart
      |   - end
      |4. Login via ssh to the Fortinet firewall and review the crash log in a readable format by using the FortiOS command “diag debug crashlog read”.
      |5. Contact Fortinet Technical support at https://support.fortinet.com/ for further assistance.""".stripMargin
)