Device restarted (uptime low)-fortinet-FortiOS

Device restarted (uptime low)-fortinet-FortiOS
0

Device restarted (uptime low)-fortinet-FortiOS

Vendor: fortinet

OS: FortiOS

Description:
Indeni will alert when a device has restarted.

Remediation Steps:
Determine why the device was restarted.

How does this work?
Indeni uses the built-in Fortinet “get system performance status” command to retrieve the current device up-time.

Why is this important?
Capture the uptime of the device. If the uptime is lower than the previous sample, the device must have reloaded.

Without Indeni how would you find this?
An administrator could login and manually run the command via CLI, check the system resources widget via the GUI, enable SNMP, or use Fortinet FortiAnalyzer.

fortios-get-system-performance-status

name: fortios-get-system-performance-status
description: Performance metrics based on "get system performance status" command
    on Fortinet firewall
type: monitoring
monitoring_interval: 1 minute
includes_resource_data: true
requires:
    vendor: fortinet
    os.name: FortiOS
    product: firewall
comments:
    memory-usage:
        why: |
            If the firewall memory becomes fully utilized, performance may be impacted and traffic may be dropped, and in extreme cases the firewall could crash. It is critical to monitor the memory usage and handle the issue prior to resource exhaustion.
        how: |
            Indeni uses the built-in Fortinet "get system performance status" command to retrieve the device memory utilization.
        without-indeni: |
            An administrator could login and manually run the command via CLI, check the system resources widget via the GUI, enable SNMP, configure a syslog server for a log message every 5 minutes containing the utilization, or use Fortinet FortiAnalyzer.
        can-with-snmp: true
        can-with-syslog: true
    cpu-usage:
        why: |
            If the firewall CPU becomes fully utilized, performance may be impacted and traffic may be dropped, and in extreme cases the firewall could crash. It is critical to monitor the memory usage and handle the issue prior to resource exhaustion.
        how: |
            Indeni uses the built-in Fortinet "get system performance status" command to retrieve the device CPU utilization.
        without-indeni: |
            An administrator could login and manually run the command via CLI, check the system resources widget via the GUI, enable SNMP, configure a syslog server for a log message every 5 minutes containing the utilization, or use Fortinet FortiAnalyzer.
        can-with-snmp: true
        can-with-syslog: true
    uptime-milliseconds:
        why: |
            Capture the uptime of the device. If the uptime is lower than the previous sample, the device must have reloaded.
        how: |
            Indeni uses the built-in Fortinet "get system performance status" command to retrieve the current device up-time.
        without-indeni: |
            An administrator could login and manually run the command via CLI, check the system resources widget via the GUI, enable SNMP, or use Fortinet FortiAnalyzer.
        can-with-snmp: true
        can-with-syslog: false
    memory-free-kbytes:
        skip-documentation: true
    memory-total-kbytes:
        skip-documentation: true
    memory-used-kbytes:
        skip-documentation: true
steps:
-   run:
        type: SSH
        command: get system performance status
    parse:
        type: AWK
        file: get_system_performance_status.parser.1.awk

cross_vendor_uptime_low

// Deprecation warning : Scala template-based rules are deprecated. Please use YAML format rules instead.

package com.indeni.server.rules.library.templatebased.crossvendor

import com.indeni.apidata.time.TimeSpan
import com.indeni.apidata.time.TimeSpan.TimePeriod
import com.indeni.server.common.data.conditions.Equals
import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library.templates.TimeIntervalThresholdOnDoubleMetricTemplateRule
import com.indeni.server.sensor.models.managementprocess.alerts.dto.AlertSeverity
import com.indeni.server.rules.ThresholdDirection
import com.indeni.server.rules.RemediationStepCondition

case class cross_vendor_uptime_low() extends TimeIntervalThresholdOnDoubleMetricTemplateRule(
  ruleName = "cross_vendor_uptime_low",
  ruleFriendlyName = "All Devices (Non-VSX): Device restarted (uptime low)",
  ruleDescription = "Indeni will alert when a device has restarted.",
  severity = AlertSeverity.CRITICAL,
  metricName = "uptime-milliseconds",
  threshold = TimeSpan.fromMinutes(60),
  metricUnits = TimePeriod.MILLISECOND,
  thresholdDirection = ThresholdDirection.BELOW,
  alertDescriptionFormat = "The current uptime is %.0f seconds which seems to indicate the device has restarted.",
  alertDescriptionValueUnits = TimePeriod.SECOND,
  baseRemediationText = "Determine why the device was restarted.",
  metaCondition = !Equals("vsx", "true")
)(
  RemediationStepCondition.VENDOR_CISCO ->
    """|
       |1. Use the "show version" or "show system reset-reason" NX-OS commands to display the reason for the reload.
       |2. Use the "show cores" command to determine if a core file was recorded during the unexpected reboot.
       |3. Run the "show process log" command to display the processes and if a core was created.
       |4. With the show logging command, review the events that happened close to the time of reboot.""".stripMargin,
  RemediationStepCondition.VENDOR_FORTINET ->
    """
      |1. Watch the system reboot time.
      |2. Review the log messages and focus on error messages that were generated at least 5 minutes prior to system reboot, especially before unexpected system reboot.
      |3. Verify the status of the scheduled restart command to making  sure it's an irregular restart
      |   - config sys global
      |   - get | grep restart
      |   - end
      |4. Login via ssh to the Fortinet firewall and review the crash log in a readable format by using the FortiOS command “diag debug crashlog read”.
      |5. Contact Fortinet Technical support at https://support.fortinet.com/ for further assistance.""".stripMargin
)