Critical process(es) down-fortinet-FortiOS

Critical process(es) down-fortinet-FortiOS
0

Critical process(es) down-fortinet-FortiOS

Vendor: fortinet

OS: FortiOS

Description:
Many devices have critical processes, usually daemons, that must be up for certain functions to work. Indeni will alert if any of these goes down.

Remediation Steps:
Review the cause for the processes being down.

How does this work?
This script logs into the Fortinet firewall through SSH and retrieves the status of running processes by running the FortiOS command fnsysctl ps. The script then compares the list of currently running processes to a known list of critical processes and checks to see that they are all up. If any are down or in an abnormal state, Indeni raises an alert.

Why is this important?
Each device has certain executable processes which are critical to stable operation. For example, on Fortinet devices, the authd process handles user authentication, and the scanunitd process handles AnitVirus protection. There are many others. If a critical process is down, this may indicate a critical failure.

Without Indeni how would you find this?
An administrator could manually login and retrieve/parse the data, or could write a script to poll the firewalls and parse the returned data.

fortios-fnsysctl-ps

name: fortios-fnsysctl-ps
description: get a list of critical processes and check to see if they are operational
type: monitoring
monitoring_interval: 10 minutes
requires:
    vendor: fortinet
    os.name: FortiOS
    product: firewall
comments:
    process-state:
        why: |
            Each device has certain executable processes which are critical to stable operation. For example, on Fortinet devices, the authd process handles user authentication, and the scanunitd process handles AnitVirus protection. There are many others. If a critical process is down, this may indicate a critical failure.
        how: |
            This script logs into the Fortinet firewall through SSH and retrieves the status of running processes by running the FortiOS command fnsysctl ps. The script then compares the list of currently running processes to a known list of critical processes and checks to see that they are all up. If any are down or in an abnormal state, Indeni raises an alert.
        without-indeni: |
            An administrator could manually login and retrieve/parse the data, or could write a script to poll the firewalls and parse the returned data.
        can-with-snmp: false
        can-with-syslog: false
steps:
-   run:
        type: SSH
        command: fnsysctl ps
    parse:
        type: AWK
        file: fnsysctl_ps.parser.1.awk

cross_vendor_critical_process_down_novsx

// Deprecation warning : Scala template-based rules are deprecated. Please use YAML format rules instead.

package com.indeni.server.rules.library.templatebased.crossvendor

import com.indeni.server.common.data.conditions.Equals
import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library.templates.StateDownTemplateRule
import com.indeni.server.rules.RemediationStepCondition

case class cross_vendor_critical_process_down_novsx() extends StateDownTemplateRule(
  ruleName = "cross_vendor_critical_process_down_novsx",
  ruleFriendlyName = "All Devices: Critical process(es) down",
  ruleDescription = "Many devices have critical processes, usually daemons, that must be up for certain functions to work. Indeni will alert if any of these goes down.",
  metricName = "process-state",
  applicableMetricTag = "process-name",
  descriptionMetricTag = "description",
  alertItemsHeader = "Processes Affected",
  descriptionStringFormat = "${scope(\"description\")}",
  alertDescription = "One or more processes which are critical to the operation of this device, are down.",
  baseRemediationText = "Review the cause for the processes being down.",
  metaCondition = !Equals("vsx", "true"))(
  RemediationStepCondition.VENDOR_CP -> "Check if \"cpstop\" was run.",
  RemediationStepCondition.VENDOR_CISCO ->
    """|
      |1. Use the "show processes cpu" NX-OS command in order to show the CPU usage at the process level.
      |2. Use the "show process cpu detail <pid> " NX-OS command to find out the CPU usage for all threads that belong to a specific process ID (PID).
      |3. Use the "show system internal sysmgr service pid <pid> " NX-OS command in order to display additional details, such as restart time, crash status, and current state, on the process/service by PID.
      |4. Run the "show system internal processes cpu" NX-OS command which is equivalent to the top command in Linux and provides an ongoing look at processor activity in real time""".stripMargin,
  RemediationStepCondition.VENDOR_FORTINET ->
    """
      |1. Login via ssh to the Fortinet firewall and run the FortiOS command "diagnose sys top [refresh_time_sec] [number_of_lines]"
        |>>> to get the Proccess-id, State, CPU & Memory utilization per process. Press <shift-P> to sort by CPU usage or <shift-M> to sort by memory usage.
      |2. Login via ssh to the Fortinet firewall and run the FortiOS command "diagnose sys top-summary '-h' " to get the command options and receive additional
        |>>> info per process. A sample command could be "diagnose sys top-summary '-s mem -i 60 -n 10' ". In case that the value to the FDS (File Descriptors)
        |>>> column keeps constantly increasing, it might indicate a memory leak problem.
      |3. Review the state of each process provided by the above commands. The normal states are S (Sleeping), R (Running) and D (Do not Disturb).
        |>>> The abnormal states are Z (Zombie) and D (Do not Disturb).
      |4. Try to restart the process which has problem by running the command "diag sys kill 11 <process-Id>". The <process-Id> can be found by the aforementioned commands.
      |5. Check the logs for any reasons why the process stops or can't restart.
      |6. If the problem persists, contact Fortinet Technical support at https://support.fortinet.com/ for further assistance.""".stripMargin.replaceAll("\n>>>", "")

)