Log disk utilization is high-fortinet-FortiOS

error
health-checks
best-practices
fortios
fortinet
Log disk utilization is high-fortinet-FortiOS
0

#1

Log disk utilization is high-fortinet-FortiOS

Vendor: fortinet

OS: FortiOS

Description:
Indeni will alert when the log disk utilization on Fortinet devices is high.

Remediation Steps:

  1. Login via https to the Fortinet firewall and then go to the menu “Log & Report” to review the Local Disk utilization pie and Historical Disk Usage graph.
    |2. Login via ssh to the Fortinet firewall and run the FortiOS command “diagnose sys logdisk usage” to review the HD usage and the HD logging space per VDOM.
    |3. If the disk is almost full, transfer the logs or data off the disk to free up space. When a disk is almost full it consumes a lot of resources to find the free space and organize the files. Clean all unused files routinely.
    |4. Remove any debug files after debugging is done.
    |5. If the FortiGate unit has a hard disk, it is enabled by default to store logs. Consider storing logs to Syslog, FortiAnalyzer or FortiCloud instead of memory or hard disk. Logging to local disk will impact overall performance and reduce the lifetime of the unit. Fortinet recommends logging to feature rich FortiCloud or FortiAnalyzer which don’t use much CPU resources.
    |6. Consider enabling the email alert FortiOS feature if the disk usage exceeds 75%. To achieve this login via https to the Fortinet firewall and then go to the menu “Log & Report” to enable the Email Alert Settings. Then choose the “Disk usage exceeds” tab. More details can be found at http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-system-administration-54/Monitoring/Alert%20email.htm
    |7. If the FortiGate unit has only flash memory, disk logging is disabled by default, and it is recommended to keep this default setting. Constant rewrites to flash drives can reduce the lifetime and efficiency of the memory.
    |8. Both logging and WAN Optimization can use hard disk space to save data. On the FortiGate, go to System > Advanced > Disk Settings to switch between Local Log and WAN Optimization. More details can be found at http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-WAN-opt-54/wan_op_intro.htm
    |9. Contact Fortinet Technical support at https://support.fortinet.com/ for further assistance.

How does this work?
This script logs into the FortiGate using SSH and retrieves the local disk information using the output of the “diag sys logdisk usage” command. The ‘diag sys logdisk usage’ command provides detailed information about how much space is currently available to the device.

Why is this important?
This metric is used to identify when a high percentage of the local disk is being utilized. It should be noticed that the FortiGate unit uses only 75 percent of the available disk capacity to avoid a high storage amount, so when there is a high percentage, it refers to the percentage of the 75 percent that is available. For example, 92 percent of the 75 percent is available. Check the link below for more information “FortiGate advanced logging techniques”: http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-logging-reporting-54/config-log-advanced.htm

Without Indeni how would you find this?
An admin would need to log into the Fortinet firewall and manually review the HD utilization. It is possible to use SNMP traps to notify the administrator when disk space usage exceeds a threshold value e.g. 80%. Besides a log message can be generated in case of high disk utilization.

fortios-diagnose-sys-logdisk-usage

#! META
name: fortios-diagnose-sys-logdisk-usage
description: FortiGate Diagnose disk utilization
type: monitoring
monitoring_interval: 20 minutes
requires:
    vendor: fortinet
    os.name: FortiOS
    product: firewall
    vdom_enabled: false
    vdom_root: true

# --------------------------------------------------------------------------------------------------
# The script publish the following metrics
#
# [fortios-hd-usage]           [0-100 hd-usage fraction ]
# [fortios-hd-log-space]       [string, the logging space]
# [fortios-hd-log-usage-vdom]  [0-100 hd-usage fraction for vdom. Note we can have many metrics with different tags]
# --------------------------------------------------------------------------------------------------


#! COMMENTS
fortios-hd-usage:
    why: |
        This metric is used to identify when a high percentage of the local disk is being utilized. It should be
        noticed that the FortiGate unit uses only 75 percent of the available disk capacity
        to avoid a high storage amount, so when there is a high percentage, it refers to the percentage of
        the 75 percent that is available. For example, 92 percent of the 75 percent is available.
        Check the link below for more information "FortiGate advanced logging techniques":
        http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-logging-reporting-54/config-log-advanced.htm
    how: |
        This script logs into the FortiGate using SSH and retrieves the local disk information using the output of
        the "diag sys logdisk usage" command. The 'diag sys logdisk usage' command
        provides detailed information about how much space is currently available to the device.
    without-indeni: |
        An admin would need to log into the Fortinet firewall and manually review the HD utilization. It is possible
        to use SNMP traps to notify the administrator when disk space usage exceeds
        a threshold value e.g. 80%. Besides a log message can be generated in case of high disk utilization.
    can-with-snmp: true
    can-with-syslog: true

fortios-hd-log-space:
    why: |
        This metric is needed to identify high local disk utilization for the disk partition used for logging.
        Check the link below for more information "FortiGate advanced logging techniques":
        http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-logging-reporting-54/config-log-advanced.htm
    how: |
        This script logs into the FortiGate using SSH and retrieves the local disk information using the output of
        the "diag sys logdisk usage" command. The 'diag sys logdisk usage' command provides detailed information
        about how much space is currently being used for logs.
    without-indeni: |
        An admin would need to log into the Fortinet firewall and manually review the HD available log space. It is
        possible to SNMP traps to notify the administrator when disk space usage exceeds a threshold value e.g. 80%.
        Besides a log message can be generated in case of high disk utilization.
    can-with-snmp: true
    can-with-syslog: true

fortios-hd-log-usage-vdom:
    why: |
       This metric is needed to identify high local disk utilization for the disk partition used for logging. The
       logging space usage is provided per VDOM. Check the link below for more information "FortiGate advanced logging
       techniques":
       http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-logging-reporting-54/config-log-advanced.htm
    how: |
        This script logs into the FortiGate using SSH and retrieves the local disk information using the output of the
        "diag sys logdisk usage" command. The 'diag sys logdisk usage' command provides detailed information about how
        much space is currently being used for logs per VDOM.
    without-indeni: |
        An admin would need to log into the Fortinet firewall and manually review the HD utilzation per VDOM. It is
        possible to use SNMP traps to notify the administrator when disk space usage exceeds a threshold value
        e.g. 80%. Besides a log message can be generated in case of high disk utilization.
    can-with-snmp: true
    can-with-syslog: true


#! REMOTE::SSH
diagnose sys logdisk usage

#! PARSER::AWK

# ------------------------------------------------------------------------------
# Helper function.
# Get as parameter a string fraction ex. '200MB/400MB' and return the fraction result.
# For the input '200MB/400MB' the result would be '50'
# ------------------------------------------------------------------------------
function getTheDivisionResultInPercentage(strFraction) {

    # Read the fraction 262MB/31509MB
    split(strFraction, fraction_array, "/")

    # Remove from fraction the "MB" in order to make the division
    # Convert string from "262MB" to "262"
    gsub(/[a-zA-Z]+/, "", fraction_array[1])
    gsub(/[a-zA-Z]+/, "", fraction_array[2])

    # Store the division of the fraction
    fraction_result = fraction_array[1] / fraction_array[2]
    return fraction_result * 100
}


#-----------------------------------------------------------------------
# Helper function.
# Split the input string using ':' as a delimiter and return second part.
# For example for input 'packets:5' the result is '5'
#-----------------------------------------------------------------------
function getSecondPart(stringWithDelim) {
    split(stringWithDelim, stringArray, ":")
    return stringArray[2]
}

# Initialize the variables
BEGIN {

    # The index of table
    index_vdom = 0

    # Initialize variables
    fortigate_hd_usage = 0
    fortigate_hd_log_space = ""

}

#Total HD usage: 262MB/31509MB
#Total HD usage: 262MB/31509 MB
/^Total HD usage:/{

    secondPart = getSecondPart($0)

    # Remove spaces (in case of ver.5.4 it could be "262MB/31509 MB")
    gsub(" ","",secondPart)
    fortigate_hd_usage = getTheDivisionResultInPercentage(secondPart)

}

#Total HD logging space: 23631MB
#Total HD logging space: 23631 MB
/^Total HD logging space:/{

    secondPart = getSecondPart($0)

    # Remove spaces (in case of ver.5.4 it could be "262 MB")
    gsub(" ","",secondPart)
    fortigate_hd_log_space = secondPart

}

#HD logging space usage for vdom "root": 18MB/11815MB
#HD logging space usage for vdom "root": 18MB/11815 MB
/^HD logging space usage for vdom/{

    # Increase the array size
    index_vdom++

    # split the line [HD logging space usage for vdom "root": 18MB/11815 MB] using ':' as delimiter
    # so the first part is [HD logging space usage for vdom "root"]
    # and the second part is [18MB/11815 MB]
    split($0, stringArray, ":")
    stringPartOne = stringArray[1]
    stringPartTwo = stringArray[2]

    # Get the name of the vdom ("root:")
    lengthOfArray = split(stringPartOne, arrayName, " ")
    name = arrayName[lengthOfArray]

    # Remove from name ("root:") the characters '"' and ':'
    gsub(/"|:/, "", name)

    # Store the data
    array_vdom[index_vdom, "name"] = name
    array_vdom[index_vdom, "usage"] = getTheDivisionResultInPercentage(stringPartTwo)
}


END {

    # Publishing metric in category "Hard Disk Usage"
    tags["name"] = "Fortinet Hard Disk Usage"
    writeDoubleMetricWithLiveConfig("fortios-hd-usage", tags, "gauge", 60, fortigate_hd_usage, "Hard Disk Usage", "percentage", "name")

    # Publishing metrics in "Default/Overview" category
    writeComplexMetricStringWithLiveConfig("fortios-hd-log-space", null,  fortigate_hd_log_space, "Fortinet Hard Disk Log Space")

    for(i = 1; i <= index_vdom; i++){
       tags_to_publish["name"] = array_vdom[i, "name"]
       writeDoubleMetricWithLiveConfig("fortios-hd-log-usage-vdom", tags_to_publish , "gauge", 60, array_vdom[i, "usage"], "Fortinet Hard Disk Usage VDOM: " array_vdom[i, "name"] , "percentage", "")
    }

}




FortinetLogDiskUsage

package com.indeni.server.rules.library.templatebased.fortinet

import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library.templates.NumericThresholdOnDoubleMetricWithItemsTemplateRule
case class FortinetLogDiskUsage() extends NumericThresholdOnDoubleMetricWithItemsTemplateRule(
    ruleName = "FortinetLogDiskUsage",
    ruleFriendlyName = "Fortinet Devices: Log disk utilization is high",
    ruleDescription = "Indeni will alert when the log disk utilization on Fortinet devices is high.",
    metricName = "fortios-hd-usage",
    threshold = 80.0,
    applicableMetricTag = "name",
    alertItemsHeader = "Affected Disks",
    alertItemDescriptionFormat = "The current disk log usage is %.0f%%",
    alertDescription = "Log disk utilization is high.",
    baseRemediationText =
       """1. Login via https to the Fortinet firewall and then go to the menu "Log & Report" to review the Local Disk utilization pie and Historical Disk Usage graph.
         |2. Login via ssh to the Fortinet firewall and run the FortiOS command “diagnose sys logdisk usage” to review the HD usage and the HD logging space per VDOM.
         |3. If the disk is almost full, transfer the logs or data off the disk to free up space. When a disk is almost full it consumes a lot of resources to find the free space and organize the files. Clean all unused files routinely.
         |4. Remove any debug files after debugging is done.
         |5. If the FortiGate unit has a hard disk, it is enabled by default to store logs. Consider storing logs to Syslog, FortiAnalyzer or FortiCloud instead of memory or hard disk. Logging to local disk will impact overall performance and reduce the lifetime of the unit. Fortinet recommends logging to feature rich FortiCloud or FortiAnalyzer which don’t use much CPU resources.
         |6. Consider enabling the email alert FortiOS feature if the disk usage exceeds 75%. To achieve this login via https to the Fortinet firewall and then go to the menu “Log & Report” to enable the Email Alert Settings. Then choose the “Disk usage exceeds” tab. More details can be found at http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-system-administration-54/Monitoring/Alert%20email.htm
         |7. If the FortiGate unit has only flash memory, disk logging is disabled by default, and it is recommended to keep this default setting. Constant rewrites to flash drives can reduce the lifetime and efficiency of the memory. 
         |8. Both logging and WAN Optimization can use hard disk space to save data. On the FortiGate, go to System > Advanced > Disk Settings to switch between Local Log and WAN Optimization. More details can be found at http://help.fortinet.com/fos50hlp/54/Content/FortiOS/fortigate-WAN-opt-54/wan_op_intro.htm
         |9. Contact Fortinet Technical support at https://support.fortinet.com/ for further assistance.""".stripMargin
    )()