NTP sync failure(s)-fortinet-FortiOS

warn
health-checks
fortios
fortinet
NTP sync failure(s)-fortinet-FortiOS
0

#1

NTP sync failure(s)-fortinet-FortiOS

Vendor: fortinet

OS: FortiOS

Description:
Indeni will alert if one or more of the configured NTP servers is not syncing correctly.

Remediation Steps:
Review the cause for the NTP sync not working.

  |1. Login via ssh to the Fortinet firewall and execute the FortiOS “execute time” and “execute date” commands to check the current date/time and the last date of NTP sync.
  |2. Login via ssh to the Fortinet firewall and execute the FortiOS “diagnose sys ntp status” to review the status of the NTP servers and configuration.
  |3. NTP uses UDP protocol (17) and port 123 to communicate between the client and the servers.  Make sure that the firewall rules allow these UDP ports and can route toward the NTP servers.
  |4. Login via ssh to the Fortinet firewall and execute the FortiOS debug commands “diag debug application ntpd -1” and “diag debug enable” and review the debug messages.
  |5. Make sure NTP authentication keys match on both ends. Review the next link for more information http://kb.fortinet.com/kb/viewContent.do?externalId=FD33783.
  |6. More NTP configuration information can be found at link http://help.fortinet.com/cli/fos50hlp/56/Content/FortiOS/fortiOS-cli-ref-56/config/system/ntp.htm.

How does this work?
This script logs into the FortiGate using SSH and retrieves the NTP servers readhabilty status information using the output of the “diagnose sys ntp status” command. The output includes the device’s server reachability status as well as information about the NTP configured parameters.

Why is this important?
This metric shows if the NTP servers used by the FortiGate are reachable. NTP servers are used to sync the time across all hosts and network devices. This is critical for things such as event correlation and logging. Use Network Time Protocol (NTP) to set the date and time if possible. However, it is important to ensure the NTP UDP port is allowed through the firewalls on your network. FortiToken synchronization requires NTP in many situations. Check the link below for more information about NTP config in FortiOS: http://kb.fortinet.com/kb/documentLink.do?externalID=FD40266

Without Indeni how would you find this?
An administrator would need to log into the device and use the “diagnose sys ntp status” command to identify if the NTP servers are reachable from the device.

fortios-diagnose-sys-ntp-status

#! META
name: fortios-diagnose-sys-ntp-status
description: FortiGate Diagnose ntp status
type: monitoring
monitoring_interval: 20 minutes
requires:
    vendor: fortinet
    os.name: FortiOS
    product: firewall
    vdom_enabled: false
    vdom_root: true

# --------------------------------------------------------------------------------------------------
# Tested with FortiOS versions v5.4.6,build1165 & v5.6.0,build1449
#
# Script Information:
# The script publish the following metrics
#
# [ntp-is-synchronized]         [1 or 0, 1 in case of 'yes']
# [ntp-sync-is-enabled]         [1 or 0, 1 in case of 'enabled']
# [ntp-server-mode]             [disabled, (copy text)]
# [ntp-server-state]            [1 or 0, for each ntp-server, 0 if ipv4/ipv6 server is unreachable]
# [ntp-servers]                 [array of the ntp-servers with 'ipaddress', 'is_reachable' and 'name' tags]
#
# --------------------------------------------------------------------------------------------------


#! COMMENTS
ntp-is-synchronized:
    why: |
        It checks if the device is sync via NTP. NTP servers are used to sync the time across all hosts and network
        devices. This is critical for things such as event correlation and logging. Use Network Time Protocol (NTP) to
        set the date and time if possible.  However, it is important to ensure the NTP UDP port is allowed through the
        firewalls on your network. FortiToken synchronization requires NTP in many situations. Check the link below for
        more information about NTP config in FortiOS:
        http://kb.fortinet.com/kb/documentLink.do?externalID=FD40266
    how: |
        This script logs into the FortiGate using SSH and retrieves the NTP sync status information using the output of
        the "diagnose sys ntp status" command. The output includes the device's ntp sync status as well as information
        about all the NTP configured parameters.
    without-indeni: |
       An administrator would need to log into the device and use the "diagnose sys ntp status" command to identify if
       NTP is synchronized.
    can-with-snmp: false
    can-with-syslog: false

ntp-sync-is-enabled:
    why: |
        This metric monitors if the use of NTP is enabled. NTP servers are used to sync the time across all hosts and
        network devices. This is critical for things such as event correlation and logging. Use Network Time Protocol
        (NTP) to set the date and time if possible. However, it is important to ensure the NTP UDP port is allowed
        through the firewalls on your network. FortiToken synchronization requires NTP in many situations. Check the
        link below for more information about NTP config in FortiOS:
        http://kb.fortinet.com/kb/documentLink.do?externalID=FD40266
    how: |
        This script logs into the FortiGate using SSH and retrieves the NTP sync status information using the output of
        the "diagnose sys ntp status" command. The output includes the device's ntp sync status as well as information
        about the NTP configured parameters.
    without-indeni: |
       An administrator would need to log into the device and use the "diagnose sys ntp status" command to identify if
       the NTP is enabled.
    can-with-snmp: false
    can-with-syslog: false

ntp-server-mode:
    why: |
        This metric shows if the device is used as NTP server. This is applicable starting with FortiOS 5.0. NTP
        servers are used to sync the time across all hosts and network devices. This is critical for things such as
        event correlation and logging. Use Network Time Protocol (NTP) to set the date and time if possible.  However,
        it is important to ensure the NTP UDP port is allowed through the firewalls on your network. FortiToken
        synchronization requires NTP in many situations. Check the link below for more information about NTP config in
        FortiOS: http://kb.fortinet.com/kb/documentLink.do?externalID=FD40266
    how: |
        This script logs into the FortiGate using SSH and retrieves the NTP server status information using the output
        of the "diagnose sys ntp status" command. The output includes the device's server status as well as information
        about the NTP configured parameters.
    without-indeni: |
        An administrator would need to log into the device and use the "diagnose sys ntp status" command to identify if
        the device is configured as NTP server.
    can-with-snmp: false
    can-with-syslog: false

ntp-server-state:
    why: |
        This metric shows if the NTP servers used by the FortiGate are reachable. NTP servers are used to sync the
        time across all hosts and network devices. This is critical for things such as event correlation and logging.
        Use Network Time Protocol (NTP) to set the date and time if possible.  However, it is important to ensure the
        NTP UDP port is allowed through the firewalls on your network. FortiToken synchronization requires NTP in many
        situations. Check the link below for more information about NTP config in FortiOS:
        http://kb.fortinet.com/kb/documentLink.do?externalID=FD40266
    how: |
        This script logs into the FortiGate using SSH and retrieves the NTP servers readhabilty status information
        using the output of the "diagnose sys ntp status" command. The output includes the device's server reachability
        status as well as information about the NTP  configured  parameters.
    without-indeni: |
        An administrator would need to log into the device and use the "diagnose sys ntp status" command to identify if
        the NTP servers are reachable from the device.
    can-with-snmp: false
    can-with-syslog: false

ntp-servers:
    why: |
        This metric shows if at least a NTP server is configured. NTP servers are used to sync the time across all
        hosts and network devices. This is critical for things such as event correlation and logging. Use Network Time
        Protocol (NTP) to set the date and time if possible. However, it is important to ensure the NTP UDP port is
        allowed through the firewalls on your network. FortiToken synchronization requires NTP in many situations.
        Check the link below for more information about NTP config in FortiOS:
        http://kb.fortinet.com/kb/documentLink.do?externalID=FD40266
    how: |
        This script logs into the FortiGate using SSH and retrieves the NTP servers configuration status information
        using the output of the "diagnose sys ntp status" command. The output includes the device's servers
        configuration status as well as information about the NTP configured  parameters.
    without-indeni: |
        An administrator would need to log into the device and use the "diagnose sys ntp status" command to identify if
        the NTP servers are reachable from the device.
    can-with-snmp: false
    can-with-syslog: false


#! REMOTE::SSH
diagnose sys ntp status

#! PARSER::AWK

# Initialize the variables
BEGIN {

    # Initialize the values of metrics to publish
    ntp_is_synchronized = 0
    ntp_sync_is_enabled = 0
    ntp_server_mode = ""

    # By default ntp_server_is_configured is 0
    ntp_server_is_configured = 0

    # By default is_all_reachable is 1. (Note that if ntp is not configured we will publish the is_all_reachable to 0)
    ntp_servers_are_all_reachable = 1

    ntp_table_index = 0

}

#synchronized: yes, ntpsync: enabled, server-mode: disabled
/^synchronized:/ {

    # Check if synchronized is 'yes'
    # The ntp_is_synchronized is initialized to 0
    if (trim($2) == "yes,") {
        ntp_is_synchronized = 1
    }


    # Check if ntpsync_status is 'enabled'
    # The ntp_sync_is_enabled is initialized to 0
    ntpsync_status = tolower(trim($4))
    if (ntpsync_status == "enabled,") {
        ntp_sync_is_enabled = 1
    }

    # Store 'server-mode:' value
    ntp_server_mode = $6

}

#ipv4 server(10.10.8.145) 10.10.8.145 -- reachable(0xff) S:0 T:0
/^ipv4 server/ || /^ipv6 server/{

    # Fortinet ntp-server is configured
    ntp_table_index++

    # Set 'ipaddress'
    ntp_table[ntp_table_index, "ipaddress"] = $3

    # Read server name for example "server(10.10.8.145)"
    ntp_server_name = $2

    # Extract from string "server(10.10.8.145)" the text inside the parenthesis
    # Avoid regex and use 'substring' (using the length of 'server(' and last character ')')
    ntp_server_name = substr(ntp_server_name, 8, length(ntp_server_name)-8)

    # Set 'name'
    ntp_table[ntp_table_index, "name"] = ntp_server_name

    # Check if ntp server is reachable
    is_reachable = 1
    if ($0 ~ / unreachable/) {
        ntp_servers_are_all_reachable = 0
        is_reachable = 0
    }

    ntp_table[ntp_table_index, "is_reachable"] = is_reachable

}

END {

    # Publishing metrics
    writeDoubleMetricWithLiveConfig("ntp-is-synchronized", null, "gauge", 60, ntp_is_synchronized, "NTP Synchronization Status", "state", "")
    writeDoubleMetricWithLiveConfig("ntp-sync-is-enabled", null, "gauge", 60, ntp_sync_is_enabled, "NTP Sync Enabled", "state", "")
    writeComplexMetricStringWithLiveConfig("ntp-server-mode", null,  ntp_server_mode, "NTP Server")
    writeComplexMetricObjectArray("ntp-servers", null, ntp_table)

    # For each ntp-server publish 'ntp-server-state'
    for(table_index=1; table_index < (ntp_table_index+1); table_index++ ){
        tags["name"] = ntp_table[table_index, "ipaddress"]
        writeDoubleMetricWithLiveConfig("ntp-server-state", tags, "gauge", 60, ntp_table[table_index, "is_reachable"], "NTP Servers Status", "state", "name")
    }


}




all_devices_ntp_not_syncing

package com.indeni.server.rules.library.templatebased.crossvendor

import com.indeni.server.rules.RuleContext
import com.indeni.server.rules.library.ConditionalRemediationSteps
import com.indeni.server.rules.library.templates.StateDownTemplateRule
import com.indeni.server.sensor.models.managementprocess.alerts.dto.AlertSeverity

/**
  *
  */
case class all_devices_ntp_not_syncing() extends StateDownTemplateRule(
  ruleName = "all_devices_ntp_not_syncing",
  ruleFriendlyName = "All Devices: NTP sync failure(s)",
  ruleDescription = "Indeni will alert if one or more of the configured NTP servers is not syncing correctly.",
  severity = AlertSeverity.WARN,
  metricName = "ntp-server-state",
  applicableMetricTag = "name",
  alertItemsHeader = "NTP Servers Affected",
  alertDescription = "One or more NTP servers configured on this device is not responding.",
  historyLength = 2,
  baseRemediationText = "Review the cause for the NTP sync not working.")(
  ConditionalRemediationSteps.VENDOR_CP -> "Review sk92602: https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk92602",
  ConditionalRemediationSteps.VENDOR_PANOS -> "Run \"show ntp\" and review the status of each NTP server. You can also review the dagger.log, based on https://live.paloaltonetworks.com/t5/tkb/articleprintpage/tkb-id/Management-TKB/article-id/2078",
  ConditionalRemediationSteps.OS_NXOS ->
    """|
      |Examples of common NTP issues are the next:
      |• NTP packets are not received.
      |• NTP packets are received, but are not processed by the NTP process on the NX-OS.
      |• NTP packets are processed, but erroneous factors or packet data causes the loss of synchronization.
      |• NTP clock-period is manually set.
      |
      |1. Check the current NTP status by running the NX-OS command "show ntp peer-status".
      |2. If the "show ntp peer-status" command does not provide any output then try to ping the NTP servers. The NTP source and vrf may need to be provided as command options.
      |3. Check the routing table with the "show ip route vrf all" NX-OS command to verify that there is routing to the NTP servers.
      |4. Check that the UDP 123 port used by NTP service is permitted to the network.
      |5. Execute the "show run ntp" NX-OS command to review the NTP current configuration.
      |6. For more information review the next Nexus NTP troubleshooting guide: https://www.cisco.com/c/en/us/support/docs/ip/network-time-protocol-ntp/116161-trouble-ntp-00.html""".stripMargin,
  ConditionalRemediationSteps.VENDOR_FORTINET ->
    """
      |1. Login via ssh to the Fortinet firewall and execute the FortiOS “execute time” and “execute date” commands to check the current date/time and the last date of NTP sync.
      |2. Login via ssh to the Fortinet firewall and execute the FortiOS “diagnose sys ntp status” to review the status of the NTP servers and configuration.
      |3. NTP uses UDP protocol (17) and port 123 to communicate between the client and the servers.  Make sure that the firewall rules allow these UDP ports and can route toward the NTP servers.
      |4. Login via ssh to the Fortinet firewall and execute the FortiOS debug commands “diag debug application ntpd -1” and “diag debug enable” and review the debug messages.
      |5. Make sure NTP authentication keys match on both ends. Review the next link for more information http://kb.fortinet.com/kb/viewContent.do?externalId=FD33783.
      |6. More NTP configuration information can be found at link http://help.fortinet.com/cli/fos50hlp/56/Content/FortiOS/fortiOS-cli-ref-56/config/system/ntp.htm.""".stripMargin
)