Next hop inaccessible-fortinet-FortiOS

error
health-checks
fortios
fortinet
Next hop inaccessible-fortinet-FortiOS
0

#1

Next hop inaccessible-fortinet-FortiOS

Vendor: fortinet

OS: FortiOS

Description:
Indeni will review the routing table and identify when a next hop router is showing as FAILED or INCOMPLETE in the ARP table.

Remediation Steps:
Determine why the next hops are not responding.

How does this work?
This alert uses the FortiOS command “get system arp” to retrieve the full ARP table for a FortiOS firewall.

Why is this important?
Tracking the ARP entry can indicate when certain hosts are failing to repsond to ARP requests. If that host is actually a next hop router, traffic may not reach its final destination. In addition, if there’s a sudden jump in the number of ARP entries that are failing, it may indicate a connectivity issue at layer 2.

Without Indeni how would you find this?
An administrator would have to login to the Fortinet firewall and execute the “get system arp” command to check this data. Alternatively, wait for an issue to occur and check the ARP system status by running the FortiOS command “get system arp”.

fortios-get-system-arp

#! META
name: fortios-get-system-arp
description: Fortinet Firewall system ARP entries
type: monitoring
monitoring_interval: 10 minutes
requires:
    vendor: fortinet
    os.name: FortiOS
    product: firewall
    vdom_enabled: false
    vdom_root: true

# --------------------------------------------------------------------------------------------------
# The script publish the following metrics
#
# [arp-table]           [complex array]
# --------------------------------------------------------------------------------------------------


#! COMMENTS
arp-table:
    why: |
        Tracking the ARP entry can indicate when certain hosts are failing to repsond to ARP requests. If that host is actually a next hop router, traffic may not reach its final destination. In addition, if there's a sudden jump in the number of ARP entries that are failing, it may indicate a connectivity issue at layer 2.
    how: |
        This alert uses the FortiOS command "get system arp" to retrieve the full ARP table for a FortiOS firewall.
    without-indeni: |
        An administrator would have to login to the Fortinet firewall and execute the "get system arp" command to check this data. Alternatively, wait for an issue to occur and check the ARP system status by running the FortiOS command "get system arp".
    can-with-snmp: false
    can-with-syslog: false


#! REMOTE::SSH
get system arp

#! PARSER::AWK

BEGIN{
    # Store mask in
    table_arp_index = 0
}

# Parse all the needed info ('targetip', 'mac' & 'interface') and store them in the table
#10.10.8.145       0          88:1d:fc:60:4b:c6 lan
#212.205.216.193   0          88:1d:fc:60:4b:c6 wan1
/^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/{

    # Increase index of table
    table_arp_index++

    table_arp[table_arp_index, "targetip"] = $1
    table_arp[table_arp_index, "mac"] = $3
    table_arp[table_arp_index, "interface"] = $4
    table_arp[table_arp_index, "success"] = 1
}

END {
    writeComplexMetricObjectArray("arp-table", null, table_arp)
}


fortios-get-router-info-routing-table-static

#! META
name: fortios-get-router-info-routing-table-static
description: Fortinet Firewall static route entries 
type: monitoring
monitoring_interval: 10 minutes
requires:
    vendor: fortinet
    os.name: FortiOS
    product: firewall
    vdom_enabled: false
    vdom_root: true

# --------------------------------------------------------------------------------------------------
# The script publish the following metrics
#
# [static-routing-table]           [complex array ]
# --------------------------------------------------------------------------------------------------


#! COMMENTS
static-routing-table:
    why: |
       Capture the static route entries that are configured on the Fortinet Firewall. It is chekced if the next hop IP
       address set to the static routes can be resolved to a MAC address via ARP . If this is not the case, an alert
       would be generated.
    how: |
       This script logins to the Fortinet Firewall and retrieves the output of the "get router info routing-table
       static" command. The output includes a table with the device's configured static routes.
    without-indeni: |
       It is possible to poll this data through SNMP but additional external logic would be required to correlate the
       static routes table with the arp table entries and resolved next hop IP addresses.
    can-with-snmp: true
    can-with-syslog: false


#! REMOTE::SSH
get router info routing-table static

#! PARSER::AWK
BEGIN{
    # Store mask in
    table_routing_index = 0
}

# Parse all the needed info ('network', 'mask' & 'next-hop') and store them in the table
#S*      0.0.0.0/0 [10/0] via 212.205.216.193, wan1
#S       10.0.0.0/8 [10/0] via 10.10.8.145, lan
#S       1.1.1.1/32 [10/0] is directly connected, port1
/^S/{

    # Ensure that the $5 is ip and not text
    if($5 ~ /[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/){

        # Increase index of table
        table_routing_index++

        # Reading network and mask. Example is "10.0.0.0/8"
        ip_mask = $2
        split(ip_mask, ip_mask_array, "/")
        table_routing[table_routing_index, "network"] = ip_mask_array[1]
        table_routing[table_routing_index, "mask"] = ip_mask_array[2]

        # Storing next-hop
        next_hop = $5

        # Removing ','
        gsub(",", "", next_hop)
        table_routing[table_routing_index, "next-hop"] = next_hop

    }
}

END {
    writeComplexMetricObjectArray("static-routing-table", null, table_routing)
}




cross_vendor_next_hop_router_inaccessible

package com.indeni.server.rules.library

import com.indeni.ruleengine.InvisibleScopeKey
import com.indeni.ruleengine.expressions.conditions.Equals
import com.indeni.ruleengine.expressions.core._
import com.indeni.ruleengine.expressions.data._
import com.indeni.ruleengine.expressions.scope.ScopeValueExpression
import com.indeni.server.common.data.conditions.True
import com.indeni.server.rules._
import com.indeni.server.rules.library.core.PerDeviceRule
import com.indeni.server.sensor.models.managementprocess.alerts.dto.AlertSeverity


case class NextHopRouterInaccessibleRule(context: RuleContext) extends PerDeviceRule with RuleHelper {

  override val metadata: RuleMetadata = RuleMetadata.builder("cross_vendor_next_hop_router_inaccessible", "All Devices: Next hop inaccessible",
    "Indeni will review the routing table and identify when a next hop router is showing as FAILED or INCOMPLETE in the ARP table.", AlertSeverity.ERROR).build()

  override def expressionTree: StatusTreeExpression = {
    StatusTreeExpression(
      // Which objects to pull (normally, devices)
      SelectTagsExpression(context.metaDao, Set(DeviceKey), True),

      // What constitutes an issue
      StatusTreeExpression(
        // The time-series we check the test condition against:
        SelectSnapshotsExpression(context.snapshotsDao, Set("arp-table", "static-routing-table")).multi(),

        // The condition which, if true, we have an issue. Checked against the time-series we've collected
        StatusTreeExpression(

          JoinSnapshotsExpression("arp-table" -> "targetip", "static-routing-table" -> "next-hop")
            .distinct(InvisibleScopeKey("next-hop", "static-routing-table")),

          Equals(
            ScopeValueExpression("success").invisible("arp-table").optional(),
            ConstantExpression(Some("0"))
          )
        ).withSecondaryInfo(
          scopableStringFormatExpression("${scope(\"static-routing-table:next-hop\")}"),
          EMPTY_STRING,
          title = "Inaccessible Next Hops",
          invisibleIdKeys = Set(InvisibleScopeKey("next-hop", "static-routing-table"))
        ).asCondition()
      ).withoutInfo().asCondition()


      // Details of the alert itself
    ).withRootInfo(
      getHeadline(),
      scopableStringFormatExpression("Some of the routes in this device have a next hop which is inaccessible."),
      ConditionalRemediationSteps("Determine why the next hops are not responding.",
        ConditionalRemediationSteps.VENDOR_CP -> "Trying pinging the next hop routers in the list above and resolve any connectivity issues one by one until all pings are successful.",
        ConditionalRemediationSteps.VENDOR_PANOS -> "Log into the device over SSH and review the output of \"show arp\" to identify failures.",
        ConditionalRemediationSteps.OS_NXOS ->
          """|
             |1. Execute the "show spanning-tree" and "show spanning-tree summary"  NX-OS commands to quickly identify the STP root for all the configured vlans.
             |2. Run the "show spanning-tree vlan X detail" NX-OS command to collect more info about the STP topology (X=vlanid).
             |3. Check the event history to find where the Topology Change Notifications originate from by running the next NX-OS command "show spanning-tree internal event-history tree X brief" , (X=vlanid).
             |4. Display the STP events of an interface with the next NX-OS command "show spanning-tree internal event-history tree Y interface X brief" , (X=vlanid, Y=interfaceid).
             |5. Consider to hard code the STP root and backup root to the core switches by configuring a lower STP priority.
             |6. Activate the recommended vPC "peer switch" NX-OS command to a pure peer switch topology in which the devices all belong to the vPC.
             |7. Consider to use Root Guard feature to enforce the root bridge placement in the network. If a received BPDU triggers an STP convergence that makes that designated port become a root port, that port is put into a root-inconsistent (blocked) state.
             |8. For more information please review the following links:
             | <a target="_blank" href="https://www.cisco.com/c/en/us/support/docs/switches/nexus-5000-series-switches/116199-technote-stp-00.html">Spanning Tree Protocol Troubleshooting on a Nexus 5000 Series Switch</a>
             | <a target="_blank" href="https://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-7000-series-switches/C07-572834-00_STDG_NX-OS_vPC_DG.pdf">Spanning Tree Design Guidelines for Cisco NX-OS Software and Virtual PortChannels</a>
          """.stripMargin,
        ConditionalRemediationSteps.VENDOR_BLUECOAT ->
          """ARP resolve failure to the next hop of the ProxySG.
            |1. Login via SSH to the ProxySG and run the  "show arp-table" command.
            |2. Check for incomplete arp enteries.
            |3. Run the "show interface all" command and check the current status of the network interface with the incomplete arp entery.
            |4. Diagnose the layer 2 connectivity between the ProxySG to the other device.
            |5. If the problem persists, contact Symantec Technical support at https://support.symantec.com for further assistance.""".stripMargin,
        ConditionalRemediationSteps.VENDOR_JUNIPER ->
          """|1. Log into the device over SSH and enter “show arp no-resolve” command to review next-hop MAC and IP address information in ARP table.
             |2. Check for a misconfiguration on interfaces or a physical issue.
             |3. Review the following article on Juniper tech support site: <a target="_blank" href="https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/show-arp.html#jd0e289">Operational Commands</a>""".stripMargin
      )
    )
  }
}