Originally published at: How to Prove an Outage is Not Caused By the Firewall | Indeni
We all know it. The Firewall is the scapegoat of the IT infrastructure stack. When something goes awry with the network, the firewall is the first ‘person’ to get the blame. Sometimes the firewall is the culprit, but many times the reason for performance degradation or loss of connectivity is related to another part of the network. As the above comic strip depicts, Firewall Administration is one of the most dreaded positions to have in IT Operations.
As the firewall administrator you are blamed for issues, and therefore branded internally as someone who causes problems or even worse, someone who ‘can’t get the job done.’ The truth of the matter is networking is not as simple as it used to be. Gone are the days of a single vendor environment managing the networking stack. With the growing adoption of virtualization and public cloud platforms, and a layered or multi-vendor approach to network security, there are millions of reasons why network performance can suffer. In addition to keeping up with the new OS versions and features that come out multiple times a year, engineers and operations also need to understand the dependencies between systems. Who has time to be trained on all of these details, and retain it?
As history always repeats itself, there will be a time in the future where your firewall gets blamed for slow network performance or the next outage. Having an unbiased third party involved in the process helps resolve the dispute quickly. Below, we outline the most common issues that firewalls receive the blame for, and how Indeni can ensure you don’t make these mistakes. In addition to notifying you of a problem, Indeni’s historical data will allow you to demonstrate a recent issue is not you, or your firewall’s fault!
Ways to prove an issue is not the firewall:
1. Debug mode was not enabled
One of the symptoms your users may encounter is spotty connectivity. If Debug mode is left enabled, the firewall will cause packets to drop. To avoid this, Indeni validates at a predefined intervals if this setting is on, ensuring the firewall does not cause any issues. Here are some of the ways Indeni checks if this feature is active on different devices:- Palo Alto Networks: Indeni logs into the Palo Alto Networks firewall through SSH and retrieves the status of the debug (on or off).
- Juniper SRX: Indeni identifies which component is enabled with traceoptions by running the command "show configuration | display set | match traceoptions" via SSH connection to the device.
- F5 Networks: Indeni logs into the F5 unit via iControl REST and retrieves the status of the tm.rstcause.pkt flag.
- Cisco Nexus: Indeni logs in to the Cisco Nexus switch using SSH and retrieves the status of running debugs. In normal operation, there should not be any debugs enabled.
2. VPN Tunnels are up
VPN tunnels are of utmost importance to the business. If there is a problem with a tunnel, you can guarantee an engineer will be asked to prioritize this above everything else. For example, if a service is down, remote branches and partners immediately assume that the VPN tunnel has failed. If the tunnel is the only available link, a failure would immediately impact service. To avoid this, Indeni performs the following tasks on firewalls:- Palo Alto Networks: Indeni uses the Palo Alto Networks API to retrieve the current status of the VPN tunnels (the equivalent of running "show vpn flow" in CLI). Indeni differentiates between issues that are not "always on" (don't have a monitor set) and those that should be. Indeni also notifies, the administrator when it identifies an issue. Furthermore, Indeni retrieves the authentication errors for each tunnel.
- Fortinet: Indeni runs the FortiOS command "get ipsec tunnel list" to retrieve IPSec VPN related information.
- Juniper: Indeni runs "show configuration security ike, show configuration security ipsec, show security ipsec inactive-tunnels, show security ipsec security-associations brief" to retrieve IPSec VPN related information.
3. Static Routes align across clusters
When setting up firewall clusters, well meaning experienced engineers are bound to make mistakes. Indeni helps new business projects from being hindered by a misconfiguration. To avoid firewall clusters from having mismatched static routes set up by accident, Indeni logs into firewalls to confirm their routes match:- Palo Alto Networks: Indeni uses the Palo Alto Networks API to retrieve the current routing table (the equivalent of running "show routing route" in CLI).
- Check Point Gaia: By parsing the GAiA configuration database, /config/active, the static routes are retrieved. You can also retrieve this information via Clish, however that creates a hefty amount of log entries in /var/log/messages.
- Linux: Indeni can identify mismatches by running the command "netstat -rn" the routes are retrieved.
- Fortinet: Indeni logs in to the Fortinet Firewall and retrieves the output of the "get router info routing-table static" command. The output includes a table with the device's configured static routes. If this data does not meet best practices, an issue is generated in the Indeni Dashboard or sent to Indeni user via email.