Originally published at: https://indeni.com/blog/palo-alto-networks-pan-os-upgrades-after-a-vulnerability-announcement/
Deployment Trends Case Study
On March 20th, 2019, approximately 1 month before this post, Palo Alto Networks announced two security vulnerabilities, with a “Medium” rating, and provided fixes via upgraded releases of each of the affected PAN-OS that are currently supported. This announcement provides a perfect test for the question:
Which do businesses value more: firewall security or firewall stability?
(Spoiler alert: it’s stability.)
To answer this question, Indeni queried Indeni Insight, our global outage tracking and metrics monitoring system. From the Palo Alto Networks announcement, we know the vulnerability was only present on 64bit systems with more than 32 GB of available memory, which are the M-500, M-600, WF-500, PA-5220, PA-5250, PA-5260 and PA-5280. From that set, we retrieved version upgrade information for a period of 1 month after the announcement.
Extent of vulnerability exposure
The first finding was the scope of the impact: within the sites and devices tracked by Indeni Insight, 9% of sites have at least one vulnerable device, and 8% of deployed NGFWs are vulnerable. The number is small but still significant.
Given the similarity between the percentages both of affected NGFWs and sites, we pivoted the query data by site to determine if there is a 1-1 relationship, but among the sites, there is a fairly wide distribution of the percent of vulnerable devices:
Just under half of the affected sites had a percent of NGFWs vulnerable of about 15%, with slightly more than a quarter of sites at about 60%. This wide distribution provides an opportunity to observe whether there is a correlation of upgrade behavior and the scope of the job, with the smallest impact at 1% of deployed devices, and the largest impact at 100% of devices at small sites (fewer than 5 NGFWs, below the noise gating threshold for this graph).
Upgrade count: zero.
Surprisingly, in the month since the vulnerability announcement, there have been zero devices upgraded. By that we don’t just mean “less than 1%”, we mean there is literally not even one NGFW with evidence of an upgrade to a non-vulnerable version of PAN-OS.
A closer inspection of the Indeni Insight data set shows that there is a small number of devices (<1%) running PAN-OS 8.0.16, but this set has no record before the vulnerability announcement. The local Insight collection node is also a new instance, so there is not a definitive way to determine whether this is a new purchase that shipped with the latest PAN-OS release, or whether there was an upgrade that occurred prior to being added to Indeni Insight. While it seems statistically unlikely that there would be zero upgrades, that explanation is the one that’s more consistent with the rest of the query data.
Desire for stability
In Indeni’s recent Palo Alto Networks Deployment Trends Report, there was strong evidence of institutional reluctance to upgrade. The top 4 most commonly deployed releases all had a well established and stable install base, with the oldest having been first deployed a year before the report, at the start of the study data. Each of those releases also grew in number during the study, with new deployments all the way up to the study’s end.
To compare the lack of upgrades against a larger baseline, we performed an additional query against Indeni Insight, extending the upgrade record retrieval for all of the vulnerable devices from one month to one year, and discovered that the cumulative upgrade rate for the entire year was only 4%. Naively assuming a relatively normal distribution provides an expected upgrade rate per month of ~0.35% – so the lack of upgrades during the month after the vulnerability announcement is not significantly unusual.
The idea that institutions prefer to qualify a release, then standardize all deployments on that release, is upheld both by Indeni’s customer interactions and anecdotal observations taken from locations such as the Palo Alto Networks Firewall subreddit. Given that each new release has at least a little new behavior, accepting a single set of known issues provides a degree of predictability that leads to network and operational stability, even if at the cost of missing out on new features, bug fixes, and vulnerability fixes. Indeni even found evidence of PAN-OS downgrades on some NGFWs, albeit between releases in the same version.
Evaluation of the vulnerability: no cause for alarm
Another factor in the lack of PAN-OS upgrades is the “Medium” rating for the two vulnerabilities. They are both related to vulnerabilities discovered in the Linux kernel, but, given the differences between a NGFW and a generic multi-user Linux server, the complexity for each attack is greatly increased in the PAN-OS environment.
PAN-SA-2019-0006 is related to CVE-2018-14634, which has a CVSS score of between 7 and 8 (v2 vs v3) and a “High” rating. That issue is an Escalation of Privilege, which could allow a local user to gain administrator privileges. On a NGFW, Best Practice is that only administrators get user accounts. This environmental modification reduces the CVSS v3 score down from “High” to “Medium” in the 6-7 range. The score is still relatively high because it could amplify the impact if chained with other exploits, but it would already require a serious security breach just to acquire the shell access necessary to exploit this vulnerability.
PAN-SA-2019-0007 is related to CVE-2018-18065, which has a CVSS score between 4 and 6 (v2 vs v3). This vulnerability could remotely cause a crash of the SNMP agent via a maliciously crafted UDP packet, assuming that the attacker knows valid credentials for SNMP. SNMPv2C, with its backward compatible community string, potentially exposes the credentials to a well-placed attacker. However, the NGFW, when deployed according to Best Practices, should be installed with its Management interface on dedicated administrative subnet with strong access controls. Gaining access to that administrative subnet would already be considered a serious security breach whose scope greatly exceeds the severity of crashing the firewall’s SNMP agent.
In summary, exploiting either of these vulnerabilities requires already having a security breach more serious than the vulnerability itself. It’s a classic case of “Someone could unlock a window in your house if they somehow broke in first.”
Conclusion: obvious in retrospect
The hypothesis that a firewall should always be kept fully patched against known exploits is simply not true for any of the environments monitored by Indeni Insight. There is a bias against release upgrades, for the predictable behavior and maintenance requirements that lead to network stability. That bias leads to widespread deployment of particular releases, which itself means that any future upgrade will require an increasingly large – and therefore complex – amount of effort, which creates a sunk cost and further increases the threshold of severity required to trigger a coordinated site-wide release migration. Against that requirement, the relatively minor impact of this particular set vulnerabilities is clearly insufficient.
In this case, the best way to deal with the risk is to have followed Best Practices from the start.
Did you know that Indeni continuously monitors Palo Alto Networks devices to validate alignment with Best Practices? Download a free trial for Indeni today.