Cisco Nexus Switches & Indeni Series

Originally published at: https://indeni.com/cisco-nexus-switches-indeni-series/

Spectre - Meltdown Vulnerabilities

The Spectre and Meltdown vulnerabilities have been described as the “worst ever CPU bug” which could let attackers to steal sensitive data. Spectre was made public along with the Meltdown vulnerability on January 3, 2018, after the affected hardware vendors had already been made aware of the issue on June 1, 2017(!). It has affected the Intel, AMD, Qualcomm and ARM processors which are used by almost all the IT vendors.

The main Network vendors e.g. Cisco Fortinet, Checkpoint, Juniper etc. reacted immediately and published official Security Advisories and bugs for these major vulnerabilities which their status is updated daily.
This article describes how Cisco Nexus Switches have been affected by these vulnerabilities and how Indeni reacted to this major issue.

Are you a Network Administrator, System Engineer, Software Engineer, Indeni Knowledge Expert (IKEs), Indeni Software Developers or Tech Geek? If yes, then read on! This article is published for you!

Spectre, Meltdown & Cisco Nexus

Let’s see first a brief description about these vulnerabilities. Meltdown and Spectre are "Information Disclosure" and "Privilege Escalation" types of vulnerabilities respectively. In brief, the Spectre vulnerability allows an attacker to read memory contents from other users' running programs and with the Meltdown vulnerability an attacker can access kernel memory from user space Rogue data cache load. The following Common Vulnerability and Exposures (CVE) IDs have been published for the Spectre and Meltdown:
  • CVE-2017-5715 - Branch Target Injection Side-Channel Information Disclosure Vulnerability (aka Spectre)
  • CVE-2017-5753 - Bounds Check Bypass Side-Channel Information Disclosure Vulnerability (aka Spectre)
  • CVE-2017-5754 - Rogue Data Cache Load Side-Channel Information Disclosure Vulnerability (aka Meltdown)
In brief, it is important to remember that despite the two different names, both Spectre and Meltdown are essentially based on the same weakness of speculative execution.

But how difficult would be for an attacker to take advantage of these vulnerabilities and to be exploited to a Cisco Nexus switch. In brief, two criteria must be met in order for these vulnerabilities to be exploited to a Nexus switch.

  • The device being targeted must utilize an affected processor
  • An attacker must be able to execute their own code on this
The majority of Cisco products including several Nexus series switches are closed systems that do not allow customers to run custom code and therefore are not vulnerable. Cisco products have been considered “potentially” vulnerable only if permit a user to execute custom code with Cisco code on the same microprocessor. Special attention is needed to the Cisco Nexus series switches deployed as a virtual device such as the Nexus 9000v and Nexus 1000v models. In this case it should be validated that the installed hardware, the operating system and the hypervisor have been patched against the Meltdown and Spectre vulnerabilities, especially if these devices have been installed in a multi-tenant hosting environment such as Amazon, Softlayer and Azure.

Cisco Nexus Vulnerable products

The following table lists the Cisco Nexus switches which are affected by the Spectre (CVE-2017-5715) and the Meltdwon (CVE-2017-5754 , CVE-2017-5753) vulnerabilities. It should be noticed that there isn’t any fix yet as is also stated to the third column.
Cisco Nexus Switch Cisco Bug ID Fixed Release Availability
Cisco Nexus 3000 Series Switches CSCvh32392 A fix is pending on upstream vendors.
Cisco Nexus 3500 Series Switches CSCvh32393 A fix is pending on upstream vendors.
Cisco Nexus 5000 Series Switches (OAC Feature) CSCvh32394 A fix is pending on upstream vendors.
Cisco Nexus 6000 Series Switches (OAC Feature) CSCvh32390 A fix is pending on upstream vendors.
Cisco Nexus 7000 Series Switches (OAC Feature, Feature Bash) CSCvh32390 A fix is pending on upstream vendors.
Cisco Nexus 9000 Series Switches - Standalone, NX-OS mode Doe A fix is pending on upstream vendors.
Finally, a list with all the Cisco Devices along with the bug id and fixed release status has been officially published by the vendor.

Any Nexus series switch not listed in the table can be considered not vulnerable. It should be mentioned that there is an ongoing investigation by Cisco for this critical issue, so it is possible products and services currently considered not vulnerable may later be considered vulnerable as more information about this vulnerabilities becomes available. It should be mentioned that the official Cisco Nexus Security Advisory for the Spectre and Meltdown vulnerability has been currently updated twenty times (Revision 1.20) starting from 4th January with the latest update to be pushed in 1st of March 2018.

Finally, Cisco has officially published that only the following Nexus products are not affected by the next vulnerabilities CVE-2017-5715 CVE-2017-5753 CVE-2017-5754:

  • Cisco Nexus 4000 Series Blade Switches
  • Cisco Nexus 9000 Series Fabric Switches - ACI mode

How can check if a Nexus switch is vulnerable

Cisco has published a different bug id for each Cisco Nexus Series switch where are described the prerequisites for an attacker to exploit the vulnerability as well as the recommended workaround. It should be mentioned that a CCO Cisco account is required to get access to the Cisco Bug Repository. This section is going to focus on the Nexus 9000 series switches and on the commands/features that should be enabled in order this switch to be vulnerable. Finally, the proposed by the vendor workaround is implemented.

The following features/roles have to be enabled in a Nexus 9000 Series switch for non-admin users in order to be able to exploit the aforementioned vulnerabilities:

  • feature bash-shell
  • guest shell enabled
  • user role dev-ops
  • username shelltype bash
Let’s see which NX-OS commands should be executed in order to diagnose if the Meltdown/Spectre vulnerabilities can be exploited to a Nexus 9000 Series switch.
  • The Feature bash-shell is enabled and displayed with the relevant “show” NX-OS command.
N9k(config)# feature bash-shell 
 
N9k# show feature | i bash
bash-shell 1 enabled
  • The guestshell is enabled.
N9k# show guestshell 
Virtual service guestshell+ detail
State : Activated
Package information
Name : guestshell.ova
Path : /isanboot/bin/guestshell.ova
Application
Name : GuestShell
Installed version : 2.3(0.0)
Description : Cisco Systems Guest Shell
Signing
Key type : Cisco release key
Method : SHA-1
Licensing
<…>
  • A user assigned to the dev-ops role is configured and verified with the relevant “show” NX-OS command.
N9k(config)# username indeni password indeni123! role dev-ops
N9k(config)# show user-account indeni
user:indeni
this user account has no expiry date
roles:dev-ops 
  • A user configured as shell type bash is configured and verified with the next NX-OS command.
N9k(config)# username indeni shelltype bash 
N9k(config)# sh run | i shelltype
username indeni shelltype bash
As a workaround for these vulnerabilities Cisco proposes the Nexus switch Administrator will not be privileged to execute third-party software and to restrict access to the various shell environments through configuration changes. It should be mentioned that this will remove shell access for all users, including admin and dev-ops users. In brief the next steps should be implemented:

✓ Disable access to the host shell by configuring "no feature bash"

N9k(config)# no feature bash
N9k(config)# sh feature | i bash
bash-shell 1 disabled

✓ Remove the guest shell by executing "guestshell destroy". Press “y” to the warning message.

N9k(config)# guestshell destroy 
You are about to destroy the guest shell and all of its contents. 
Be sure to save your work. Are you sure you want to continue? (y/n) [n] y

✓ Change any configuration setting user's shell from bash to vsh: "username shelltype vsh".

N9k(config)# sh run | i usern
username indeni password 5 $5$24j6l4xd$1kY03FskmSTX2/wf2H/X4XB3x3mFlgPB.7/d/ClqYg9
role dev-ops
username indeni shelltype bash
N9k(config)# no username indeni shelltype bash
 
N9k(config)# username indeni password indeni123! role dev-ops 
N9k(config)# username indeni shelltype vsh 

✓ Remove any users with dev-ops role

N9k(config)# sh user-account 
user:admin
this user account has no expiry date
roles:network-admin 
user:indeni
this user account has no expiry date
roles:dev-ops 
N9k(config)# conf t
N9k(config)# no username indeni
N9k(config)# sh user-account 
user:admin
this user account has no expiry date
roles:network-admin 
N9k(config)# 

NOTE1: If any shell access mentioned above is required, it should be sure the user does not run code/binary software from untrusted 3rd parties.

NOTE2: Admin level users have access to run third party code and configure features/roles. The assumption is that admin level roles are provided only to trusted-users.

It should be stated that Nexus are not vulnerable in their default configuration. An admin must configure or provision Guest Shell, or Feature Bash on the Nexus 9k.

Finally, the other Nexus series switches such as the Nexus 3000, 5000 & 7000 Series switches can also be vulnerable if the NX-OS Open Agent Container (OAC) feature is enabled. The relevant NX-OS command is illustrated below where the OAC feature is in activated state.

switch# show virtual-service detail
 
Virtual service oac detail
  State                 : Activated
  Package information
    Name                : oac.ova
    Path                : bootflash:/oac.ova
    Application
      Name              : OpenAgentContainer

This feature allows customers to load their own container based applications to an NX-OS based device. OAC is not enabled by default. For a device to be vulnerable, an administrator would need to install and activate this feature.

Indeni-Nexus vs Meltdown-Spectre

Indeni Knowledge Expert team has already started to develop the required scripts to get the relevant to Meltdown and Spectre metrics. In particular, the relevant scripts are planned to be merged to the Indeni coming release. So, the Nexus switches will be analyzed by Indeni and an alert along with detailed remediation steps will be provided to the user in case that the conditions for Nexus switch exploitation are enabled.

Then the Network Administration will perform a proactively action to protect the Nexus switching infrastructure by this serious vulnerability until a final patch is published by the vendors. Finally, this information can be published to the Indeni insight providing valuable information regarding the total number of the vulnerable Nexus switches and reviewing the trends in the IT Cisco Nexus products.

Conclusion

IT companies like Google, Microsoft, and Apple are publishing software patches to protect against the Meltdown and Spectre vulnerabilities. Cisco is proposing fixes and configuration which can make the Network devices and Nexus switches resilient against to these serious threats. A complete solution can be offered by the hardware & chip manufacturers such as Intel, AMD and ARM. The solution is currently implemented by putting strong fences around the flawed chips which they would slow down or even thwart Meltdown and Spectre attacks. Many of these patches have turned out to have issues causing systems from slowing down performance up to 30%(!) or to not be able to boot.

We discussed in this article how the Nexus series switches are affected by these vulnerabilities. Then we investigated to the lab how we can identify if a Nexus switch is vulnerable and how a Network Administrator can secure the Nexus series switch by applying the configuration proposed by the vendor. Finally, it was highlighted how the Indeni responded to this serious issue and how it can analyze and proactively inform the Administrator if the Nexus switches are vulnerable and how to implement the recommended solution published by Cisco.

Finally, do you want to see the Spectre and Meltdown exploitation in action? Then, you can visit the meltdownattack.com where have been published video demonstrating Meltdown and Spectre exploitation. Enjoy!