Originally published at: https://indeni.com/deployment-of-the-cisco-nexus-fex-technology-and-analysis-by-indeni/
Cisco launched the Nexus 2000 Fabric Extender (FEX) series in 2009. The FEX technology is based on the IEEE 802.1BR standard. The Nexus 2000 Series Fabric Extenders behave logically like remote line cards of the parent Cisco Nexus switch. The parent switch can be a Cisco Nexus 5000 Series Switch, Nexus 6000 Series Switch, Nexus 7000 Series Switch or a Cisco Nexus 9000 Series Switch. Though the Nexus parent switch and Nexus 2000 series switch are physically separate, they are managed as a single entity from a software image, configuration, and STP perspective. FEXs are a cost-efficient way to design the access layer and they can offer simplicity to the data center access operations and architecture. Finally, the Nexus 2000 series switches offer a broad range of interface options including 40 Gigabit Ethernet, 10 Gigabit Ethernet, 1 Gigabit Ethernet, 100 MB and Fibre Channel over Ethernet (FCoE).
Here’s a list of the Nexus 2000 Series switch FEX technology topics covered in this article:
- Cisco Nexus 2k series switch introduction
- Fabric Extender Features Overview & Guidelines
- Analysis of the Nexus Fabric Extender technology by Indeni
- Alerting and proactively monitoring of the FEX by Indeni
- Summary
Cisco Nexus 2k series switches FEX introduction
The Cisco Nexus 2000 platform offers over 10 models starting with a 24-port 1GE (Nexus 2224TP) all the way up to 48-port 1/10GE SFP/SFP+ (Nexus 2300). The following Nexus 2000s Series switches are described below: [table id=11 /]Cisco Nexus FEX Main Features Overview
Enterprises need to simplify their data center implementation, reduce operational complexity, increase scalability and flexibility, and simplify management. The Cisco Nexus 2k series switches and the fabric extender technology are a highly scalable unified solution that meets all these requirements.Let’s have a quick overview of the NX-OS FEX technology and Nexus 2k series switches which are widely deployed from small to large scale Data Centers.
✓ Cisco Fabric Extender (FEX) ports are designed to support end host connectivity
✓ Offers a broad range of connectivity options, including 100 Mbps, 1, 10, and 40 Gigabit Ethernet, copper and fiber
✓ Cisco FEX ports do not support connectivity to any Layer 2 switch which generate spanning-tree BPDU packets. A Cisco FEX port set by default the port in err disable status in case that receives a BPDU packet
✓ Deployed with deep shared buffers that can absorb bursts of traffic from the wide variety of workloads
✓ Each parent Nexus switch can support a maximum number of Cisco FEX and this value is based on the model of the parent switch.
✓ Cisco Fabric Extender uplinks toward the parent switch can be configured as port channel connections in order to offer high availability and load sharing.
✓ If the Cisco FEX is single-homed to only one member of the switch pair, it is configured as a standard port channel. However, if the Cisco FEX is dual-homed to both members of the vPC switch pair it is configured as a vPC on the port channel
✓ FEX switches are compact 1RU and built for nonstop operation with redundant hot-swappable power supplies and hot-swappable fan tray with redundant fans
✓ Offer unified ports for flexible LAN and SAN deployment with support for Ethernet, Fibre Channel, and Fiber Channel over Ethernet (FCoE) connectivity
Cisco Nexus & Indeni Lab Environment
This section describes the Cisco Nexus switches, Indeni platform and Software versions deployed at the lab environment in order to setup and test the FEX technology. The Nexus switches are configured with the FEX technology in order to be discovered and analyzed by Indeni. The deployed CISCO Nexus lab physical topology is illustrated in the next diagram.
Cisco Nexus - Indeni Lab Physical Topology
The software versions information of the Network devices and Indeni equipped to the lab can be found in the following table.[table id=12 /]
Nexus Hardware and Software revisions
Finally, a brief description for the used Cisco Nexus switches for deploying the FEX technology and testing the Indeni vPC analysis capabilities is provided below.Nexus 5548UP: A powerful and widely deployed Nexus switch commonly used as “End of Row” at a Data Center with thirty-two fixed Unified ports 1/10 Gbps fixed SFP+ on the base chassis along with one expansion slot.
Nexus 2248TP-E: A popular Nexus FEX with forty-eight ports 100M/1000BaseT Enhanced Host interfaces (server interfaces) and four 10-Gigabit Ethernet Fabric uplinks.
Cisco Nexus FEX & Indeni Analysis
Indeni currently monitors and analyzes several important metrics related to the NX-OS FEX technology. Indicative metrics are described below:✓ FEX connectivity status and service degradation metrics ✓ FEX diagnostic metrics which guarantee the optimum operation of the N2k series switch ✓ FEX environmental metrics
The “show fex” and “show fex detail” NX-OS command are executed at the Nexus 5k switch deployed at the lab to check the FEX connectivity status, Model, S/N, Uplinks type etc. The output of this NX-OS command proves that the FEX switches toward the N5k switches are operational without any issue (status: online).
[sourcecode language=“plain” highlight=“1”]
N5k-DOWN# show fex
FEX FEX FEX FEX Fex
Number Description State Model Serial
100 FEX0100 Online N2K-C2248TP-1GE FOX1842GGZ3
112 FEX2k - under N5kUP Online N2K-C2248TP-1GE FOX1938GDUY
[/sourcecode]
[sourcecode language=“plain” highlight=“1”]
N5k-DOWN# show fex detail
FEX: 100 Description: FEX0100 state: Online
FEX version: 7.0(7)N1(1) [Switch version: 7.0(7)N1(1)]
FEX Interim version: 7.0(7)N1(1)
Switch Interim version: 7.0(7)N1(1)
Extender Serial: 1434332423432
Extender Model: N2K-C2248TP-1GE, Part No: 73-13232-02
Card Id: 99, Mac Addr: bc:16:f5:ab:db:61, Num Macs: 64
Module Sw Gen: 12594 [Switch Sw Gen: 21]
post level: complete
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/31
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po100 - Interface Up. State: Active
Eth1/31 - Interface Up. State: Active
[/sourcecode]
Let’s continue! We are connecting to the Indeni server and review the discovered Nexus switches deployed at the lab. The dual Nexus 5k switches previously configured with the vPC and FEX technology have been successfully discovered.
We review the Indeni FEX analysis of one of the aforementioned Nexus 5k series switches. As is illustrated in the next capture, it is necessary initially to choose one of the discovered Nexus switch e.g. N5k-down and then to choose the “More Device Info” tab on the right side.
Indeni has completed the Fabrix Extender analysis which does not need in most of the cases more than a few minutes. We review the Indeni analysis by selecting the first tab of the FEX named “FEX Connectivity Status”. It is verified that the FEX connectivity “online” status between the vPC peers as was displayed also by running the relevant NX-OS command.
We are moving to the next tab named “FEX Diagnostic Status” which provides the results of the diagnostic tests of a Fabric Extender chassis collected by Indeni. The provided information includes metrics such as ASIC (application-specific integrated circuit) status, Temperature sensors, Power Supplies and SPROMs (Serial Programmable Read-only Memory) diagnostic tests results. The list of all the NX-OS diagnostic results analyzed by Indeni can be found in the next capture.
We move forward by selecting the next tabs named “FEX Fan status”, “FEX Module Power status”, “FEX Power Supply status”, “FEX Temperature status” which provide real time analysis of the Nexus with the environmental sensor status including FAN, Power-Supply, Power and Temperature status respectively.
Let’s move to the Indeni Lab and see how Indeni responds in case of failure of the vPC connectivity status. The FEX uplink is set down to one of the N5k switches installed at the lab.
[sourcecode language=“plain”]
N5k-UP# sh fex detail
FEX: 112 Description: FEX2k - under N5kUP state: Offline
FEX version: 7.0(2)N1(1) [Switch version: 7.0(2)N1(1)]
FEX Interim version: 7.0(2)N1(1)
Switch Interim version: 7.0(2)N1(1)
Extender Serial: FOX1938GPUY
Extender Model: N2K-C2248TP-1GE, Part No: 73-13232-02
[/sourcecode]
Indeni instantly reacts to this change, analyzes the relevant NX-OS outputs in the backend and triggers the next alert.
Let’s see the details of this alert raised by Indeni.
The FEX is in Offline state! The network administrator or NOC is notified for this critical issue and detailed remediation steps are provided to resolve it. Below can be found a list of possible connectivity FEX states where Indeni can analyze, trigger an alert and provide detailed remediation steps.
- Online - FEX is up and interfaces are online.
- Offline - FEX is down and interfaces are not available.
- Online Sequence - FEX is in the process of coming online.
- Offline Sequence - FEX is in the process of going offline.
- Discovered - FEX has been detected. Online sequence will begin shortly.
- FEX AA Upg Fail - Upgrade of Dual Homed FEX failed.
- AA Version Mismatch - FEX is connected via vPC to two parent switches running different NX-OS versions. vPC can be established but dual home FEX is not possible to work.
- Fex Type Mismatch - Configured FEX type does not match physical FEX inserted.
Finally, Indeni analyzes all of the FEX environmental metrics and provides information for all these metrics with remediation steps as is illustrated in the next capture.
Summary
This article describes the main components and features of the Nexus Fabric Extender (FEX) Nexus technology. A pair of Nexus 5k and N2K-C2248TP-1GE switches are used at a lab environment by Indeni in order to get a dynamic report of the FEX activated features and status. In addition, it is described how to get access to the FEX Indeni Analysis tabs and captures are provided to describe the NX-OS FEX capabilities offered by Indeni via the Live Config tab. It is demonstrated how the Indeni platform continuously analyzes the relevant NX-OS outputs and triggers the relevant alerts including detailed remediation steps on how to check the status of the feature, how to apply it and finally a reference to the Cisco configuration guide or best practices design guide are provided.This article is part of the Nexus & Indeni series of articles discussing the cool features and powerful capabilities that Indeni offers for the NX-OS FEX technology. More articles about Indeni and the Cisco Nexus technologies are coming soon. Stay tuned 
Learn more by joining the Indeni Community. If you liked this article please share it by clicking on one of the social media sharing icons at the top of this page. Thank you to Vasileios Bouloukos for his contributions to this article.