Originally published at: How to use Cisco Nexus OSFP over vPC Review | Indeni
Last year Cisco introduced the Layer 3 over vPC feature. By using this feature, a Layer 3 device can form dynamic routing peering adjacency between both the vPC peers in a vPC network architecture. Routing adjacency between a Layer 3 device and a vPC peer can be formed now without Layer 3 inter-switch links between the vPC peers but over the vPC peer-link introducing essential advantages.
This article is part of the series of vPC Nexus-Indeni articles. This blog reviews the benefits introduced by this new feature introduced to NX-OS. The OSPF routing protocol over vPC is configured and tested in a lab environment. We will discuss the configurations, implementation and verification steps in detail.
Cisco Nexus Layer 3 routing over vPC
Before the introduction of the Layer 3 routing over vPCA a Network Architect had the following routing options:- Use separate Layer 3 links to connect Layer 3 devices to the vPC domain.
- Only static routing for a Layer 3 device attached to a vPC domain. The static routes should point to the HSRP address configured on the vPC peer devices.
- When both Layer3 and Layer2 traffic are needed two different links are required. An individual dedicated Layer 3 link is used for routed traffic and a separate Layer 2 link for Layer2 traffic. In case of redundancy, multiple links should be grouped in portchannels increasing the required number of interconnections for the design.
- The vPC peer switches should have the same number of VLANs
- The peer-gateway feature should be enabled before configuring the Layer 3 over vPC feature. This feature allows the vPC standby peer to forward packets on behalf of the Active peer. So, it is saved bandwidth by avoiding traffic over the peer link.
- Adjacency is formed on the vPC VLAN
- Equal L3 metrics across vPC interfaces
- Requires explicit configuration of the "layer3 peer-router” NX-OS command to the vPC
- No Static routes limitations
- No extra Parallel links
- No additional design changes
- Route peering across vPC’s over existing infrastructure
- Routing between vPC DCI
- Faster convergence when a link or device fails
Table 1: Nexus Routing over vPC supported NX-OS versions
More details with the supported Network Topologies and NX-OS versions can be found to the next link published by Cisco:https://www.cisco.com/c/en/us/support/docs/ip/ip-routing/118997-technote-nexus-00.html
Cisco Nexus Lab Environment
The following section depicts the Cisco Nexus switches and Software versions deployed at the lab environment in order to configure and test the Routing over vPC feature. The deployed CISCO Nexus lab physical topology is illustrated in the next diagram.
Figure 1: Layer 2 Setup for OSPF routing over vPC 10
The hostnames, models and software versions information of the Network devices to the lab can be found to the following table:  Table 2: Nexus Hardware and Software revisions
The following figure provides an easy mnemonic to elaborate the numbering convention of the deployed NX-OS versions.Figure: 2: NX-OS Release explanation
Here’s a brief description for the used Cisco Nexus switches for deploying and testing the vPC technology:Nexus 93180YC-EX: The Cisco Nexus 93180YC-EX Switch has 48 1/10/25G-Gbps Small Form Pluggable Plus (SFP+) ports and 6 40/100-Gbps Quad SFP+ (QSFP+) uplink ports. All ports are line rate, delivering 3.6 Tbps of throughput in a 1-rack-unit (1RU) form factor.
Nexus 9504: It provides high performance 1, 10, 25, 40, 50 and 100GE GE in a compact 7-rack-unit modular chassis. This versatile modular switch is designed for high-density, end-of-row, and high-performance aggregation-layer deployments in both traditional and Cisco Application Centric Infrastructure (ACI)–enabled data centers.
Implementation and Testing
The pair of Nexus 93180YC-EX is configured with vPC and the Layer3 routing over vPC is activated. Then the OSPF routing protocol is activated between the vPC peers 93180YC-EX and the vPC edge device Nexus 9504. The Layer 3 topology is illustrated to the next network diagram.
Figure 3: Layer 3 Setup for OSPF routing over vPC
vPC Configuration and Verification
The vPC peer configuration is applied to the pair of the Nexus 93180YC-EX switches. Details about how to configure and verify the operation of the vPC can be found in the Nexus and Indeni series:Initially, we validate that both Nexus peer switches run an image that supports the Layer 3 over vPC feature and that the vPC basic setup is operational. Then, to activate the Layer 3 routing over vPC featue the peer-gateway feature is enabled and configured on both the peers.
The layer 3 routing over vPC is enabled by running the NX-OS command “layer3 peer-router” in the vpc domain to both vPC peers.
Note-1: If you configure the “layer3 peer-router” command without enabling the peer-gateway feature, a syslog notification message is received to enable the peer-gateway feature.
Note-2: If you configure the “layer3 peer-router” command only on one of the peers or you disable it on one peer, the operational state of layer 3 peer-router gets disabled.
Here’s the relevant configuration from both vPC peers:
[sourcecode language=“plain” highlight=“8,9,11”]
feature vpc
vpc domain 1
role priority 100
peer-keepalive destination 169.254.0.2 source 169.254.0.1 vrf
vpc-keepalive
delay restore 360
peer-gateway
layer3 peer-router
track 10
auto-recovery
ip arp synchronize
interface port-channel1
description *** VPC PEER LINK ***
switchport mode trunk
switchport trunk allowed vlan 1,102
spanning-tree port type network
vpc peer-link
[/sourcecode]
Figure: 4: N9k-A vPC Peer-link configuration& Keepalive configuration
[sourcecode language=“plain” highlight=“6,7”]
feature vpc
vpc domain 1
peer-keepalive destination 169.254.0.1 source 169.254.0.2 vrf vpc-keepalive
delay restore 360
peer-gateway
layer3 peer-router
track 10
auto-recovery
ip arp synchronize
interface port-channel1
description *** VPC PEER LINK ***
switchport mode trunk
switchport trunk allowed vlan 1,102
spanning-tree port type network
vpc peer-link
interface port-channel2
description *** VPC KEEPALIVE LINK ***
no switchport
vrf member vpc-keepalive
ip address 169.254.0.1/30
[/sourcecode]
Figure 5: N9k-B vPC Peer-link & Keepalive configuration
The show vpc brief command is executed at the Nexus 93k-A switch deployed at the lab to check the vPC domain ID, the peer-link status, the keepalive message status, whether the configuration consistency is successful, and if a peer link has formed. It also states the status of the vPC Port Channels. The output of this NX-OS command proves that the l3 routing and the peer-gateway features have been activated over the vPC.
[sourcecode language=“plain” highlight=“12,18”]
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 197
Track object : 10
Peer Gateway : Enabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 360s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Enabled
vPC Peer-link status
id Port Status Active vlans
1 Po1 up 1,102
[/sourcecode]
Figure 6: show vPC output
OSPF Configuration & Verification
The OSPFv2 is configured across the previously configured vPC pair of the Nexus N93k and the N95k. The next table illustrates the IP addressing scheme used for the Routing over the vPC.Table 3: Nexus Transit L3 VLAN
The OSPFv2 is an IETF link-state protocol for IPv4 networks. It has the following main prerequisites, limitations and guidelines for Nexus switches:- You have configured at least one interface for IPv4 that can communicate with a remote OSPFv2 neighbor.
- You have installed the Enterprise Services license.
- You have enabled the OSPF feature (see the “Enabling OSPFv2” section).
- You can have up to four instances of OSPFv2 in a VDC.
- Cisco NX-OS displays areas in dotted decimal notation regardless of whether you enter the area in decimal or dotted decimal notation.
- BFD with OSPF over vPC peer links is not currently supported.
The OSPFv2 router identifier is configured by assigning it IP address of the loopback. This IP address identifies the OSPFv2 instance and must exist on a configured interface in the system. Below are the relevant OSPF configurations for all the Nexus switches:
[sourcecode language=“plain”]
feature ospf
router ospf 1
router-id 10.138.255.1
passive-interface default
interface Vlan102
description *** OSPF Peering with N95k ***
no shutdown
ip address 10.138.254.3/29
no ip ospf passive-interface
ip router ospf 1 area 0.0.0.0
interface loopback0
ip address 10.138.255.1/32
[/sourcecode]
Figure 7: OSPF neighbor configuration to N93k-A
[sourcecode language=“plain”]
feature ospf
router ospf 1
router-id 10.138.255.2
passive-interface default
interface loopback0
ip address 10.138.255.2/32
[/sourcecode]
Figure: 8: OSPF neighbor configuration to N93k-B
[sourcecode language=“plain”]</pre>
<pre>
feature ospf
router ospf 1
router-id 10.0.254.1
passive-interface default
interface Vlan102
description *** OSPF Peering with N93k over vPC ***
no shutdown
ip address 10.138.254.1/29
no ip ospf passive-interface
ip router ospf 1 area 0.0.0.0
interface loopback0
ip address 10.0.254.1/32
[/sourcecode]
Figure 9: OSPF neighbor configuration to N95k
The OSPF configuration passive-interface default command is used to suppress routing updates on all interfaces except of the interfaces that are manually configured. Finally, all networks have been configured to the default backbone area (Area 0). The Area ID is a 32-bit value that you can enter as a number or in dotted decimal notation, such as 10.10.10.1. Cisco NX-OS always displays the area in dotted decimal notation.
[sourcecode language=“plain”]
sh ip ospf neighbors
OSPF Process ID 1 VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
10.0.254.1 1 FULL/DROTHER 1w0d 10.138.254.1 Vlan102
10.138.255.2 1 FULL/BDR 1w0d 10.138.254.4 Vlan102
[/sourcecode]
Figure 10: OSPF neighbors from N93k-A
[sourcecode language=“plain”]
sh ip ospf neighbors
OSPF Process ID 1 VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
10.0.254.1 1 FULL/DROTHER 1w0d 10.138.254.1 Vlan102
10.138.255.1 1 FULL/DR 1w0d 10.138.254.3 Vlan102
[/sourcecode]
Figure 11: OSPF neighbors from N93k-B
[sourcecode language=“plain”]
sh ip ospf ne
OSPF Process ID 1 VRF default
Total number of neighbors: 3
Neighbor ID Pri State Up Time Address Interface
10.138.255.1 1 FULL/DR 1w0d 10.138.254.3 Vlan102
10.138.255.2 1 FULL/BDR 1w0d 10.138.254.4 Vlan102
[/sourcecode]
Figure 12: OSPF neighbors from N95k
The OSPF area network type is set to broadcast. OSPFv2 As is illustrated above the Nexus switches establish a DR and BDR that controls LSA flooding on the network. OSPFv2 uses the well-known IPv4 multicast addresses 224.0.0.5 and a MAC address of 0100.5300.0005 to communicate with neighbors. All other nexus switches establish adjacency with the DR and the BDR and use the IPv4 multicast address 224.0.0.6 to send LSA updates to the DR and BDR.
Finally, the routing table below illustrates the received OSPF routes of the N95k Nexus switch. The ECMP (equal routes) works as expected and each route has two next hops toward to each of the Nexus N93k vPC peer switches.
[sourcecode language=“plain”]
sh ip route ospf-1
IP Route Table for VRF “default”
‘*’ denotes best ucast next-hop
‘**’ denotes best mcast next-hop
‘[x/y]’ denotes [preference/metric]
‘%<string>’ in via output denotes VRF <string>
10.138.0.0/24, ubest/mbest: 2/0
*via 10.138.254.3, Vlan102, [110/80], 6d13h, ospf-1, intra
*via 10.138.254.4, Vlan102, [110/80], 6d13h, ospf-1, intra
10.138.1.0/24, ubest/mbest: 2/0
*via 10.138.254.3, Vlan102, [110/80], 6d13h, ospf-1, intra
*via 10.138.254.4, Vlan102, [110/80], 6d13h, ospf-1, intra
[/sourcecode]
Figure 13: OSPF routes from N95k
The routing table from a Nexus vPC peer switch N93k switch is below is where all the OSPF routes learnt via the N95k have a single next hop which is the IP address of the N95k.
[sourcecode language=“plain”]
sh ip route ospf-1
IP Route Table for VRF “default”
‘*’ denotes best ucast next-hop
‘**’ denotes best mcast next-hop
‘[x/y]’ denotes [preference/metric]
‘%<string>’ in via output denotes VRF <string>
10.0.254.1/32, ubest/mbest: 1/0
*via 10.138.254.1, Vlan102, [110/20], 1w0d, ospf-1, type-2
10.139.1.0/24, ubest/mbest: 1/0
*via 10.138.254.1, Vlan102, [110/83], 1w0d, ospf-1, inter
10.139.17.0/24, ubest/mbest: 1/0
*via 10.138.254.1, Vlan102, [110/83], 1w0d, ospf-1, inter
[/sourcecode]
Figure 14: OSPF routes from N93k switches
Cisco Nexus and Indeni
Indeni currently analyzes several important metric related to the OSPF routing over vPC feature like these described below:- vPC role configuration/operation status
- vPC type 2 consistency status
- vPC advanced features status such as peer-gateway, vPC graceful consistency check, auto-recovery
- vPC peer link status or any of the members link vPC status
- vPC peer keepalive status
- OSPF neighbor status
Summary
This article describes the Layer 3 Routing feature recently introduced to Cisco NX-OS. The main advantages of this feature are described before be tested at the Lab. The steps to configure OSPF over vPC are described and implemented to the Nexus Lab environment. Testing and verification commands proves the fully operation of this feature. Finally, it is mentioned how the Indeni analyzes and get insight of the Routing over vPC feature.This article is part of a Series of articles which highlight cool NX-OS features and powerful analysis capabilities that Indeni offers for the Nexus switches. More articles for Cisco Nexus and Indeni are coming soon. Stay tuned 
Learn more by joining the Indeni Community. If you liked this article please share it by clicking on one of the social media sharing icons at the top of this page. Thank you to Vasileios Bouloukos for his contributions to this article.
