How to use Cisco Nexus OSFP over vPC Review: Cisco Nexus OSFP vPC Implementation & Verification

Originally published at: How to use Cisco Nexus OSFP over vPC Review | Indeni

Cisco Nexus virtual Port Channel (vPC) is a virtualization technology launched in the mid of 2009. The majority of Cisco Nexus Series Switches (Nexus 9000, 7000, 5000 and 3000 Series) supports the Cisco Nexus virtual Port Channel (vPC), which is a virtualization technology launched in the mid of 2009. In brief, the vPC enables the deployment of a link aggregation from a downstream network device to two individual and independent Cisco NX-OS switches (vPC peers).

Last year Cisco introduced the Layer 3 over vPC feature. By using this feature, a Layer 3 device can form dynamic routing peering adjacency between both the vPC peers in a vPC network architecture. Routing adjacency between a Layer 3 device and a vPC peer can be formed now without Layer 3 inter-switch links between the vPC peers but over the vPC peer-link introducing essential advantages.

This article is part of the series of vPC Nexus-Indeni articles. This blog reviews the benefits introduced by this new feature introduced to NX-OS. The OSPF routing protocol over vPC is configured and tested in a lab environment. We will discuss the configurations, implementation and verification steps in detail.

Cisco Nexus Layer 3 routing over vPC

Before the introduction of the Layer 3 routing over vPCA a Network Architect had the following routing options:
  • Use separate Layer 3 links to connect Layer 3 devices to the vPC domain.
  • Only static routing for a Layer 3 device attached to a vPC domain. The static routes should point to the HSRP address configured on the vPC peer devices.
  • When both Layer3 and Layer2 traffic are needed two different links are required. An individual dedicated Layer 3 link is used for routed traffic and a separate Layer 2 link for Layer2 traffic. In case of redundancy, multiple links should be grouped in portchannels increasing the required number of interconnections for the design.
By taking advantage of the new Layer3 Routing over vPC design, a Layer 3 device can form peering adjacency between both the vPC peers in a vPC setup. Routing adjacency is now formed without Layer 3 inter-switch links between the vPC peers but over the vPC peer-link. Take the following prerequisites into consideration:
  • The vPC peer switches should have the same number of VLANs
  • The peer-gateway feature should be enabled before configuring the Layer 3 over vPC feature. This feature allows the vPC standby peer to forward packets on behalf of the Active peer. So, it is saved bandwidth by avoiding traffic over the peer link.
  • Adjacency is formed on the vPC VLAN
  • Equal L3 metrics across vPC interfaces
  • Requires explicit configuration of the "layer3 peer-router” NX-OS command to the vPC
The embedded vPC loop avoidance mechanism is still available for all traffic, but now the TTL of the traffic sent over a peer link is not decremented. The benefits of Dynamic Routing over vPC feature are summarized below:
  • No Static routes limitations
  • No extra Parallel links
  • No additional design changes
  • Route peering across vPC’s over existing infrastructure
  • Routing between vPC DCI
  • Faster convergence when a link or device fails
The following table summarizes the NX-OS versions and the Routing Protocols Adjacencies supported over vPC VLANs.

![|421x258](upload://bEuJtTzALFSrZr1xkmCEWy2La0N.png)Table 1: Nexus Routing over vPC supported NX-OS versions

More details with the supported Network Topologies and NX-OS versions can be found to the next link published by Cisco:

https://www.cisco.com/c/en/us/support/docs/ip/ip-routing/118997-technote-nexus-00.html

Cisco Nexus Lab Environment

The following section depicts the Cisco Nexus switches and Software versions deployed at the lab environment in order to configure and test the Routing over vPC feature. The deployed CISCO Nexus lab physical topology is illustrated in the next diagram.

![|433x421](upload://2g3ywS5ml18QFcgkUS6iyC7saa4.png)Figure 1: Layer 2 Setup for OSPF routing over vPC 10

The hostnames, models and software versions information of the Network devices to the lab can be found to the following table: ![|433x219](upload://q54zfuAyCBLg2seEJeUtDr91Fm9.png) Table 2: Nexus Hardware and Software revisions

The following figure provides an easy mnemonic to elaborate the numbering convention of the deployed NX-OS versions.

![|649x272](upload://1kTVnd3E3yKfCKd8mKRMaQGyn2C.png)Figure: 2: NX-OS Release explanation

Here’s a brief description for the used Cisco Nexus switches for deploying and testing the vPC technology:

Nexus 93180YC-EX: The Cisco Nexus 93180YC-EX Switch has 48 1/10/25G-Gbps Small Form Pluggable Plus (SFP+) ports and 6 40/100-Gbps Quad SFP+ (QSFP+) uplink ports. All ports are line rate, delivering 3.6 Tbps of throughput in a 1-rack-unit (1RU) form factor.

Nexus 9504: It provides high performance 1, 10, 25, 40, 50 and 100GE GE in a compact 7-rack-unit modular chassis. This versatile modular switch is designed for high-density, end-of-row, and high-performance aggregation-layer deployments in both traditional and Cisco Application Centric Infrastructure (ACI)–enabled data centers.

Implementation and Testing

The pair of Nexus 93180YC-EX is configured with vPC and the Layer3 routing over vPC is activated. Then the OSPF routing protocol is activated between the vPC peers 93180YC-EX and the vPC edge device Nexus 9504. The Layer 3 topology is illustrated to the next network diagram.

![|441x433](upload://A3iAMZOv60xzKIP5nEFKLJqf52j.png)Figure 3: Layer 3 Setup for OSPF routing over vPC

vPC Configuration and Verification

The vPC peer configuration is applied to the pair of the Nexus 93180YC-EX switches. Details about how to configure and verify the operation of the vPC can be found in the Nexus and Indeni series:

Initially, we validate that both Nexus peer switches run an image that supports the Layer 3 over vPC feature and that the vPC basic setup is operational. Then, to activate the Layer 3 routing over vPC featue the peer-gateway feature is enabled and configured on both the peers.

The layer 3 routing over vPC is enabled by running the NX-OS command “layer3 peer-router” in the vpc domain to both vPC peers.

Note-1: If you configure the “layer3 peer-router” command without enabling the peer-gateway feature, a syslog notification message is received to enable the peer-gateway feature.

Note-2: If you configure the “layer3 peer-router” command only on one of the peers or you disable it on one peer, the operational state of layer 3 peer-router gets disabled.

Here’s the relevant configuration from both vPC peers:

[sourcecode language=“plain” highlight=“8,9,11”]
feature vpc

vpc domain 1
role priority 100
peer-keepalive destination 169.254.0.2 source 169.254.0.1 vrf
vpc-keepalive
delay restore 360
peer-gateway
layer3 peer-router
track 10
auto-recovery
ip arp synchronize

interface port-channel1
description *** VPC PEER LINK ***
switchport mode trunk
switchport trunk allowed vlan 1,102
spanning-tree port type network
vpc peer-link
[/sourcecode]

Figure: 4: N9k-A vPC Peer-link configuration& Keepalive configuration

[sourcecode language=“plain” highlight=“6,7”]
feature vpc

vpc domain 1
peer-keepalive destination 169.254.0.1 source 169.254.0.2 vrf vpc-keepalive
delay restore 360
peer-gateway
layer3 peer-router
track 10
auto-recovery
ip arp synchronize

interface port-channel1
description *** VPC PEER LINK ***
switchport mode trunk
switchport trunk allowed vlan 1,102
spanning-tree port type network
vpc peer-link

interface port-channel2
description *** VPC KEEPALIVE LINK ***
no switchport
vrf member vpc-keepalive
ip address 169.254.0.1/30
[/sourcecode]

Figure 5: N9k-B vPC Peer-link & Keepalive configuration

The show vpc brief command is executed at the Nexus 93k-A switch deployed at the lab to check the vPC domain ID, the peer-link status, the keepalive message status, whether the configuration consistency is successful, and if a peer link has formed. It also states the status of the vPC Port Channels. The output of this NX-OS command proves that the l3 routing and the peer-gateway features have been activated over the vPC.

[sourcecode language=“plain” highlight=“12,18”]
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 197
Track object : 10
Peer Gateway : Enabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled, timer is off.(timeout = 240s)
Delay-restore status : Timer is off.(timeout = 360s)
Delay-restore SVI status : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router : Enabled
vPC Peer-link status

id Port Status Active vlans


1 Po1 up 1,102
[/sourcecode]

Figure 6: show vPC output

OSPF Configuration & Verification

The OSPFv2 is configured across the previously configured vPC pair of the Nexus N93k and the N95k. The next table illustrates the IP addressing scheme used for the Routing over the vPC.

![|687x105](upload://3ENB064LTMLcKgD73bHxVWZuMsH.png)Table 3: Nexus Transit L3 VLAN

The OSPFv2 is an IETF link-state protocol for IPv4 networks. It has the following main prerequisites, limitations and guidelines for Nexus switches:
  • You have configured at least one interface for IPv4 that can communicate with a remote OSPFv2 neighbor.
  • You have installed the Enterprise Services license.
  • You have enabled the OSPF feature (see the “Enabling OSPFv2” section).
  • You can have up to four instances of OSPFv2 in a VDC.
  • Cisco NX-OS displays areas in dotted decimal notation regardless of whether you enter the area in decimal or dotted decimal notation.
  • BFD with OSPF over vPC peer links is not currently supported.
The first step in configuring OSPFv2 after enabling the OSPF NX-OS feature, is to create an OSPFv2 instance. Assign a unique instance tag for this OSPFv2 instance. The instance tag can also be any string.

The OSPFv2 router identifier is configured by assigning it IP address of the loopback. This IP address identifies the OSPFv2 instance and must exist on a configured interface in the system. Below are the relevant OSPF configurations for all the Nexus switches:

[sourcecode language=“plain”]
feature ospf

router ospf 1
router-id 10.138.255.1
passive-interface default

interface Vlan102
description *** OSPF Peering with N95k ***
no shutdown
ip address 10.138.254.3/29
no ip ospf passive-interface
ip router ospf 1 area 0.0.0.0

interface loopback0
ip address 10.138.255.1/32

[/sourcecode]

Figure 7:  OSPF neighbor configuration to N93k-A

[sourcecode language=“plain”]
feature ospf

router ospf 1
router-id 10.138.255.2
passive-interface default

interface loopback0
ip address 10.138.255.2/32
[/sourcecode]

Figure: 8:  OSPF neighbor configuration to N93k-B

[sourcecode language=“plain”]</pre>
<pre>
feature ospf
router ospf 1
router-id 10.0.254.1
passive-interface default

interface Vlan102
description *** OSPF Peering with N93k over vPC ***
no shutdown
ip address 10.138.254.1/29
no ip ospf passive-interface
ip router ospf 1 area 0.0.0.0

interface loopback0
ip address 10.0.254.1/32
[/sourcecode]

Figure 9: OSPF neighbor configuration to N95k

The OSPF configuration passive-interface default command is used to suppress routing updates on all interfaces except of the interfaces that are manually configured. Finally, all networks have been configured to the default backbone area (Area 0). The Area ID is a 32-bit value that you can enter as a number or in dotted decimal notation, such as 10.10.10.1. Cisco NX-OS always displays the area in dotted decimal notation.

[sourcecode language=“plain”]
sh ip ospf neighbors
OSPF Process ID 1 VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
10.0.254.1 1 FULL/DROTHER 1w0d 10.138.254.1 Vlan102
10.138.255.2 1 FULL/BDR 1w0d 10.138.254.4 Vlan102
[/sourcecode]

Figure 10: OSPF neighbors from N93k-A

[sourcecode language=“plain”]
sh ip ospf neighbors
OSPF Process ID 1 VRF default
Total number of neighbors: 2
Neighbor ID Pri State Up Time Address Interface
10.0.254.1 1 FULL/DROTHER 1w0d 10.138.254.1 Vlan102
10.138.255.1 1 FULL/DR 1w0d 10.138.254.3 Vlan102
[/sourcecode]

Figure 11: OSPF neighbors from N93k-B

[sourcecode language=“plain”]
sh ip ospf ne
OSPF Process ID 1 VRF default
Total number of neighbors: 3
Neighbor ID Pri State Up Time Address Interface
10.138.255.1 1 FULL/DR 1w0d 10.138.254.3 Vlan102
10.138.255.2 1 FULL/BDR 1w0d 10.138.254.4 Vlan102
[/sourcecode]

Figure 12: OSPF neighbors from N95k

The OSPF area network type is set to broadcast. OSPFv2 As is illustrated above the Nexus switches establish a DR and BDR that controls LSA flooding on the network. OSPFv2 uses the well-known IPv4 multicast addresses 224.0.0.5 and a MAC address of 0100.5300.0005 to communicate with neighbors. All other nexus switches establish adjacency with the DR and the BDR and use the IPv4 multicast address 224.0.0.6 to send LSA updates to the DR and BDR.

Finally, the routing table below illustrates the received OSPF routes of the N95k Nexus switch. The ECMP (equal routes) works as expected and each route has two next hops toward to each of the Nexus N93k vPC peer switches.

[sourcecode language=“plain”]
sh ip route ospf-1
IP Route Table for VRF “default”
‘*’ denotes best ucast next-hop
‘**’ denotes best mcast next-hop
‘[x/y]’ denotes [preference/metric]
‘%<string>’ in via output denotes VRF <string>
10.138.0.0/24, ubest/mbest: 2/0
*via 10.138.254.3, Vlan102, [110/80], 6d13h, ospf-1, intra
*via 10.138.254.4, Vlan102, [110/80], 6d13h, ospf-1, intra
10.138.1.0/24, ubest/mbest: 2/0
*via 10.138.254.3, Vlan102, [110/80], 6d13h, ospf-1, intra
*via 10.138.254.4, Vlan102, [110/80], 6d13h, ospf-1, intra
[/sourcecode]

Figure 13: OSPF routes from N95k

The routing table from a Nexus vPC peer switch N93k switch is below is where all the OSPF routes learnt via the N95k have a single next hop which is the IP address of the N95k.

[sourcecode language=“plain”]

sh ip route ospf-1
IP Route Table for VRF “default”
‘*’ denotes best ucast next-hop
‘**’ denotes best mcast next-hop
‘[x/y]’ denotes [preference/metric]
‘%<string>’ in via output denotes VRF <string>
10.0.254.1/32, ubest/mbest: 1/0
*via 10.138.254.1, Vlan102, [110/20], 1w0d, ospf-1, type-2
10.139.1.0/24, ubest/mbest: 1/0
*via 10.138.254.1, Vlan102, [110/83], 1w0d, ospf-1, inter
10.139.17.0/24, ubest/mbest: 1/0
*via 10.138.254.1, Vlan102, [110/83], 1w0d, ospf-1, inter
[/sourcecode]

Figure 14: OSPF routes from N93k switches

Cisco Nexus and Indeni

Indeni currently analyzes several important metric related to the OSPF routing over vPC feature like these described below:
  • vPC role configuration/operation status
  • vPC type 2 consistency status
  • vPC advanced features status such as peer-gateway, vPC graceful consistency check, auto-recovery
  • vPC peer link status or any of the members link vPC status
  • vPC peer keepalive status
  • OSPF neighbor status

Summary

This article describes the Layer 3 Routing feature recently introduced to Cisco NX-OS. The main advantages of this feature are described before be tested at the Lab. The steps to configure OSPF over vPC are described and implemented to the Nexus Lab environment. Testing and verification commands proves the fully operation of this feature. Finally, it is mentioned how the Indeni analyzes and get insight of the Routing over vPC feature.

This article is part of a Series of articles which highlight cool NX-OS features and powerful analysis capabilities that Indeni offers for the Nexus switches. More articles for Cisco Nexus and Indeni are coming soon. Stay tuned :relaxed:

Learn more by joining the Indeni Community. If you liked this article please share it by clicking on one of the social media sharing icons at the top of this page. Thank you to Vasileios Bouloukos for his contributions to this article.

 

I would recommend to read the article from the next link because here it has issues with the text format and figures.

Enjoy :slight_smile:

https://indeni.com/blog/how-to-use-cisco-nexus-osfp-over-vpc-review-cisco-nexus-osfp-vpc-implementation-verification/

1 Like

Great article. Is there any issues with peering to a VRRP address when routing over vPC? For example if the two Nexus 93k’s were acting as a vlan gateway using VRRP (VIP 10.x.x.1, N93k-A 10.x.x.2, N93k-B 10.x.x.3), can this VRRP VIP be used as the route peering address? Or is it best to peer with the SVI’s .2 & .3 addressing?

1 Like

Good question @geoffw82! And welcome to the community :slight_smile: @Vasileios_Bouloukos @Brad_Spilde et al., surfacing this question – do you have any insight, or do you know someone who could answer?