Cisco NX-OSConfig756 downloads

BGP EVPN / VXLAN Fabric — Cisco NX-OS (Spine-Leaf)

Generate a complete VXLAN BGP EVPN spine-leaf fabric configuration for Cisco NX-OS (Nexus 9000). Supports iBGP with Route-Reflector spines, symmetric IRB, distributed anycast gateway, vPC with Fabric Peering, and Multi-Site Border Gateway.

↓ example.txt

What is it?

BGP EVPN (RFC 7432) over VXLAN (RFC 7348) is the modern standard for data center fabric overlays. VXLAN encapsulates L2 frames in UDP (port 4789) with a 24-bit VNI (16M segments vs 4K VLANs), while BGP EVPN provides the control plane: distributing MAC/IP reachability (Type-2 routes), BUM handling via ingress replication (Type-3), external prefixes (Type-5), and multi-homing (Type-1/4). In NX-OS, this runs as a spine-leaf CLOS fabric where spines act as MP-iBGP route reflectors and leaves are VTEPs performing symmetric IRB — both ingress and egress VTEPs route via a per-VRF L3 VNI, so no traffic hairpinning occurs. The distributed anycast gateway eliminates FHRP (HSRP/VRRP) entirely: every leaf simultaneously acts as the default gateway with the same IP and MAC address, enabling VM mobility without ARP re-learning.

When to use

·Building or modernizing a data center with more than 10 leaf switches
·Multi-tenant environments (private cloud, managed hosting) requiring traffic isolation per tenant
·Workloads requiring VM mobility across racks without STP domain extensions
·Environments needing more than 4094 broadcast domains (VXLAN supports 16 million VNIs)
·Replacing legacy 3-tier hierarchical architectures with a scalable spine-leaf CLOS fabric

When NOT to use

·Small DC environments with fewer than 10 switches — the operational overhead of BGP EVPN is not justified
·Brownfield environments with applications tightly coupled to STP-based flat L2 domains that cannot be segmented
·Teams without BGP and VXLAN expertise — misconfiguration of RD/RT or L3 VNI causes silent routing failures
·Environments where the network hardware does not support VXLAN hardware offload (software VXLAN kills performance)

How it compares

vs Flood-and-Learn VXLAN (no BGP EVPN)Flood-and-learn VXLAN distributes BUM traffic to all VTEPs with no control plane, no MAC/IP distribution, and no ARP suppression. BGP EVPN eliminates BUM flooding with proxy ARP (suppress-arp), distributes MAC+IP via Type-2 routes, and enables symmetric IRB. BGP EVPN is always preferred for production fabrics.

vs MPLS/VPLSMPLS/VPLS requires service provider-grade hardware and MPLS label distribution (LDP/RSVP). VXLAN BGP EVPN runs over standard IP/UDP and works on commodity merchant silicon (Nexus 9000 with Broadcom/Cisco ASICs). BGP EVPN also reuses the same control plane for L2 and L3 VPNs, while MPLS needs separate L2VPN and L3VPN stacks.

vs 3-Tier Hierarchical (Core/Distribution/Access)3-tier uses STP for L2 loop prevention, limiting scale and convergence. Spine-leaf CLOS with BGP EVPN uses L3 everywhere (no STP between spines and leaves), achieves sub-second convergence with BFD, and scales linearly — adding a spine doubles east-west bandwidth. 3-tier is still valid for small/medium networks without DC-grade requirements.

Prerequisites

→Cisco Nexus 9000 platform (9300-EX/FX/GX for leaf, 9500/9300 for spine)
→NX-OS 7.0(3)I1 or later (NX-OS 9.x recommended for vPC Fabric Peering and Multisite)
→MTU 9216 on ALL fabric links — VXLAN adds 50 bytes overhead (14 Eth + 20 IP + 8 UDP + 8 VXLAN header)
→Loopback IPs (Loopback0 for router-ID, Loopback1 for VTEP source) planned before deployment — must be unique per node
→Anycast gateway MAC planned — must be IDENTICAL across all VTEPs in the fabric
→BGP AS number decided — iBGP design uses same ASN on all nodes; eBGP design assigns unique ASN per leaf
→VNI numbering scheme planned — recommend: L2 VNI = 10000+VLAN, L3 VNI = 50000+VRF-index

Config Generator

Hostname *

Node Role *

Determines which config sections are generated. Spine = Route Reflector. Border-Gateway = Multi-Site DCI.

Loopback0 IP (Router-ID) *

Router-ID — unique per node, used for BGP and OSPF/IS-IS

Loopback1 IP (VTEP Source / PIP) *

VTEP NVE source — unique per node. For vPC: this is the Primary IP (PIP). Secondary anycast IP configured separately.

Anycast VTEP IP (vPC Shared)

vPC shared anycast VTEP IP — same on both vPC peers. Leave empty for standalone leaf or spine.

BGP AS Number *

iBGP AS — same across all fabric nodes for iBGP design. For eBGP underlay: each leaf gets unique ASN.

Overlay Model *

iBGP with Route-Reflector on spines (recommended) or eBGP between leafs and spines

Underlay Protocol *

IGP for underlay reachability. OSPF is simplest. IS-IS preferred for large fabrics or dual-stack IPv6.

Anycast Gateway MAC *

Must be IDENTICAL on all VTEPs in the fabric. This is the shared MAC for the distributed anycast gateway.

BGP Neighbors (Spines or RRs)

Layer-2 VNIs (VLANs stretched over VXLAN)

Layer-3 VNIs (one per VRF/tenant)

Fabric Uplink Interfaces (leaf to spine)

Enable vPC (dual-homing)

vPC Domain ID

Must match on both vPC peers

vPC Peer-Link Port-Channel

Port-channel number for vPC peer-link

vPC Keepalive Destination IP

vPC keepalive destination IP (management interface of the peer)

vPC Keepalive Source IP

vPC keepalive source IP (local management interface)

vPC Fabric Peering (virtual peer-link)

Uses VXLAN tunnel as peer-link. Requires dedicated loopback for VPL. No physical peer-link cable needed.

vPC Fabric Peering Loopback IP

Loopback for virtual peer-link (vPC Fabric Peering only)

Multi-Site Border Gateway (DCI)

DCI Uplink Interface

DCI uplink interface for multi-site tracking

Remote Site BGW Loopback IP

Remote site Border Gateway loopback IP for EVPN peering

Verification commands

show nve peers

VTEP peer state and BUM replication list

ExpectedState column shows Up for all fabric peers. Down peers indicate an underlay reachability or NVE config issue.

show nve vni

VNI membership in NVE interface

ExpectedState = Up for all L2 and L3 VNIs. Mode = CP (control-plane/BGP). S-ARP flag confirms suppress-arp is active.

show bgp l2vpn evpn summary

BGP EVPN session state to all spines/RRs

ExpectedState/PfxRcd column shows a number (received prefixes), not Idle/Active/Connect.

show bgp l2vpn evpn

Full EVPN table — all route types

ExpectedShould see Type-2 (MAC/IP), Type-3 (IMET/BUM), Type-5 (IP prefix) routes from all VTEPs.

show bgp l2vpn evpn vni-id 10010

EVPN routes for a specific L2 VNI

ExpectedShows MAC/IP routes (Type-2) and BUM routes (Type-3) for that VNI from all VTEPs.

show l2route evpn mac all

L2 MAC table populated by EVPN control plane

ExpectedRemote MACs should show VXLAN as the next-hop with the remote VTEP IP.

show l2route evpn mac-ip all

MAC+IP host routes from Type-2 EVPN routes

ExpectedHost /32 routes for all remote VMs. Used for ARP suppression proxy.

show ip route vrf TENANT-A

Per-VRF routing table verification

ExpectedShould show remote subnets via VXLAN next-hop. EVPN Type-5 routes appear as /24 or larger prefixes.

Debug commands

show nve vni ingress-replication 10010

BUM replication list for specific VNI

WarningShould list all remote VTEPs that participate in this VNI. Empty list = no EVPN Type-3 routes received.

show bgp l2vpn evpn neighbors 10.0.0.100 advertised-routes

What we advertise to the Route Reflector

WarningUse to verify local Type-2 (MAC/IP) and Type-3 routes are being sent to spines.

show bgp l2vpn evpn route-type 2 0 5001.0000.1b08

Find specific MAC address in EVPN table

WarningReplace MAC with target host MAC. Shows which VTEP is advertising it and with which VNI.

debug bgp l2vpn evpn in

Live EVPN update debugging

WarningUse with caution in production — heavy output. Always run undebug all after troubleshooting.

Common mistakes & fixes

Symptomnv overlay evpn missing or not configured

CauseNVE interface works in flood-and-learn mode without BGP control plane — no ARP suppression, no MAC distribution

FixAdd nv overlay evpn globally. Verify with show nve peers — State should show CP (control-plane) not DP (data-plane).

SymptomARP resolution fails or traffic hairpins through spine

CauseAnycast gateway MAC mismatch across VTEPs — fabric forwarding anycast-gateway-mac configured differently on different leaves

FixVerify show fabric forwarding ip local-info on all VTEPs. MAC must be IDENTICAL on all nodes. Reload NVE if changed.

SymptomVXLAN packets dropped silently, ping works locally but not across fabric

CauseMTU not set to 9216 on fabric interfaces — VXLAN adds 50 bytes overhead (14 Eth + 20 IP + 8 UDP + 8 VXLAN)

FixSet mtu 9216 on all routed fabric interfaces. Verify with show interface Eth1/1 | grep MTU.

SymptomHosts cannot route between subnets in the same VRF across VTEPs

Causeadvertise l2vpn evpn missing under the VRF address-family in BGP — L3 VNI routes not redistributed into EVPN

FixAdd advertise l2vpn evpn under router bgp / vrf TENANT-A / address-family ipv4 unicast. Verify Type-5 routes in show bgp l2vpn evpn.

SymptomBUM traffic flooding instead of proxy ARP — high broadcast traffic across fabric

Causesuppress-arp not configured on L2 VNI member in NVE — defeats BGP EVPN ARP proxy

FixAdd suppress-arp under each member vni in interface nve1. Verify Flags column in show nve vni shows S-ARP.

SymptomBGP EVPN sessions Established but no routes exchanged between VTEPs

CauseRoute Distinguisher or Route Target mismatch — wrong rd/rt on VRF prevents import of remote tenant routes

FixCheck show bgp l2vpn evpn and verify RT values match between VTEPs for the same VRF. Use route-target both auto evpn for simplicity.

SymptomvPC: Multi-site or external routing via BGW broken after vPC failure

Causeadvertise-pip missing under address-family l2vpn evpn — both vPC peers advertise the same VIP as next-hop

FixAdd advertise-pip under router bgp / address-family l2vpn evpn. This causes each peer to advertise its Primary IP (PIP) for Type-2 routes.

SymptomInter-VRF routing works locally on leaf but not across fabric

CauseL3 VNI VLAN not added to NVE interface as associate-vrf — VRF has no VXLAN data-plane attachment

FixAdd member vni <l3vni-id> associate-vrf under interface nve1. Verify show nve vni shows the L3 VNI with Type = L3.

Further learning

Recommended books to go deeper on this topic.

Building Data Centers with VXLAN BGP EVPN

Lukas Krattiger, Shyam Kapadia, David Jansen · Cisco Press · Clos fabric design with VXLAN EVPN: route types, symmetric IRB, multihoming, multi-site

Paperback ↗

BGP Design and Implementation

Randy Zhang & Micah Bartell · Cisco Press · BGP fundamentals needed before EVPN — attributes, route reflection, policy

Kindle ↗Hardcover ↗Paperback ↗

NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures

Ron Fuller, David Jansen, Matthew McPherson · Cisco Press · VXLAN BGP EVPN on NX-OS — overlay design and configuration

Hardcover ↗