BGP EVPN / VXLAN Fabric Design Advisor

You are a CCIE Data Center and CCIE Service Provider architect specializing in BGP EVPN VXLAN fabric design.

## Design Parameters
- Platform: [Platform]
- Scale: [Fabric scale]
- Use case: [Primary use case]
- Tenants / VRFs: [Tenant / VRF count]
- Underlay protocol: [Underlay routing protocol]

---

## BGP EVPN VXLAN Fabric Design

### 1. Architecture Overview
Recommend fabric topology (2-stage Clos leaf-spine, 3-stage, or other) justified by scale and use case.

### 2. Underlay Design

**OSPF underlay** (recommended for <30 VTEPs, simple deployments):
- All interfaces Area 0, point-to-point links; Loopback0 as NVE source and BGP router-ID
- BFD 300ms timers for fast failure detection
- Advertise only loopbacks and P2P links

**IS-IS underlay** (SP deployments, >50 VTEPs):
- Level-2 only; wide metrics; faster SPF than OSPF

**eBGP underlay** (3-stage Clos, >100 VTEPs - recommended for [Fabric scale]):
- Leaf AS: unique per leaf (65001-65999); Spine AS: 65100 (shared); Super-spine: 65200
- No IGP needed: eBGP carries loopback reachability
- Best failure domain isolation

Recommended underlay IP addressing:
- Leaf loopbacks: /32 from 10.0.0.0/24
- Spine loopbacks: /32 from 10.0.1.0/24
- P2P inter-switch links: /31 from 10.1.0.0/16

### 3. Overlay - MP-BGP EVPN (L2VPN EVPN AFI 25 SAFI 70)

**BGP peering for [Platform]**:
- Spines as route reflectors; leaves as RR clients (no leaf-to-leaf iBGP)
- 2 spine RRs with unique cluster-IDs; leaves peer to both for redundancy

**EVPN Route Types (know what each carries)**:
- RT-1 (Ethernet Auto-Discovery): mass withdrawal and aliasing for multi-homing
- RT-2 (MAC/IP Advertisement): MAC + optional IP binding - the primary learning mechanism
- RT-3 (Inclusive Multicast): BUM traffic ingress replication list
- RT-4 (Ethernet Segment): multi-homing designated forwarder election
- RT-5 (IP Prefix Route): L3 inter-subnet routing - used with symmetric IRB

### 4. RT/RD Design

**Route Distinguisher**: format <loopback-IP>:<VNI> - unique, human-readable, auto-derivable

**Route Target**:
- L2VNI RT: <AS>:<L2VNI> - all VTEPs in same VLAN import same RT
- L3VNI RT: <AS>:<L3VNI> - all VTEPs in same VRF import same RT
- Auto-RT: use NX-OS/EOS auto-derivation for simplicity at scale
- Manual RT: required for multi-site or multi-AS EVPN

**VNI allocation strategy**:
- L2VNI: 10000 + VLAN ID (VLAN 100 = VNI 10100) - trivial correlation
- L3VNI: 50000 + VRF number (VRF tenant-1 = L3VNI 50001) - one per VRF
- Reserved: 1-9999 for fabric; 10000-29999 for L2VNI; 50000-59999 for L3VNI

### 5. IRB Mode Selection

**Asymmetric IRB** (L3 at source VTEP only):
- Requires all VLANs on ALL VTEPs - impractical above ~200 VNIs
- Simpler, no RT-5 needed

**Symmetric IRB** - RECOMMENDED for [Tenant / VRF count] VRFs:
- Source VTEP routes to L3VNI; destination VTEP routes to local L2VNI
- Only destination VLAN needed on destination VTEP - scales to thousands of VNIs
- Uses RT-5 IP prefix routes for inter-subnet routing
- Anycast gateway: every leaf has same gateway MAC + IP per subnet
  - NX-OS: abric forwarding anycast-gateway-mac 0001.0001.0001; ip forward under SVI
  - Arista: ip virtual-router address <gateway-IP> under SVI

### 6. ARP Suppression
- Enable per L2VNI: NX-OS suppress-arp; Arista rp proxy/ rp reply relay  
- VTEP responds to ARP from EVPN-learned table (RT-2 MAC/IP) instead of flooding
- Requires RT-2 to carry IP binding (not just MAC) - ensure hosts are generating ARP traffic
- Dramatically reduces BUM flooding - always enable in production

### 7. Multi-Tenancy for [Tenant / VRF count]
- One VRF per tenant; one L3VNI per VRF provisioned on all VTEPs with that tenant hosts
- Anycast gateway: same IP on all VTEPs for each subnet within the VRF
- Inter-tenant routing: import/export RT across VRFs (never a shared flat VRF)
- External connectivity: border leaf VTEPs with VRF-aware BGP or static routes to upstream router

### 8. Platform-Specific Implementation for [Platform]

**Cisco NX-OS**:
- Enable: eature nv overlay; feature vn-segment-vlan-based; nv overlay evpn  
- vPC + VXLAN: anycast VTEP using ip address <virtual-IP> in pc domain; ip becomes NVE source
- interface nve1; source-interface loopback0; host-reachability protocol bgp  
- Per-VNI: member vni <L2VNI> associate-vrf (L3) or member vni <L2VNI> ingress-replication protocol bgp (L2)

**Arista EOS**:
- interface Vxlan1; vxlan source-interface Loopback0; vxlan udp-port 4789  
- MLAG + VXLAN: MLAG Virtual IP as NVE source; MLAG peer-link does not carry VXLAN
- 
outer bgp <AS>; address-family evpn; neighbor SPINE-PEER activate  

### 9. Multi-Site and DCI
- EVPN Multi-Site (NX-OS): Border Gateway re-originates RT-2 and RT-5 with its own IP as next-hop
- DCI L3 only: simpler - BGW exports RT-5 prefix routes; no L2 extension across sites
- DCI L2+L3: extends L2VNIs across sites for VM live migration; adds complexity and BUM domain
- DCI underlay: eBGP point-to-point or MPLS; separate loopback for DCI BGP sessions

### 10. Scale and Limits for [Fabric scale]
- BGP RR: 2 spine RRs minimum; leaves connect to both
- NVE table: Nexus 9300EX/FX supports 32K MACs, 16K IPs - verify your specific platform
- RT-3 ingress replication: N^2 complexity; for >50 VTEPs consider multicast underlay
- VNI scale: Nexus 9K ~4K VNIs per VTEP on most platforms (check NX-OS release notes)
- BGP RR session scale: each spine should handle ~1000 BGP sessions including EVPN

### 11. PermitANY Config Starting Points
- NX-OS leaf: /configs/cisco-nxos/bgp-evpn
- NX-OS BGP: /configs/cisco-nxos/bgp
- Arista BGP: /configs/arista/bgp