Troubleshooting · python · 444 copies
Network Path Analyzer
A traceroute shows you less than half the path. Layer 2 switches never appear in one because they do not decrement TTL, and a firewall that happily forwards your probe can still be dropping the port you care about -- the probe is ICMP or UDP and matches a different rule. This walks the real path for one flow, from the source's own gateway so the path is the one that traffic takes, enters every device on it, and names the hop that hurts: the saturated uplink, the access port throwing CRCs, the link that flapped at 3am, or the rule that denies the traffic, by name. Counters are read by cause rather than listed -- CRC is the cable, late collisions are a duplex mismatch, output drops are not an error at all -- and the age of those counters is printed beside them, because a total is not a rate. Pivots to the HSRP or VRRP active member on its own, and a field it could not read prints as a dash, never as a zero.
- Arista EOS
- FortiOS
- Cisco IOS
- Cisco IOS-XE
- Cisco IOS-XR
- Junos
- Cisco NX-OS
- PAN-OS
$
pip install "netmiko>=4.0" pyyamlterminal
network_path_analyzer.py
#!/usr/bin/env python3
"""
Network Path Analyzer
READ THE DOCUMENTATION BEFORE YOU RUN THIS:
https://permitany.com/scripts/network-path-analyzer
It covers what the script does, whether it is safe to run, the
inventory file and credentials it needs, how to read the output,
and what to do when something fails.
pip install "netmiko>=4.0" pyyaml
"""
# SAFE: read-only. It sends show commands, traceroute, and the policy lookup
# each platform provides. It never enters configuration mode and has no code
# path that writes to a device. See AUTHORISATION in the documentation: the
# policy lookup is NOT a show command on every platform.
#
# EXIT CODES 0 path walked, nothing wrong 1 findings 2 the run failed
#
# CREDENTIALS come from the environment, never from an argument:
# NET_USER / NET_PASS / NET_ENABLE (not needed at all with --offline-dir)
import argparse
import csv
import hashlib
import os
import re
import sys
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
except Exception:
pass
NETMIKO_DRIVER = {
"ios": "cisco_ios", "iosxe": "cisco_ios", "iosxr": "cisco_xr",
"nxos": "cisco_nxos", "junos": "juniper_junos", "eos": "arista_eos",
"panos": "paloalto_panos", "fortios": "fortinet", "asa": "cisco_asa",
}
PROTO_NUM = {"tcp": "6", "udp": "17", "icmp": "1"}
# ---------------------------------------------------------------------------
# Command table. Every parser below declares whether its format was verified
# against real device output or modelled from documentation. A parser that
# does not recognise what it got returns the raw text and COULD_NOT_VERIFY,
# never a guess: a plausible wrong answer in a path analysis sends somebody to
# the wrong device in the middle of an incident.
# ---------------------------------------------------------------------------
CONFIDENCE = {k: "modelled" for k in NETMIKO_DRIVER}
CMD = {
"ios": {
"po_members": "show etherchannel summary | include {intf}",
"addr2intf": "show ip interface brief | include {ip}",
"vrf_list": "show vrf",
"version": "show version",
"logs": "show logging | include {intf}",
"arp": "show ip arp {ip}", "arp_vrf": "show ip arp vrf {vrf} {ip}",
"mac": "show mac address-table address {mac}",
"cdp": "show cdp neighbors {intf} detail",
"lldp": "show lldp neighbors {intf} detail",
"route": "show ip route {dst}", "route_vrf": "show ip route vrf {vrf} {dst}",
"exact": "show ip cef exact-route {src} {dst}",
"intf": "show interfaces {intf}", "traceroute": "traceroute {dst}",
"hsrp": "show standby brief", "vrrp": "show vrrp brief",
"session": None, "policy": None,
},
"nxos": {
"po_members": "show port-channel summary interface {intf}",
"addr2intf": "show ip interface brief | include {ip}",
"vrf_list": "show vrf",
"version": "show version",
"logs": "show logging logfile | include {intf}",
"arp": "show ip arp {ip}", "arp_vrf": "show ip arp {ip} vrf {vrf}",
"mac": "show mac address-table address {mac}",
"cdp": "show cdp neighbors interface {intf} detail",
"lldp": "show lldp neighbors interface {intf} detail",
"route": "show ip route {dst}", "route_vrf": "show ip route {dst} vrf {vrf}",
"exact": None, "intf": "show interface {intf}",
"traceroute": "traceroute {dst}",
"hsrp": "show hsrp brief", "vrrp": "show vrrp brief",
"session": None, "policy": None,
},
"iosxr": {
"po_members": "show bundle {intf}",
"addr2intf": "show ipv4 interface brief | include {ip}",
"vrf_list": "show vrf all",
"version": "show version",
"logs": "show logging | include {intf}",
"arp": "show arp {ip}", "arp_vrf": "show arp vrf {vrf} {ip}",
"mac": None, "cdp": "show cdp neighbors {intf} detail",
"lldp": "show lldp neighbors {intf} detail",
"route": "show route {dst}", "route_vrf": "show route vrf {vrf} {dst}",
"exact": "show cef exact-route {src} {dst}",
"intf": "show interfaces {intf}", "traceroute": "traceroute {dst}",
"hsrp": "show hsrp brief", "vrrp": "show vrrp brief",
"session": None, "policy": None,
},
"eos": {
"po_members": "show port-channel {intf} detailed",
"addr2intf": "show ip interface brief | include {ip}",
"vrf_list": "show vrf",
"version": "show version",
"logs": "show logging | include {intf}",
"arp": "show ip arp {ip}", "arp_vrf": "show ip arp vrf {vrf} {ip}",
"mac": "show mac address-table address {mac}",
"cdp": None, "lldp": "show lldp neighbors {intf} detail",
"route": "show ip route {dst}", "route_vrf": "show ip route vrf {vrf} {dst}",
"exact": None, "intf": "show interfaces {intf}",
"traceroute": "traceroute {dst}",
"hsrp": None, "vrrp": "show vrrp brief",
"session": None, "policy": None,
},
"junos": {
"po_members": "show interfaces {intf} extensive | match Link:",
"addr2intf": "show interfaces terse | match {ip}",
"vrf_list": "show route instance summary",
"version": "show version",
"logs": "show log messages | match {intf}",
"arp": "show arp hostname {ip}", "arp_vrf": "show arp hostname {ip}",
"mac": "show ethernet-switching table | match {mac}",
"cdp": None, "lldp": "show lldp neighbors interface {intf}",
"route": "show route {dst}", "route_vrf": "show route {dst} table {vrf}.inet.0",
"exact": None, "intf": "show interfaces {intf} extensive",
"traceroute": "traceroute {dst}",
"hsrp": None, "vrrp": "show vrrp summary",
"session": "show security flow session destination-prefix {dst} destination-port {dport}",
# The ONLY platform where the policy lookup is itself a show, so an SRX
# can be walked end to end with nothing but read access.
"policy": "show security match-policies from-zone {from_zone} to-zone {to_zone} "
"source-ip {src} destination-ip {dst} source-port 1024 "
"destination-port {dport} protocol {proto_name}",
"policy_class": "show", "needs_zones": True,
},
"panos": {
"po_members": None,
"addr2intf": "show interface all | match {ip}",
"vrf_list": "show routing summary",
"version": "show system info",
"logs": None,
"arp": "show arp all | match {ip}", "arp_vrf": "show arp all | match {ip}",
"mac": None, "cdp": None, "lldp": "show lldp neighbors all",
# Chained on purpose: the FIB lookup gives the egress interface, and
# the egress interface gives the zone that security-policy-match
# REQUIRES. Invented zones return a rule that is not the one traffic
# would match.
"route": "test routing fib-lookup virtual-router {vrf} ip {dst}",
"route_vrf": "test routing fib-lookup virtual-router {vrf} ip {dst}",
"exact": None, "intf": "show interface {intf}",
"traceroute": "traceroute host {dst}",
"hsrp": None, "vrrp": "show high-availability state",
"session": "show session all filter source {src} destination {dst} "
"destination-port {dport} protocol {proto}",
"policy": "test security-policy-match from {from_zone} to {to_zone} "
"source {src} destination {dst} destination-port {dport} protocol {proto}",
"policy_class": "test", "needs_zones": True,
},
"fortios": {
"po_members": None,
"addr2intf": "get system interface | grep {ip}",
"vrf_list": "get router info vrf",
"version": "get system status",
"logs": None,
"arp": "get system arp | grep {ip}", "arp_vrf": "get system arp | grep {ip}",
"mac": None, "cdp": None, "lldp": None,
"route": "get router info routing-table details {dst}",
"route_vrf": "get router info routing-table details {dst}",
"exact": None, "intf": "get system interface physical",
"traceroute": "execute traceroute {dst}",
"hsrp": None, "vrrp": "get system ha status",
"session": "diagnose sys session filter dst {dst}",
"policy": "diagnose firewall iprope lookup {src} 1024 {dst} {dport} {proto} {intf}",
"policy_class": "diagnose", "needs_zones": False,
},
"asa": {
"po_members": None,
"addr2intf": "show interface ip brief | include {ip}",
"vrf_list": None,
"version": "show version",
"logs": "show logging | include {intf}",
"arp": "show arp | include {ip}", "arp_vrf": "show arp | include {ip}",
"mac": None, "cdp": None, "lldp": None,
"route": "show route {dst}", "route_vrf": "show route {dst}",
"exact": None, "intf": "show interface {intf}",
"traceroute": "traceroute {dst}",
"hsrp": None, "vrrp": "show failover state",
"session": "show conn address {dst} port {dport}",
"policy": "packet-tracer input {intf} {proto_name} {src} 1024 {dst} {dport}",
"policy_class": "privileged-exec", "needs_zones": False,
},
}
CMD["iosxe"] = CMD["ios"]
FIREWALL_OS = {"panos", "fortios", "asa", "junos"}
AUTH_NOTES = {
"junos": "show only. 'show security match-policies' IS a show, so an SRX needs "
"nothing beyond read access",
"panos": "show plus the test class: 'test routing fib-lookup' and "
"'test security-policy-match'",
"asa": "show plus 'packet-tracer', which usually needs privilege 15",
"fortios": "show/get plus the SPECIFIC command 'diagnose firewall iprope lookup'. "
"Never authorise 'diagnose *' -- that family is not read-only and some "
"branches change state",
"ios": "show plus 'traceroute', which is not a show command",
}
for _k in ("iosxe", "nxos", "iosxr", "eos"):
AUTH_NOTES[_k] = AUTH_NOTES["ios"]
# ----------------------------------------------------------------- inventory
def load_inventory(path):
try:
import yaml
except ImportError:
raise SystemExit("PyYAML is not installed. Run: pip install pyyaml")
try:
with open(path, encoding="utf-8") as fh:
doc = yaml.safe_load(fh) or {}
except OSError as exc:
raise SystemExit("could not read the inventory: %s" % exc)
defaults = doc.get("defaults") or {}
devices = doc.get("devices")
if not devices:
raise SystemExit("the inventory has no 'devices:' list")
out = {}
for i, raw in enumerate(devices, 1):
d = dict(defaults)
d.update(raw or {})
if not d.get("host"):
raise SystemExit("device %d has no 'host'" % i)
osn = str(d.get("os", "")).lower()
if osn not in NETMIKO_DRIVER:
raise SystemExit(
"device %s has os '%s'. Valid values: %s"
% (d["host"], osn or "(missing)", ", ".join(sorted(NETMIKO_DRIVER))))
d["os"] = osn
d["port"] = int(d.get("port", 22))
d["device_name"] = d.get("device_name") or d["host"]
# Connect here; match on any of these.
d["mgmt"] = d.get("mgmt") or d["host"]
addrs = [d["host"], d["mgmt"]] + list(d.get("addresses") or [])
d["addresses"] = [a for a in addrs if a]
for a in d["addresses"]:
if a in out and out[a] is not d:
raise SystemExit(
"address %s is listed on two devices (%s and %s). An address "
"can only belong to one." % (a, out[a]["device_name"], d["device_name"]))
out[a] = d
return out
def credentials():
user, pwd = os.environ.get("NET_USER"), os.environ.get("NET_PASS")
if not user or not pwd:
raise SystemExit(
"NET_USER and NET_PASS must be set in the environment.\n"
" Linux/macOS : export NET_USER=admin; read -s NET_PASS; export NET_PASS\n"
" PowerShell : $env:NET_USER='admin'; $env:NET_PASS='...'\n"
"They are read from the environment on purpose: an argument is visible "
"in 'ps auxww' and lands in shell history.")
return user, pwd, os.environ.get("NET_ENABLE") or pwd
# ----------------------------------------------------------------- transport
def slug(host, command):
"""Filename for one command's output.
A readable prefix plus a short digest of the WHOLE command. The prefix
alone is not enough -- two policy lookups differ only in the port, far past
any sensible truncation -- and the full command blew past the Windows path
limit on the first run.
"""
h = "".join(c if c.isalnum() else "-" for c in host)
keep = "".join(c if c.isalnum() else "-" for c in command)[:48].strip("-")
digest = hashlib.sha1(command.encode("utf-8")).hexdigest()[:8]
return "%s__%s__%s.txt" % (h, keep, digest)
class Transport(object):
"""Live SSH, or a replay of saved command output. Same interface either way.
Offline replay is how this was tested without a lab, and it is how you hand
a colleague the exact output a finding came from.
"""
def __init__(self, args):
self.a = args
self.sessions = {}
def run(self, host, osn, command):
if self.a.offline_dir:
path = os.path.join(self.a.offline_dir, slug(host, command))
if not os.path.exists(path):
return None
with open(path, encoding="utf-8", errors="replace") as fh:
return fh.read()
conn = self.sessions.get(host)
if conn is None:
try:
from netmiko import ConnectHandler
except ImportError:
raise SystemExit("netmiko is not installed. Run: pip install 'netmiko>=4.0'")
user, pwd, secret = credentials()
conn = ConnectHandler(device_type=NETMIKO_DRIVER[osn], host=host,
username=user, password=pwd, secret=secret,
fast_cli=False, conn_timeout=20)
self.sessions[host] = conn
out = conn.send_command(command, read_timeout=120)
if self.a.save_dir:
if not os.path.isdir(self.a.save_dir):
os.makedirs(self.a.save_dir)
with open(os.path.join(self.a.save_dir, slug(host, command)),
"w", encoding="utf-8") as fh:
fh.write(out)
return out
def close(self):
for c in self.sessions.values():
try:
c.disconnect()
except Exception:
pass
# One interface, one name. `show ip interface brief` prints Gi1/0/12 while
# `show interfaces` prints GigabitEthernet1/0/12, so without this the same
# port appears under two names in one report and reads as two ports.
# Longest prefix first, or "Gi" would claim "GigabitEthernet" before the full
# form is tried. Junos (ge-0/0/0), PAN-OS (ethernet1/3) and FortiOS already
# print their canonical form, so they are absent on purpose.
ABBREV = [("TwentyFiveGigE", ("TwentyFiveGigE", "Twe")),
("HundredGigE", ("HundredGigE", "Hu")),
("TenGigabitEthernet", ("TenGigabitEthernet", "TenGigE", "TenGig", "Te")),
("FortyGigabitEthernet", ("FortyGigabitEthernet", "FortyGigE", "Fo")),
("GigabitEthernet", ("GigabitEthernet", "GigEthernet", "Gi")),
("FastEthernet", ("FastEthernet", "Fa")),
("Port-channel", ("Port-channel", "port-channel", "Po")),
("Ethernet", ("Ethernet", "Eth", "Et")),
("Vlan", ("Vlan", "Vl")),
("Tunnel", ("Tunnel", "Tu")),
("Loopback", ("Loopback", "Lo")),
("Management", ("Management", "Mgmt", "Ma"))]
# Logical interfaces. An SVI, a tunnel or a loopback has no link, so it has no
# negotiated speed and no duplex -- what a platform prints there is the
# CONFIGURED bandwidth, used for metric arithmetic. Printing it as if it were
# a link rate is how "Vlan220 up 1G full" ends up in a report about a port
# that does not exist. Receive errors cannot appear on one either.
LOGICAL = ("vlan", "loopback", "tunnel", "port-channel", "irb", "ae", "reth",
"bvi", "virtual-template", "management")
def canon_intf(name):
"""Expand an ABBREVIATION to its full form; leave a full name alone.
The distinction matters because this name is also what goes into
`show interfaces <x>`. PAN-OS writes ethernet1/3 and Junos writes
ge-0/0/0: those ARE the full names, and "correcting" the capitalisation
sent the device a command about an interface it does not have. Expansion
is for Gi/Te/Vl, which every platform that uses them also accepts in full.
"""
if not name:
return name
for full, forms in ABBREV:
if name.lower().startswith(full.lower()) and name[len(full):][:1].isdigit():
return name # already full -- the device's own spelling
for f in forms:
if f.lower() != full.lower() and name.lower().startswith(f.lower()):
rest = name[len(f):]
if rest[:1].isdigit():
return full + rest
return name
def intf_kind(name):
""""logical" for an SVI/tunnel/loopback/aggregate, else "physical".
A LAG is listed as logical deliberately: it has member links with their
own speeds, so the aggregate's own duplex is not a fact about any cable.
"""
if not name:
return "physical"
low = canon_intf(name).lower()
return "logical" if any(low.startswith(x) for x in LOGICAL) else "physical"
# ----------------------------------------------------------------- parsers
MAC_RE = re.compile(r"\b([0-9a-fA-F]{4}\.[0-9a-fA-F]{4}\.[0-9a-fA-F]{4}|"
r"(?:[0-9a-fA-F]{2}[:-]){5}[0-9a-fA-F]{2})\b")
IP_RE = re.compile(r"\b(?:\d{1,3}\.){3}\d{1,3}\b")
# Case-insensitive on purpose: PAN-OS writes "ethernet1/3" and Junos writes
# "ge-0/0/0". A case-sensitive version silently failed to find the egress on
# both and reported the output as unrecognised.
INTF_RE = re.compile(
r"\b((?:TenGigabitEthernet|GigabitEthernet|FortyGigE|HundredGigE|"
r"Port-channel|Ethernet|Vlan|Tunnel|Loopback|"
r"Gi|Te|Fo|Hu|Twe|Fa|Eth|Et|Po|Vl|Tu|Lo|"
r"xe-|ge-|et-|ae|irb|reth|lt-|st)"
r"[0-9][\w/.:-]*)\b", re.I)
def mac_for(osn, mac):
"""The MAC table is queried in the platform's own notation.
Cisco wants aaaa.bbbb.cccc, everyone else wants aa:bb:cc:dd:ee:ff. Sending
the wrong one returns an empty table, which reads exactly like "the host is
not there" -- a wrong answer that looks like a finding.
"""
hexs = re.sub(r"[^0-9a-fA-F]", "", mac).lower()
if len(hexs) != 12:
return mac
if osn in ("ios", "iosxe", "nxos", "iosxr"):
return "%s.%s.%s" % (hexs[0:4], hexs[4:8], hexs[8:12])
return ":".join(hexs[i:i + 2] for i in range(0, 12, 2))
def parse_arp(text):
"""MAC for an address. Format varies; the MAC itself does not."""
if not text:
return None
m = MAC_RE.search(text)
return m.group(1).lower().replace("-", ":") if m else None
def parse_mac_port(text):
"""Physical port from a MAC table entry.
Deliberately takes the LAST interface-looking token on the line: every
vendor puts the port at the end, and the VLAN or the entry type sits
before it.
"""
kind, where = classify_mac_entry(text)
return where if kind == "local" else None
# A MAC learned over a fabric is not on a port of this switch. FabricPath
# prints a switch-id (1.2.65535 -- swid.ssid.LID), and VXLAN/EVPN prints the
# nve interface with a peer VTEP. Treating either as a local port is how a
# tool reports a topology that does not exist.
FABRICPATH_RE = re.compile(r"\b(\d{1,4})\.(\d{1,4})\.(\d{1,5})\b")
NVE_RE = re.compile(r"\b(nve\d+|Vx\d+|vxlan\d*)\b", re.I)
PEER_RE = re.compile(r"(?:peer[- ]?ip|peer|via)\s*[:=]?\s*((?:\d{1,3}\.){3}\d{1,3})", re.I)
def classify_mac_entry(text):
"""(kind, where). kind is "local", "fabricpath", "vxlan" or None.
Checked before the port is read, because on a fabric there is no port to
read and the honest answer -- which leaf to go and look at -- is in the
switch-id or the VTEP address instead.
"""
if not text:
return None, None
for line in text.split(chr(10)):
if not MAC_RE.search(line):
continue
m = NVE_RE.search(line)
if m:
inline = re.search(r"\((((\d{1,3}\.){3}\d{1,3}))\)", line)
peer = PEER_RE.search(line) or PEER_RE.search(text)
if inline:
return "vxlan", inline.group(1)
return "vxlan", (peer.group(1) if peer else m.group(1))
hits = INTF_RE.findall(line)
if hits:
return "local", canon_intf(hits[-1])
# Only once no interface was found: a switch-id is three dotted
# numbers, and so is plenty of other output.
f = FABRICPATH_RE.search(line)
if f:
return "fabricpath", "%s.%s" % (f.group(1), f.group(2))
return None, None
def parse_po_members(text):
"""[(member, state)] from a port-channel summary.
The state letters matter as much as the names: a member that is (s)
suspended or (D) down is carrying nothing, and a flow hashed to a
suspended member is a flow that is being dropped by a port-channel the
aggregate counters call healthy.
"""
out = []
if not text:
return out
for line in text.split(chr(10)):
for m in re.finditer(r"(" + INTF_RE.pattern + r")\s*\(([A-Za-z]{1,2})\)", line):
name = canon_intf(m.group(1))
if intf_kind(name) == "physical":
out.append((name, m.group(3)))
if not out: # platforms that print no state letters
for line in text.split(chr(10)):
for hit in INTF_RE.findall(line):
nm = canon_intf(hit)
if intf_kind(nm) == "physical" and nm not in [x[0] for x in out]:
out.append((nm, ""))
return out
def parse_ecmp(text):
"""How many next hops the route offers.
exact-route says which one THIS flow takes. It does not say that four
others exist, so a report can be clean about one fifth of the topology
and read as clean about all of it.
"""
if not text:
return 0
hops = set(re.findall(r"(?:via|next-hop|nexthop|to)\s+((?:\d{1,3}\.){3}\d{1,3})",
text, re.I))
# IOS and IOS-XE write a route descriptor block per path, as
# "* 10.0.0.1, from 10.0.0.1, 1w2d ago, via Port-channel1" -- there "via"
# introduces the INTERFACE, so the pattern above finds nothing on the one
# format most of this catalogue runs against.
hops |= set(re.findall(r"^\s*\*?\s*((?:\d{1,3}\.){3}\d{1,3}),\s", text, re.M))
return len(hops)
def parse_neighbor(text):
"""(hostname, mgmt-ip, platform-guess) from CDP or LLDP detail.
This is what removes the need to probe a device to learn what it is: the
CURRENT device already told us, before we connect.
"""
if not text:
return None, None, None
name = None
for pat in (r"Device ID:\s*(\S+)", r"System Name:\s*(\S+)",
r"System name\s*:\s*(\S+)", r"Chassis id:\s*(\S+)"):
m = re.search(pat, text, re.I)
if m:
name = m.group(1).split(".")[0].strip('"')
break
ip = None
for pat in (r"IP address:\s*(" + IP_RE.pattern + r")",
r"Management Address(?:es)?\s*:?\s*(?:IP:\s*)?(" + IP_RE.pattern + r")",
r"mgmt-ip\s*[:=]\s*(" + IP_RE.pattern + r")"):
m = re.search(pat, text, re.I)
if m:
ip = m.group(1)
break
if ip is None:
m = IP_RE.search(text)
ip = m.group(0) if m else None
return name, ip, guess_platform(text)
PLATFORM_HINTS = [
("nxos", r"\bNX-OS\b|\bNexus\b"),
("iosxr", r"\bIOS[- ]XR\b"),
("iosxe", r"\bIOS[- ]XE\b"),
("eos", r"\bArista\b|\bEOS\b"),
("junos", r"\bJuniper\b|\bJUNOS\b|\bSRX\b|\bMX\d|\bEX\d"),
("panos", r"\bPalo Alto\b|\bPAN-OS\b"),
("fortios", r"\bFortinet\b|\bFortiGate\b|\bFortiOS\b"),
("asa", r"\bASA\b|\badaptive security\b"),
("ios", r"\bCisco IOS\b|\bcisco WS-C|\bcisco C\d"),
]
def guess_platform(text):
if not text:
return None
for osn, pat in PLATFORM_HINTS:
if re.search(pat, text, re.I):
return osn
return None
def parse_route_egress(text):
"""Egress interface from a route lookup.
Returns None rather than a guess when the format is not recognised; the
caller turns that into COULD_NOT_VERIFY with the raw output attached.
"""
if not text:
return None, None
nh = None
m = re.search(r"via\s+(" + IP_RE.pattern + r")", text)
if m:
nh = m.group(1)
m = re.search(r"(?:via\s+\S+,\s*|,\s*|interface\s+|egress-interface\s+)"
r"(" + INTF_RE.pattern + r")", text, re.I)
if m:
return m.group(1), nh
hits = INTF_RE.findall(text)
return (hits[-1] if hits else None), nh
MODEL_PATTERNS = [
r"Model [Nn]umber\s*:\s*(\S+)",
r"cisco\s+(\S+)\s+\(.*?\)\s+processor",
r"cisco\s+(Nexus\S*(?:\s+\S+)?)\s+Chassis",
r"cisco\s+(ASR\S+|NCS\S+|CRS\S+)",
r"\bArista\s+(\S+)",
r"^Model:\s*(\S+)",
r"^model:\s*(\S+)",
r"Version:\s*(FortiGate-\S+)",
r"Hardware:\s+(\S+?),",
r"\bcisco\s+(WS-\S+|C9\d\S*|ISR\S*|CSR\S*)",
]
# The role changes how a line reads: a saturated uplink on an access switch
# and on a core router are different conversations.
ROLE_BY_OS = {"panos": "firewall", "fortios": "firewall", "asa": "firewall"}
ROLE_HINTS = [
("firewall", r"(ASA|FPR|PA-\d|FortiGate|SRX)"),
("switch", r"(WS-C|C9[23]\d\d|Nexus|DCS-|EX\d|QFX)"),
("router", r"(ISR|ASR|CSR|NCS|CRS|MX\d|C8\d\d\d)"),
]
def parse_vrf_list(text):
"""VRF or routing-instance names a device holds.
Deliberately conservative: anything that does not look like a name is
dropped rather than tried, because probing an invented table name wastes
a round trip per hop and pollutes the output with errors.
"""
if not text:
return []
names, skip = [], {"name", "default", "vrf", "interfaces", "protocols",
"instance", "type", "state", "master", "mgmt"}
for line in text.split(chr(10))[1:]:
tok = line.split()
if not tok:
continue
cand = tok[0].strip()
if (cand.lower() in skip or len(cand) > 40
or not re.match(r"^[A-Za-z][\w.:-]*$", cand)):
continue
if cand not in names:
names.append(cand)
return names[:20]
def parse_hostname(text):
"""The device's own name. It is in its configuration and its prompt, so
an inventory is not needed to print something readable."""
if not text:
return None
for pat in (r"^hostname\s+(\S+)", r"^\s*hostname:\s*(\S+)",
r"^(\S+)\s+uptime is", r"^Hostname:\s*(\S+)",
r"Host name:\s*(\S+)", r"^Model:\s*\S+\s*$"):
m = re.search(pat, text, re.I | re.M)
if m and m.lastindex:
return m.group(1).strip().rstrip(",;")
return None
def parse_model(text, osn=None):
"""(model, role). Either may be None, and None prints as a dash.
Inventing a model is worse than leaving it blank."""
role = ROLE_BY_OS.get(osn)
if not text:
return None, role
model = None
for pat in MODEL_PATTERNS:
m = re.search(pat, text, re.I | re.M)
if m:
model = m.group(1).strip().rstrip(",;")
break
if role is None and model:
for r, pat in ROLE_HINTS:
if re.search(pat, model, re.I):
role = r
break
if role is None:
m = re.search(r"Capabilities:\s*([^\n]*)", text, re.I)
if m:
caps = m.group(1).lower()
role = "router" if "router" in caps else (
"switch" if ("switch" in caps or "bridge" in caps) else None)
return model, role
def parse_interface(text):
"""Everything an engineer reads off one interface, and nothing invented.
A field that could not be read stays absent and prints as a dash. It must
never default to zero: "0 CRC errors" and "I could not find the CRC
counter" are opposite findings, and a tool that conflates them says a
broken link is healthy.
"""
if not text:
return {}
t, out = text, {}
# -- state ---------------------------------------------------------
m = re.search(r"line protocol is (\w+(?: \w+)?)", t, re.I)
if m:
out["status"] = m.group(1).lower()
else:
m = re.search(r"\b(?:is )?(administratively down|up|down)\b", t, re.I)
if m:
out["status"] = m.group(1).lower()
m = re.search(r"\b(Full|Half)-duplex", t, re.I)
if m:
out["duplex"] = m.group(1).lower()
# -- MTU. "Works for ping, fails for the application" is an MTU black
# hole, and the minimum along a path is only knowable per interface.
m = re.search(r"\bMTU\s+(\d{3,5})\b", t, re.I)
if m:
out["mtu"] = int(m.group(1))
# -- bandwidth. IOS prints Kbit; others print Mb/s or Gb/s ----------
m = re.search(r"\bBW\s+(\d+)\s*Kbit", t, re.I)
if m:
out["speed"] = int(m.group(1))
else:
m = re.search(r"(\d+)\s*([MG])b/s", t, re.I)
if m:
out["speed"] = int(m.group(1)) * (1000000 if m.group(2).upper() == "G" else 1000)
else:
m = re.search(r"Speed[ :]+(\d+)", t, re.I)
if m:
out["speed"] = int(m.group(1)) * 1000
# -- load ----------------------------------------------------------
m = re.search(r"input rate\s+(\d+)\s*bits/sec", t, re.I)
if m:
out["in_bps"] = int(m.group(1))
m = re.search(r"output rate\s+(\d+)\s*bits/sec", t, re.I)
if m:
out["out_bps"] = int(m.group(1))
# -- counters. Absent stays absent ---------------------------------
COUNTERS = (
("in_err", r"(\d+) input errors"),
("crc", r"(\d+) CRC"),
("frame", r"(\d+) frame\b"),
("overrun", r"(\d+) overrun"),
("ignored", r"(\d+) ignored"),
("runts", r"(\d+) runts"),
("giants", r"(\d+) giants"),
("out_err", r"(\d+) output errors"),
("collisions", r"(\d+) collisions"),
("late_coll", r"(\d+) late collision"),
("resets", r"(\d+) interface resets"),
("drops", r"output drops:?\s*(\d+)"),
("carrier", r"(\d+) lost carrier"),
("babbles", r"(\d+) babbles"),
)
for key, pat in COUNTERS:
m = re.search(pat, t, re.I)
if m:
out[key] = int(m.group(1))
# Input queue: size/max/drops/flushes -- queue drops are the punt path
# running out, which is a different fault from output congestion.
m = re.search(r"Input queue:\s*\d+/\d+/(\d+)/(\d+)", t, re.I)
if m:
out["inq_drops"], out["inq_flushes"] = int(m.group(1)), int(m.group(2))
# -- how old are those counters ------------------------------------
# Without this, a total is not a rate. 48,000 drops since Tuesday and
# since 2019 are different findings.
m = re.search(r"Last clearing of .*counters\s+(.+?)\s*$", t, re.I | re.M)
if m:
out["counters_age"] = m.group(1).strip()
# -- when it last changed state ------------------------------------
for pat in (r"last flapped\s*:?\s*(.+?)\s*$", # Junos, EOS
r"Last link flapped\s*:?\s*(\S+)", # NX-OS
r"Time since last interface status change:\s*(\S+)"): # EOS
m = re.search(pat, t, re.I | re.M)
if m:
out["last_flap"] = m.group(1).strip()
break
m = re.search(r"is up,? (?:line protocol is up )?\(?(?:connected)?\)?.*?"
r"for\s+(\S+)", t, re.I)
if m:
out["up_since"] = m.group(1)
# Traffic recency is not uptime, but on a quiet link it is the only clue
# the platform gives.
m = re.search(r"Last input\s+(\S+?),\s*output\s+(\S+?),", t, re.I)
if m:
out["last_in"], out["last_out"] = m.group(1).rstrip(","), m.group(2).rstrip(",")
return out
def utilisation(info):
"""Percent of nominal bandwidth, or None when either number is missing.
None, not zero: an interface whose speed could not be read is unknown,
not idle, and printing 0% on a saturated link is the kind of wrong answer
that keeps a ticket open for a week.
"""
try:
bw_bps = int(info["speed"]) * 1000 # speed is normalised to Kbit
worst = max(info.get("in_bps", 0), info.get("out_bps", 0))
return round(worst * 100.0 / bw_bps, 1) if bw_bps else None
except (KeyError, ValueError, TypeError, ZeroDivisionError):
return None
FLAP_LOG = re.compile(
r"(?P<when>\w{3}\s+\d+\s[\d:.]+|\d{4}-\d\d-\d\dT[\d:.]+|\*?\w{3}\s+\d+\s+[\d:]+)"
r"[^%]*%(?:LINK|LINEPROTO|ETHPORT|ETH)[-\w]*:?\s*(?P<what>.*)", re.I)
def parse_flaps(text, intf):
"""Link up/down events for one interface, newest first.
This exists because `show interfaces` on IOS and IOS-XE carries NO
last-flap field at all -- only NX-OS, EOS and Junos print one. On Cisco
classic the syslog buffer is the only place the answer lives, and leaving
it unread meant the most useful field on a flapping path was blank on the
most common platform.
"""
if not text:
return []
events = []
short = intf_short(intf)
for line in text.split(chr(10)):
if intf not in line and (not short or short not in line):
continue
m = FLAP_LOG.search(line)
if m:
events.append((m.group("when").strip().rstrip(":"), m.group("what").strip()[:90]))
elif re.search(r"\b(up|down)\b", line, re.I) and "%" in line:
events.append(("", line.strip()[:110]))
return list(reversed(events))[:6]
def intf_short(intf):
"""Gi1/0/1 and GigabitEthernet1/0/1 are the same port, and syslog picks
whichever it likes. Match on the number, which both share."""
if not intf:
return ""
m = re.search(r"([0-9][\w/.:-]*)$", intf)
return m.group(1) if m else ""
def interpret(info):
"""[(cause, meaning, the counters that support it)].
Each cause carries ONLY its own counters. The first version quoted every
counter on every finding, so "physical" and "punt" printed identical
numbers and the reader could not tell which evidence belonged to which
conclusion -- which defeats the point of interpreting them at all.
"""
out = []
def add(kind, keys, meaning):
present = [(k, info[k]) for k in keys if info.get(k)]
if present:
out.append((kind, meaning, present))
add("physical", ("crc", "runts", "frame", "giants"),
"CRC, runts or framing errors point at the cable, the optic or a "
"speed/duplex mismatch. Load has nothing to do with it")
add("duplex", ("late_coll", "collisions"),
"late collisions on a switched link mean a DUPLEX MISMATCH, almost "
"always half on one end")
add("congestion", ("drops",),
"output drops are the queue overflowing. Not an error: the interface "
"is simply full at times")
add("punt", ("inq_drops", "inq_flushes"),
"input-queue drops are the control plane falling behind, which is a "
"different fault from a full link")
add("receive", ("overrun", "ignored"),
"overruns and ignores mean the receiver could not take the packets -- "
"buffers or CPU, not the cable")
add("flapping", ("resets", "carrier"),
"interface resets and lost carrier count the times the link was bounced")
if info.get("duplex") == "half":
out.append(("duplex", "half duplex on a modern link is a negotiation "
"failure, not a setting", [("duplex", "half")]))
return out
STATE_WORDS = {"active", "standby", "init", "listen", "speak", "learn",
"master", "backup"}
def parse_hsrp_active(text, local_name=None):
"""(is_this_device_active, active_ip), or (None, None) when unparsed.
Read by COLUMN, not by searching for the word "Active": in
`show standby brief` that word is also the heading of the column holding
the active peer's address, so a text search matches the header of every
standby's output and concludes nothing. The first version did exactly
that and silently kept walking from the standby.
Layout: Interface Grp Pri P State Active Standby VirtualIP
so the token after the state is the active peer, or "local" when this
device holds it.
"""
if not text:
return None, None
for line in text.split(chr(10)):
low = line.lower()
if "interface" in low and ("grp" in low or "group" in low):
continue
toks = line.split()
idx = next((i for i, t in enumerate(toks) if t.lower() in STATE_WORDS), None)
if idx is None:
continue
state = toks[idx].lower()
if state in ("active", "master"):
return True, None
nxt = toks[idx + 1] if idx + 1 < len(toks) else ""
if nxt.lower() == "local":
return True, None
return False, (nxt if IP_RE.fullmatch(nxt) else None)
return None, None
def parse_traceroute(text):
"""[(hop_number, ip)]. Unresponsive hops are kept as (n, None), because a
gap in the path is information rather than something to drop."""
hops = []
if not text:
return hops
for line in text.split(chr(10)):
m = re.match(r"\s*(\d{1,2})\s+(.*)$", line)
if not m:
continue
ips = IP_RE.findall(m.group(2))
hops.append((int(m.group(1)), ips[0] if ips else None))
return hops
# Every vendor words an empty session table differently, and getting this
# wrong is not cosmetic: the walker concludes sessions exist, skips the policy
# simulation, and reports NOT DETERMINED on a firewall that would have
# answered. "No Active Sessions" defeated the first version of this.
EMPTY_SESSION = re.compile(
r"no\s+(?:active\s+|matching\s+|running\s+)?sessions?\b"
r"|\b0\s+sessions?\b"
r"|total\s+session\s*[:=]?\s*0\b"
r"|^\s*$", re.I | re.M)
DENY_WORDS = re.compile(r"\b(deny|drop|denied|dropped|block|blocked)\b", re.I)
ALLOW_WORDS = re.compile(r"\b(allow|permit|accept|allowed|permitted)\b", re.I)
def parse_policy(text):
"""(action, rule_name) from a policy lookup, on any platform.
Returns (None, None) when the output is not recognised. An unparsed
policy answer must never become a verdict: "I could not read it" and
"the firewall permits this" are opposite findings.
"""
if not text:
return None, None
rule = None
# PAN-OS opens with "rule-name; index: 14" { so the name runs to the
# first quote or semicolon; taking the whole quoted string would report
# the index as part of the rule name.
for pat in (r'^\s*"([^";]+)', r"Rule:\s*(\S+)", r"policy name\s*[:=]\s*(\S+)",
r"Policy:\s*(\S+)", r"rule\s+id\s+(\d+)", r"policy_id=(\d+)",
r"Matched rule:\s*(.+?)\s*$", r"policy\s+id\s*[:=]\s*(\d+)",
r"access-group\s+(\S+)", r"access-list\s+(\S+)"):
m = re.search(pat, text, re.I | re.M)
if m:
# Trailing punctuation belongs to the surrounding syntax, not the
# name: Junos prints "Policy: allow-sql," and reporting the comma
# sends someone looking for a rule that does not exist.
rule = m.group(1).strip().rstrip(",;:\"'")
break
action = None
if DENY_WORDS.search(text):
action = "DENY"
if ALLOW_WORDS.search(text) and action is None:
action = "ALLOW"
return action, rule
# ----------------------------------------------------------------- model
class Hop(object):
def __init__(self, order, kind, name, ip, osn):
self.order = order # position in the path
self.kind = kind # l2 / l3 / firewall / endpoint
self.name = name
self.ip = ip
self.osn = osn
self.ingress = None
self.egress = None
self.health = {}
self.util = None
self.flaps = [] # link up/down events from syslog
self.model = None
self.role = None # switch / router / firewall
self.downstream = None # the port facing the source host
self.health_down = {}
self.util_down = None
self.vrf = None # the table the route lookup answered from
self.vrf_source = None # inventory / global / discovered
self.ingress = None # the interface the flow arrives on
self.zone_in = None # security zone, firewalls only
self.svi_in = None # the SVI a physical ingress is routed through
self.svi_out = None
self.next_hop = None # the L3 next hop, to resolve an egress SVI
self.members = [] # (port, state, health, util) of a port-channel
self.ecmp = 0 # how many next hops the route offered
self.asymmetric = None # the reverse-path interface, when it differs
self.zone_out = None
self.health_in = {}
self.util_in = None
self.policy_action = None
self.policy_rule = None
self.policy_source = None # "session" or "simulation"
self.notes = [] # COULD_NOT_VERIFY reasons, raw snippets
class Walker(object):
def __init__(self, t, inv, args):
self.t = t
self.inv = inv
self.a = args
self.hops = []
self.findings = []
self.unchecked = []
self.src_ip = None # for the reverse FIB lookup
self.path_mtu = None
self.mtu_at = []
# -- helpers ------------------------------------------------------
def dev(self, host, osn=None):
d = self.inv.get(host)
if d:
return d["device_name"], d["os"], d.get("vrf")
return host, osn, None
def autodetect(self, addr):
"""Ask the device what it is, when CDP/LLDP did not reach it and the
inventory does not list it.
This is the fallback that makes the inventory optional. It costs one
extra session per unknown hop, which is why CDP/LLDP is still tried
first -- that answer arrives before the connection does.
"""
if self.a.offline_dir:
return None
try:
from netmiko import SSHDetect
except ImportError:
return None
try:
user, pwd, _ = credentials()
guess = SSHDetect(device_type="autodetect", host=addr,
username=user, password=pwd,
conn_timeout=15).autodetect()
except Exception:
return None
for osn, driver in NETMIKO_DRIVER.items():
if guess == driver:
return osn
return None
def connect_addr(self, addr):
"""Where to open the session for a device seen at `addr`.
A traceroute hands back the hop's ingress interface, and plenty of
networks only accept SSH on the management interface. Matching on one
address and connecting to another is the whole point of the `mgmt`
and `addresses` fields.
"""
d = self.inv.get(addr)
# .get, not [] -- a record synthesised from --gateway-os never went
# through load_inventory and has no mgmt key.
return d.get("mgmt", addr) if d else addr
def ask(self, host, osn, key, **kw):
tpl = CMD.get(osn, {}).get(key)
if not tpl:
return None, "%s has no '%s' equivalent" % (osn, key)
try:
cmd = tpl.format(**kw)
except KeyError as exc:
return None, "missing %s for '%s'" % (exc, key)
out = self.t.run(self.connect_addr(host), osn, cmd)
if out is None:
return None, "no output for: %s" % cmd
return out, None
def finding(self, sev, hop, what, detail):
self.findings.append((sev, hop, what, detail))
# -- phase 1: the gateway, and HSRP/VRRP -------------------------
def resolve_active_gateway(self, gw_host):
name, osn, vrf = self.dev(gw_host)
if osn is None:
osn = self.a.gateway_os or self.autodetect(gw_host)
if osn is None:
raise SystemExit(
"could not work out what %s is. SSH autodetection did not answer, so "
"either the credentials are wrong, the device refuses SSH from here, "
"or it is a platform this tool does not know. Pass --gateway-os to "
"say outright: %s" % (gw_host, ", ".join(sorted(NETMIKO_DRIVER))))
for key in ("hsrp", "vrrp"):
out, err = self.ask(gw_host, osn, key, ip="", vrf=vrf or "")
if out is None:
continue
is_local, active_ip = parse_hsrp_active(out)
if is_local is None:
self.unchecked.append(
"%s: %s output was not recognised, so which unit is active is "
"unknown. If this is a redundant pair, the path below may be the "
"standby's and not the one traffic takes." % (name, key.upper()))
continue
if is_local:
self.hops.append(Hop(0, "l3", name + " (active " + key.upper() + ")",
gw_host, osn))
return gw_host, osn
if active_ip:
self.finding("INFO", name,
"entered the standby, moved to the active",
"%s is the %s standby. Everything below was measured from "
"%s, which is where the traffic actually goes."
% (name, key.upper(), active_ip))
a_name, a_os, _ = self.dev(active_ip, osn)
self.hops.append(Hop(0, "l3", a_name + " (active " + key.upper() + ")",
active_ip, a_os or osn))
return active_ip, (a_os or osn)
self.unchecked.append(
"%s is a %s standby but the active's address could not be read, so the "
"walk continued from the standby." % (name, key.upper()))
self.hops.append(Hop(0, "l3", name, gw_host, osn))
return gw_host, osn
# -- phase 2: downstream to the host ------------------------------
def walk_to_source(self, gw_host, gw_os, host_ip):
"""Downstream from the gateway to the source host.
ARP gives the MAC, the MAC table gives the port, the neighbour on
that port gives the next switch. A port with no neighbour is where the
host lives. This is the only way to see L2: switches never appear in a
traceroute, because they do not decrement TTL."""
name, osn, vrf = self.dev(gw_host, gw_os)
out, err = self.ask(gw_host, osn, "arp_vrf" if vrf else "arp",
ip=host_ip, vrf=vrf or "")
mac = parse_arp(out)
if not mac:
self.unchecked.append(
"%s has no ARP entry for %s%s. The host may be silent, in another "
"VLAN, or behind a different gateway -- the L2 segment was not walked."
% (name, host_ip, (" in vrf " + vrf) if vrf else ""))
return
cur_host, cur_os, order = gw_host, osn, -1
seen = set()
while True:
cname, cos, _ = self.dev(cur_host, cur_os)
out, err = self.ask(cur_host, cos, "mac", mac=mac_for(cos, mac))
port = parse_mac_port(out)
if not port:
self.unchecked.append(
"%s: could not read a port for %s from the MAC table%s"
% (cname, mac, (" (%s)" % err) if err else ""))
return
# Finding the gateway's downstream port through a MAC table does
# NOT make it a switch -- it routes. Listing it twice, once
# tagged L2, was simply wrong. Attach the port to the hop that
# already exists for it.
existing = next((x for x in self.hops if x.ip == cur_host), None)
if existing is not None:
probe = Hop(existing.order, existing.kind, existing.name, cur_host, cos)
probe.egress = canon_intf(port)
self.fill_identity(probe)
self.fill_health(probe)
existing.downstream = canon_intf(port)
existing.health_down = probe.health
existing.util_down = probe.util
existing.model = existing.model or probe.model
existing.role = existing.role or probe.role
# And the name. The L2 walk reaches the gateway before the
# traceroute does, so without this its model was filled in
# here and fill_identity then short-circuited on the later
# call, leaving the one device that matters most showing as a
# bare address.
if existing.ip and existing.name.startswith(existing.ip) and not probe.name.startswith(existing.ip):
existing.name = probe.name + existing.name[len(existing.ip):]
else:
h = Hop(order, "l2", cname, cur_host, cos)
h.egress = canon_intf(port)
self.fill_identity(h)
self.fill_health(h)
self.hops.insert(0, h)
nb_out, _ = self.ask(cur_host, cos, "cdp", intf=port)
if not nb_out:
nb_out, _ = self.ask(cur_host, cos, "lldp", intf=port)
nname, nip, nos = parse_neighbor(nb_out)
if not nip:
ep = Hop(order - 1, "endpoint", "source", host_ip, None)
ep.ingress = canon_intf(port)
ep.notes.append("attached to %s %s (access port, no CDP/LLDP neighbour)"
% (cname, canon_intf(port)))
self.hops.insert(0, ep)
return
if nip in seen:
self.unchecked.append("L2 walk looped at %s; stopped" % nip)
return
seen.add(nip)
inv_os = self.inv.get(nip, {}).get("os")
cur_host, cur_os, order = nip, (inv_os or nos), order - 1
if cur_os is None:
self.unchecked.append(
"%s is on the L2 path but neither CDP/LLDP nor the inventory "
"identified its platform, so it was not inspected." % nip)
return
def walk_to_dest_host(self, dst):
"""From the destination's gateway down to the port hostB is plugged
into.
The mirror of walk_to_source, and it was missing: the far end of the
path stopped at the gateway, but a host is not plugged into its
gateway. The access switch between them never appears in a traceroute
-- switches do not decrement TTL -- so it has to be walked the same
way, by MAC.
"""
gw = next((h for h in reversed(self.hops)
if h.kind in ("l3", "firewall") and h.ip and h.osn), None)
if gw is None:
return
vrf = gw.vrf if gw.vrf_source in ("inventory", "discovered") else None
out, err = self.ask(gw.ip, gw.osn, "arp_vrf" if vrf else "arp",
ip=dst, vrf=vrf or "")
mac = parse_arp(out)
if not mac:
self.unchecked.append(
"%s has no ARP entry for %s%s, so the L2 segment in front of the "
"destination was not walked. The host may be down, silent, or behind "
"a different gateway." % (gw.name, dst, (" in vrf " + vrf) if vrf else ""))
return
cur_host, cur_os, order = gw.ip, gw.osn, (self.hops[-1].order if self.hops else 0)
seen = set()
while True:
cname, cos, _ = self.dev(cur_host, cur_os)
out, err = self.ask(cur_host, cos, "mac", mac=mac_for(cos, mac))
kind, where = classify_mac_entry(out)
if kind == "fabricpath":
self.unchecked.append(
"%s learned %s over FabricPath from switch-id %s, not on a local "
"port. The host is behind that switch -- go and look there; this "
"walk cannot follow it." % (cname, mac, where))
return
if kind == "vxlan":
self.unchecked.append(
"%s learned %s over VXLAN from VTEP %s, not on a local port. The "
"host is behind that VTEP -- go and look there; this walk cannot "
"follow it." % (cname, mac, where))
return
if kind != "local":
self.unchecked.append(
"%s: could not read a port for %s from the MAC table%s"
% (cname, mac, (" (%s)" % err) if err else ""))
return
port = where
existing = next((x for x in self.hops if x.ip == cur_host), None)
if existing is not None:
# The gateway already has a hop. Its port toward the
# destination IS its egress -- do not list the device twice.
if not existing.egress:
existing.egress = port
self.fill_health(existing)
else:
order += 1
h = Hop(order, "l2", cname, cur_host, cos)
h.egress = port
self.fill_identity(h)
self.fill_health(h)
self.hops.append(h)
nb_out, _ = self.ask(cur_host, cos, "cdp", intf=port)
if not nb_out:
nb_out, _ = self.ask(cur_host, cos, "lldp", intf=port)
nname, nip, nos = parse_neighbor(nb_out)
if not nip:
ep = Hop(order + 1, "endpoint", "destination", dst, None)
last = next((x for x in reversed(self.hops) if x.ip == cur_host), None)
ep.notes.append("attached to %s %s (access port, no CDP/LLDP neighbour)"
% ((last.name if last else cname), port))
self.hops.append(ep)
return
if nip in seen:
self.unchecked.append("L2 walk toward the destination looped at %s; stopped" % nip)
return
seen.add(nip)
inv_os = self.inv.get(nip, {}).get("os")
cur_host, cur_os = nip, (inv_os or nos)
if cur_os is None:
self.unchecked.append(
"%s is on the L2 path in front of the destination but neither "
"CDP/LLDP nor the inventory identified its platform, so it was "
"not inspected." % nip)
return
def check_links(self):
"""Compare the two ends of each link, and compute the path MTU.
Both of these are only answerable ACROSS devices, which is why a
per-device walk never found them:
A DUPLEX MISMATCH does not show as "mismatch" anywhere. It shows as
late collisions on the half-duplex end and CRC on the full-duplex
end, and either one alone reads as a bad cable. Only the pair names
the fault.
AN MTU MISMATCH is invisible until something sends a large packet.
"Ping works, the application hangs" is this, and for TCP/1433
returning large result sets it is a weekly occurrence. The minimum
along the path is what the flow actually gets.
"""
prev = None
mtus = []
for h in self.hops:
if h.kind == "endpoint":
continue
for intf, hl in ((h.ingress, h.health_in), (h.egress, h.health)):
if intf and hl.get("mtu"):
mtus.append((h.name, intf, hl["mtu"]))
if prev is not None and prev.egress and h.ingress:
a, b = prev.health, h.health_in
link = "%s %s <-> %s %s" % (prev.name, prev.egress, h.name, h.ingress)
da, db = a.get("duplex"), b.get("duplex")
if da and db and da != db:
self.finding("HIGH", prev.name, "duplex mismatch on %s" % link,
"%s is %s and %s is %s. This shows as late collisions "
"on one end and CRC on the other, and neither end alone "
"names it" % (prev.egress, da, h.ingress, db))
sa, sb = a.get("speed"), b.get("speed")
lag = any(intf_kind(x) == "logical" for x in (prev.egress, h.ingress))
if sa and sb and sa != sb and not lag:
self.finding("HIGH", prev.name, "speed mismatch on %s" % link,
"%s is %s and %s is %s" % (prev.egress, fmt_rate(sa),
h.ingress, fmt_rate(sb)))
ma, mb = a.get("mtu"), b.get("mtu")
if ma and mb and ma != mb:
self.finding("HIGH", prev.name, "MTU mismatch on %s" % link,
"%s is %d and %s is %d. Large packets are dropped in "
"one direction; small ones, including ping, are not"
% (prev.egress, ma, h.ingress, mb))
prev = h
if mtus:
lo = min(x[2] for x in mtus)
self.path_mtu = lo
self.mtu_at = [x for x in mtus if x[2] == lo]
if lo < 1500:
self.finding("MEDIUM", self.mtu_at[0][0],
"the path MTU is %d, not 1500" % lo,
"narrowest at %s. If anything in the path drops the ICMP "
"fragmentation-needed message, PMTUD fails silently and "
"the symptom is a connection that opens and then hangs"
% ", ".join("%s %s" % (x[0], x[1]) for x in self.mtu_at))
# -- phase 3: upstream to the destination -----------------------------------
def walk_to_dst(self, gw_host, gw_os, src, dst, dport, proto):
name, osn, vrf = self.dev(gw_host, gw_os)
out, err = self.ask(gw_host, osn, "traceroute", dst=dst)
if out is None:
raise SystemExit("could not run traceroute from %s: %s" % (name, err))
# The gateway is the first device the traffic crosses and it was never
# inspected for its UPSTREAM interface: it is the traceroute's origin,
# not one of its hops, so it fell through the loop below. Its uplink is
# as able to be the saturated one as any other.
gw_hop = next((x for x in self.hops if x.ip == gw_host and x.kind != "l2"), None)
if gw_hop is not None:
self.inspect_l3(gw_hop, src, dst, dport, proto)
hops = parse_traceroute(out)
if not hops:
self.unchecked.append(
"%s: the traceroute output was not recognised, so the upstream path "
"is unknown." % name)
return
self.unchecked.append(
"The traceroute probe is not your flow. It is UDP or ICMP while the flow "
"is %s/%s, so under ECMP it can hash to a different path than the "
"application takes." % (proto.upper(), dport))
for n, ip in hops:
if ip is None:
h = Hop(n, "l3", "(no reply)", None, None)
h.notes.append("hop did not answer. Common where ICMP is rate-limited "
"or filtered; not a fault on its own")
self.hops.append(h)
continue
inv = self.inv.get(ip)
hos = inv["os"] if inv else None
hname = inv["device_name"] if inv else ip
if hos is None:
hos = self.autodetect(ip)
h = Hop(n, "l3", hname, ip, hos)
if hos is None:
h.notes.append(
"could not be identified. CDP/LLDP did not reach it, it is not "
"in an inventory, and SSH autodetection did not answer -- most "
"often a device you do not own, or one that refuses SSH from "
"here. Not inspected")
self.unchecked.append("hop %d (%s) was not inspected: unknown platform" % (n, ip))
self.hops.append(h)
continue
self.inspect_l3(h, src, dst, dport, proto)
self.hops.append(h)
if h.policy_action == "DENY":
self.finding("HIGH", hname, "traffic is denied here",
"rule %s denies %s -> %s:%s. This is where the flow stops."
% (h.policy_rule or "(unnamed)", src, dst, dport))
def resolve_vrf(self, h, dst):
"""(vrf, how, route_output). The table this destination is actually
reachable in on THIS device.
Order: what the inventory says, then the global table, then every VRF
the device admits to having. If more than one answers, say so -- the
same prefix legitimately exists in several VRFs, and picking one is a
coin flip dressed up as a finding.
"""
_, _, inv_vrf = self.dev(h.ip, h.osn)
if inv_vrf:
out, _ = self.ask(h.ip, h.osn, "route_vrf", dst=dst, vrf=inv_vrf, src=dst)
return inv_vrf, "inventory", out
out, _ = self.ask(h.ip, h.osn, "route", dst=dst, vrf="default", src=dst)
if out and parse_route_egress(out)[0]:
return None, "global", out
listing, _ = self.ask(h.ip, h.osn, "vrf_list")
names = parse_vrf_list(listing)
if not names:
return None, "global", out
hits = []
for name in names:
o, _ = self.ask(h.ip, h.osn, "route_vrf", dst=dst, vrf=name, src=dst)
if o and parse_route_egress(o)[0]:
hits.append((name, o))
if len(hits) == 1:
return hits[0][0], "discovered", hits[0][1]
if len(hits) > 1:
found = ", ".join(x[0] for x in hits)
h.notes.append(
"%s is reachable in MORE THAN ONE table here (%s). Which one "
"carries this flow depends on the ingress interface, so the "
"egress below was NOT resolved -- tell the tool which VRF with "
"a vrf: field in the inventory" % (dst, found))
return None, "ambiguous", None
return None, "global", out
def inspect_l3(self, h, src, dst, dport, proto):
vrf, how, out = self.resolve_vrf(h, dst)
h.vrf, h.vrf_source = vrf, how
if how == "ambiguous":
return
egress, nh = parse_route_egress(out)
h.ecmp = parse_ecmp(out)
# On a CONNECTED route there is no next hop, and the peer on the
# other end of the egress is the destination host itself -- which
# is exactly what an SVI has to be resolved through.
h.next_hop = nh or dst
if egress is None:
h.notes.append("route lookup output not recognised")
h.egress = canon_intf(egress)
# ECMP: the probe may have hashed differently than the flow, so resolve
# the real egress with the 5-tuple where the platform can.
ex, _ = self.ask(h.ip, h.osn, "exact", src=src, dst=dst)
if ex:
e2 = INTF_RE.search(ex)
if e2 and egress and canon_intf(e2.group(1)) != canon_intf(egress):
if h.ecmp > 1:
h.notes.append("ECMP: %d equal-cost paths; exact-route puts THIS "
"flow on %s. The others were not examined"
% (h.ecmp, canon_intf(e2.group(1))))
else:
h.notes.append("the route lookup says %s but exact-route for this "
"flow says %s. The flow uses %s"
% (egress, canon_intf(e2.group(1)), canon_intf(e2.group(1))))
h.egress = canon_intf(e2.group(1))
self.fill_identity(h)
self.fill_ingress(h)
self.fill_health(h)
self.fill_members(h)
if h.osn in FIREWALL_OS:
h.kind = "firewall"
self.inspect_policy(h, src, dst, dport, proto)
def port_behind_svi(self, h, svi, peer_ip):
"""The physical port behind an SVI, for a peer reachable through it.
Traffic does not arrive on a Vlan interface. It arrives on a port
whose access or trunk configuration puts it in that VLAN, and the SVI
is only where the packet is routed afterwards. The SVI has no cable,
no optic, no duplex and no CRC counter, so reporting it as the
interface hides the only thing that can actually be broken.
ARP for the peer gives a MAC and the MAC table gives the port -- the
same two lookups the L2 walk downstream already uses.
"""
if not peer_ip:
return None, "no neighbour address on that side"
vrf = h.vrf if h.vrf_source in ("inventory", "discovered") else None
out, _ = self.ask(h.ip, h.osn, "arp_vrf" if vrf else "arp",
ip=peer_ip, vrf=vrf or "")
mac = parse_arp(out)
if not mac:
return None, "no ARP entry for %s" % peer_ip
out, _ = self.ask(h.ip, h.osn, "mac", mac=mac_for(h.osn, mac))
port = parse_mac_port(out)
if not port:
return None, "%s not in the MAC table" % mac
return canon_intf(port), None
def resolve_side(self, h, intf, peer_ip, side):
"""(interface_to_report, svi_label). Turns an SVI into the physical
port it is reached through, and keeps the SVI as context because the
VLAN is still what the reader needs to know."""
if not intf or intf_kind(intf) != "logical" or not intf.lower().startswith(
("vlan", "irb", "bvi")):
return intf, None
port, why = self.port_behind_svi(h, intf, peer_ip)
if port:
return port, intf
h.notes.append("%s %s is an SVI; physical port unknown (%s)" % (side, intf, why))
return intf, None
def fill_ingress(self, h):
"""The interface this flow ARRIVES on.
A traceroute hop answers from the interface the probe came in on, so
the hop's own address names its ingress. That matters more than it
sounds: CRC, runts and input errors are counted on the RECEIVING
interface, so a path report that shows only egress is showing the end
of each cable where the receive errors are not.
"""
if not h.ip or not h.osn or h.ingress:
return
# A FIB lookup back toward the SOURCE is the real ingress. The hop's
# own reply address is only a hint: a router may source a TTL-exceeded
# from a loopback, from the egress, or from whatever
# `icmp source-address` says, and inside an MPLS LSP a P router does
# no IP lookup at all.
back = None
if self.src_ip:
vrf = h.vrf if h.vrf_source in ("inventory", "discovered") else None
rout, _ = self.ask(h.ip, h.osn, "route_vrf" if vrf else "route",
dst=self.src_ip, vrf=vrf or "default", src=h.ip)
back, _nh = parse_route_egress(rout)
if back:
h.ingress = canon_intf(back)
out, _ = self.ask(h.ip, h.osn, "addr2intf", ip=h.ip)
if not out:
return
replied_on = None
for line in (out or "").split(chr(10)):
if h.ip not in line:
continue
hits = INTF_RE.findall(line)
if hits:
replied_on = canon_intf(hits[0])
break
if h.ingress and replied_on and replied_on != h.ingress:
# The return path leaves by a different interface than the one the
# hop answered from. On a stateful firewall anywhere in between,
# this is the flow dying with every routing table saying it should
# work -- and it is the commonest cause of exactly that complaint.
h.asymmetric = replied_on
self.finding("MEDIUM", h.name,
"asymmetric routing at %s" % h.name,
"the return path toward %s leaves by %s, but the hop answered "
"from %s. Traffic is not following the same interfaces in both "
"directions; a stateful device in this path will drop it"
% (self.src_ip, h.ingress, replied_on))
h.ingress = h.ingress or replied_on
if not h.ingress:
h.notes.append("could not map %s to an interface, so the ingress side "
"was not inspected" % h.ip)
return
# The hop before this one is on the other end of the ingress, so it
# is the address to resolve the SVI through.
prev = next((x.ip for x in reversed(self.hops)
if x.ip and x.ip != h.ip and x.order < h.order), None)
h.ingress, h.svi_in = self.resolve_side(h, h.ingress, prev, "ingress")
info, _ = self.ask(h.ip, h.osn, "intf", intf=h.ingress)
h.health_in = parse_interface(info)
h.util_in = utilisation(h.health_in)
self.check_health_side(h, h.ingress, h.health_in, h.util_in)
def fill_identity(self, h):
"""Model and role, so the path reads as a topology instead of a list
of addresses."""
if h.model or not h.osn or not h.ip:
return
out, _ = self.ask(h.ip, h.osn, "version")
h.model, h.role = parse_model(out, h.osn)
# startswith, not ==: the gateway's name carries an "(active HSRP)"
# suffix, so an exact match never fired on the one device where the
# name matters most.
if h.ip and h.name.startswith(h.ip): # no inventory entry for it
real = parse_hostname(out)
if real:
h.name = real + h.name[len(h.ip):]
if h.role is None and h.osn in FIREWALL_OS:
h.role = "firewall"
def fill_health(self, h):
"""Interface detail, plus the syslog lookup for platforms whose
`show interfaces` carries no last-flap field -- which is IOS and
IOS-XE, the most common ones on any path."""
if not h.egress:
return
h.egress, h.svi_out = self.resolve_side(h, h.egress, h.next_hop, "egress")
info, _ = self.ask(h.ip, h.osn, "intf", intf=h.egress)
h.health = parse_interface(info)
h.util = utilisation(h.health)
if "last_flap" not in h.health:
logs, _ = self.ask(h.ip, h.osn, "logs", intf=h.egress)
h.flaps = parse_flaps(logs, h.egress)
if h.flaps:
h.health["last_flap"] = h.flaps[0][0] or "see log"
# No note when there were no events. It was true on nearly every
# port and said nothing happened in more words than nothing takes.
self.check_health(h)
def fill_members(self, h):
"""The member links of a port-channel egress.
A flow takes exactly ONE member. A Po at 40% can have one member at
100%, and the aggregate counter says the link is fine while that flow
is being tail-dropped on the member it hashed to. The aggregate is
the one number that cannot answer the question being asked.
"""
if not h.egress or not h.osn or not h.egress.lower().startswith(
("port-channel", "ae", "bundle-ether")):
return
out, _ = self.ask(h.ip, h.osn, "po_members", intf=h.egress)
if out is None:
self.unchecked.append(
"%s: %s is a port-channel and its members were not read on this "
"platform. The aggregate utilisation can hide a member at 100%%."
% (h.name, h.egress))
return
for port, state in parse_po_members(out):
info, _ = self.ask(h.ip, h.osn, "intf", intf=port)
mh = parse_interface(info)
mu = utilisation(mh)
h.members.append((port, state, mh, mu))
if state and state.lower() in ("s", "d", "sd", "sm"):
self.finding("HIGH", h.name,
"%s member %s is %s" % (h.egress, port,
"suspended" if state.lower().startswith("s")
else "down"),
"a flow hashed to this member is dropped, and the "
"port-channel's own counters look healthy")
elif mu is not None and mu >= 90:
self.finding("HIGH", h.name,
"%s member %s is at %.1f%%" % (h.egress, port, mu),
"the port-channel aggregate hides this: a flow hashes to "
"ONE member, and this one is saturated")
def check_health(self, h):
self.check_health_side(h, h.egress, h.health, h.util)
def check_health_side(self, h, intf, health, util):
st = health.get("status")
if st and st not in ("up",):
self.finding("HIGH", h.name, "interface %s is %s" % (intf, st), "")
if util is not None and util >= self.a.util_warn:
self.finding("HIGH" if util >= 90 else "MEDIUM", h.name,
"%s is at %.1f%%" % (intf, util),
"saturation here explains loss and latency downstream")
if health.get("duplex") == "half":
self.finding("HIGH", h.name, "%s is HALF duplex" % intf,
"a negotiation failure. Expect late collisions and terrible "
"throughput that looks like packet loss")
# Interpret rather than list. The age of the counters decides whether a
# total is alarming: 48,000 drops since Tuesday and since 2019 are
# different findings, and most reports never say which.
age = health.get("counters_age")
age_note = ("counters last cleared %s" % age) if age else \
("counter age unknown -- a total is not a rate until you know "
"when it was last cleared")
# interpret(health), NOT interpret(h.health): the parameterised
# version was left pointing at the egress, so the ingress counters
# were never interpreted and the egress ones were interpreted twice.
for kind, meaning, evidence in interpret(health):
counts = ", ".join("%s %s" % (k, v) for k, v in evidence)
sev = "HIGH" if kind in ("physical", "duplex") else "MEDIUM"
self.finding(sev, h.name, "%s: %s (%s)" % (intf, kind, counts),
"%s. %s" % (meaning, age_note))
fl = health.get("last_flap")
if fl and not re.search(r"\b(never|\d+w\d|\d+y)\b", fl, re.I):
detail = "a recent flap on the path explains intermittency"
if h.flaps and intf == h.egress: # the log was read for that port
detail += ". Log: " + "; ".join("%s %s" % e for e in h.flaps[:3])
self.finding("MEDIUM", h.name, "%s last flapped %s" % (intf, fl), detail)
def inspect_policy(self, h, src, dst, dport, proto):
"""Session table first, because it is a show on every firewall and it
shows what is actually happening. Simulation second, because when the
traffic is blocked there IS no session -- and that is exactly the case
where you need to know which rule denies it."""
pnum = PROTO_NUM.get(proto, "6")
out, err = self.ask(h.ip, h.osn, "session", src=src, dst=dst,
dport=dport, proto=pnum)
if out and out.strip() and not EMPTY_SESSION.search(out):
act, rule = parse_policy(out)
if rule or act:
h.policy_action, h.policy_rule, h.policy_source = act, rule, "session"
return
h.notes.append("sessions exist but no rule name could be read from them")
cls = CMD[h.osn].get("policy_class")
if CMD[h.osn].get("needs_zones"):
zin, zout = self.zones(h, src, dst)
h.zone_in, h.zone_out = zin, zout
if not (zin and zout):
h.notes.append(
"no live session, and the policy simulation needs the ingress and "
"egress zones which could not be resolved. Sending invented zones "
"would return a rule that is not the one traffic matches, so it "
"was not sent")
self.unchecked.append("%s: policy not determined (zones unknown)" % h.name)
return
else:
zin = zout = None
out, err = self.ask(h.ip, h.osn, "policy", src=src, dst=dst, dport=dport,
proto=pnum, proto_name=proto, intf=h.egress or "",
from_zone=zin or "", to_zone=zout or "")
if out is None:
h.notes.append(
"no live session and the policy lookup returned nothing. On this "
"platform that command is '%s' class -- if the account only permits "
"show, it was refused. See AUTHORISATION." % (cls or "?"))
self.unchecked.append(
"%s: policy NOT determined. Absence of a session is not permission."
% h.name)
return
act, rule = parse_policy(out)
h.policy_action, h.policy_rule, h.policy_source = act, rule, "simulation"
if act is None:
h.notes.append("policy lookup ran but its output was not recognised")
def zones(self, h, src, dst):
"""(ingress_zone, egress_zone) for the platforms whose policy lookup
demands them.
Resolved, never guessed. A FIB lookup toward the DESTINATION gives the
egress interface and a FIB lookup toward the SOURCE gives the ingress
one; the zone of each comes from `show interface`, because the FIB
lookup itself does not print a zone.
The first version matched a bare zone pattern anywhere, which also matched
inside "ingress zone:" and produced from=TRUST to=TRUST -- a lookup
that returns a real rule for a path that does not exist.
"""
def zone_of(intf):
if not intf:
return None
out, _ = self.ask(h.ip, h.osn, "intf", intf=intf)
m = re.search(r"^\s*zone\s*[:=]\s*(\S+)", out or "", re.I | re.M)
return m.group(1).rstrip(",;") if m else None
out_d, _ = self.ask(h.ip, h.osn, "route", dst=dst, vrf="default", src=src)
eg, _nh = parse_route_egress(out_d)
out_s, _ = self.ask(h.ip, h.osn, "route", dst=src, vrf="default", src=dst)
ing, _nh2 = parse_route_egress(out_s)
zi, zo = zone_of(ing), zone_of(eg)
if zi is None or zo is None:
# Last resort: some builds do print the zones in the lookup itself.
# LINE-ANCHORED. A loose /zone[:=]/ also matches inside
# "ingress zone:", which is how the first two versions of this
# produced from=TRUST to=TRUST and asked the firewall about a path
# that does not exist.
for text, pat, slot in (
(out_d, r"^[ ]*zone[ ]*[:=][ ]*(\S+)", "zo"),
(out_d, r"^[ ]*ingress[ -]zone[ ]*[:=][ ]*(\S+)", "zi")):
m = re.search(pat, text or "", re.I | re.M)
if m:
if slot == "zo" and zo is None:
zo = m.group(1)
if slot == "zi" and zi is None:
zi = m.group(1)
return zi, zo
# What each platform calls the table a route was resolved in. Saying
# "global table" on a Palo Alto is simply wrong -- PAN-OS has virtual
# routers, Junos has routing instances, FortiOS has VDOMs -- and being wrong
# in the vocabulary is how a reader stops trusting the rest of the line.
TABLE_WORD = {
"ios": ("global table", "vrf %s"),
"iosxe": ("global table", "vrf %s"),
"iosxr": ("default VRF", "vrf %s"),
"nxos": ("vrf default", "vrf %s"),
"eos": ("default VRF", "vrf %s"),
"junos": ("inet.0, master instance", "routing-instance %s"),
"panos": ("virtual-router (default)", "virtual-router %s"),
"fortios": ("VDOM root", "VDOM %s"),
"asa": ("the routing table", "vrf %s"),
}
def table_label(osn, vrf, source):
"""How to name the table this hop's route came from, in its own dialect."""
if source == "ambiguous":
return "TABLE AMBIGUOUS"
glob, named = TABLE_WORD.get(osn, ("global table", "vrf %s"))
if not vrf:
return glob
how = {"inventory": " (from inventory)", "discovered": " (discovered)"}.get(source, "")
return (named % vrf) + how
# ----------------------------------------------------------------- report
SEV = {"HIGH": 0, "MEDIUM": 1, "INFO": 2}
COUNTER_KEYS = ("in_err", "out_err", "crc", "frame", "overrun", "ignored", "runts",
"giants", "collisions", "late_coll", "resets", "drops", "carrier",
"inq_drops", "inq_flushes")
def fmt_rate(kbit, logical=False):
"""A Kbit/s figure as an engineer says it."""
try:
mb = int(kbit) / 1000
except (TypeError, ValueError):
return str(kbit)
r = "%gG" % (mb / 1000) if mb >= 1000 else "%gM" % mb
return ("bw " + r) if logical else r
def fmt_health(h):
return fmt_health_dict(h.health, h.util, intf_kind(h.egress))
def fmt_health_dict(i, util, kind="physical"):
if not i:
return "-"
bits = []
if i.get("status"):
bits.append(i["status"])
if i.get("speed"):
# On a logical interface this is the CONFIGURED bandwidth, not a
# negotiated link rate, and the two must not be printed the same way.
bits.append(fmt_rate(i["speed"], kind == "logical"))
if i.get("duplex") and kind != "logical":
bits.append(i["duplex"][:4])
if i.get("mtu") and i["mtu"] != 1500:
bits.append("mtu %d" % i["mtu"])
bits.append("util %.1f%%" % util if util is not None else "util -")
for k, lbl in (("in_err", "inErr"), ("out_err", "outErr"), ("crc", "CRC"),
("runts", "runts"), ("giants", "giants"), ("frame", "frame"),
("overrun", "ovrun"), ("ignored", "ignrd"), ("late_coll", "lateColl"),
("resets", "resets"), ("drops", "drops"), ("inq_drops", "inQdrop"),
("carrier", "lostCarr")):
if i.get(k):
bits.append("%s %d" % (lbl, i[k]))
if i.get("last_flap"):
bits.append("flap %s" % i["last_flap"])
if i.get("up_since"):
bits.append("up %s" % i["up_since"])
if i.get("counters_age"):
bits.append("(cleared %s)" % i["counters_age"])
return " ".join(bits) if bits else "-"
def report(w, args):
line = "=" * 78
print(line)
print("network-path-analyzer %s -> %s %s/%s"
% (args.src, args.dst, args.proto.upper(), args.dport))
print(line)
print("Gateway given : %s" % args.gateway)
print("")
print("PATH")
print("-" * 78)
for h in w.hops:
tag = {"l2": "L2", "l3": "L3", "firewall": "FW", "endpoint": " "}.get(h.kind, " ")
what = " ".join(x for x in (h.role, h.model) if x)
conf = " [%s/%s]" % (h.osn, CONFIDENCE.get(h.osn, "?")) if h.osn else ""
print(" %-2s %-24s %-14s %-22s%s"
% (tag, h.name[:24], h.ip or "-", what[:22] or "-", conf))
# Both ends of every transit device. Receive errors land on the
# ingress and transmit problems on the egress, so showing one side
# shows the end of the cable where half the evidence is not.
ingress = h.downstream or h.ingress
in_health = h.health_down if h.downstream else h.health_in
in_util = h.util_down if h.downstream else h.util_in
def side(label, intf, health, util, zone, svi):
# The zone belongs beside the interface, not on a line of its own:
# on a firewall it is the first thing read, and an interface
# without its zone is half an answer.
# The SVI stays beside the port: the physical interface is what
# can break, the VLAN is what the reader is thinking in.
where = intf + (" (%s)" % svi if svi else "") + (" zone %s" % zone if zone else "")
print(" %-9s %-30s %s"
% (label, where, fmt_health_dict(health, util, intf_kind(intf))))
if ingress and h.kind != "endpoint":
side("ingress", ingress, in_health, in_util, h.zone_in, h.svi_in)
if h.egress:
side("egress", h.egress, h.health, h.util, h.zone_out, h.svi_out)
# Only where a route lookup actually happened. An L2 switch
# bridged the frame and consulted no table, so claiming one sends
# the reader hunting for a route that was never involved.
#
# Where it did happen, always say which table, in the platform's
# own vocabulary: a lookup that silently used the global table
# while the traffic lives in a VRF looked exactly like a correct
# answer before this line existed.
if h.kind in ("l3", "firewall"):
print(" %-9s %s%s" % ("routed in",
table_label(h.osn, h.vrf, h.vrf_source),
" ECMP: %d equal-cost paths"
% h.ecmp if h.ecmp > 1 and not any(
x.startswith("ECMP") for x in h.notes)
else ""))
# A flow takes one member, so the members are the fact and the
# aggregate is the summary that hides it.
for port, state, mh, mu in h.members:
print(" %-9s %-30s %s"
% (" member", port + (" (%s)" % state if state else ""),
fmt_health_dict(mh, mu, "physical")))
if h.kind == "firewall":
if h.policy_action:
src_lbl = {"session": "live session", "simulation": "policy simulation"}[h.policy_source]
print(" %-9s %s rule %s (from the %s)"
% ("policy", h.policy_action, h.policy_rule or "(unnamed)", src_lbl))
else:
print(" %-9s NOT DETERMINED" % "policy")
for note in h.notes:
print(" %-9s %s" % ("note", note))
print("")
if w.findings:
order = sorted(w.findings, key=lambda f: SEV.get(f[0], 9))
cur = None
for sev, where, what, detail in order:
if sev != cur:
cur = sev
print("%s (%d)" % (cur, len([x for x in order if x[0] == cur])))
print("-" * 78)
print(" [%s] %s" % (where, what))
if detail:
print(" %s" % detail)
print("")
else:
print("No findings on the path that was walked.\n")
print("WHAT THIS WALK DOES NOT PROVE")
print("-" * 78)
base = [
"One direction only. The return path from %s can differ, and asymmetric" % args.dst,
" routing is a common cause of what looks like one-way loss.",
"The traceroute probe is not your flow: under ECMP it can hash elsewhere.",
" Where the platform offers it, the egress was re-resolved with the real",
" 5-tuple and the note says so.",
"A firewall that permits the traceroute probe can still deny your port:",
" the probe matches a different rule.",
"Counters are cumulative since the last clear. A large total is not a rate.",
"Every parser in this build is marked 'modelled' until it has been run",
" against that platform's real output. A hop whose output was not",
" recognised says so above rather than reporting a clean result.",
]
for t in base + w.unchecked:
print(" - %s" % t if not t.startswith(" ") else " %s" % t)
print("")
print("AUTHORISATION USED")
print("-" * 78)
for osn in sorted({h.osn for h in w.hops if h.osn}):
print(" %-8s %s" % (osn, AUTH_NOTES.get(osn, "-")))
def write_csv(path, w, args):
with open(path, "w", encoding="utf-8-sig", newline="") as fh:
c = csv.writer(fh)
cols = ["order", "kind", "device", "ip", "platform", "model", "role",
"ingress", "ingress_util", "egress", "vrf", "vrf_source", "status",
"duplex", "speed_kbit", "util_pct"] + list(COUNTER_KEYS) + \
["last_flap", "up_since", "counters_age", "last_in", "last_out",
"policy_action", "policy_rule", "policy_source", "flap_log", "notes"]
c.writerow(cols)
for h in w.hops:
i = h.health
row = [h.order, h.kind, h.name, h.ip or "", h.osn or "",
h.model or "", h.role or "",
h.downstream or h.ingress or "",
"" if (h.util_down if h.downstream else h.util_in) is None
else (h.util_down if h.downstream else h.util_in),
h.egress or "", h.vrf or "", h.vrf_source or "",
i.get("status", ""), i.get("duplex", ""), i.get("speed", ""),
"" if h.util is None else h.util]
row += [i.get(k, "") for k in COUNTER_KEYS]
row += [i.get("last_flap", ""), i.get("up_since", ""),
i.get("counters_age", ""), i.get("last_in", ""), i.get("last_out", ""),
h.policy_action or "", h.policy_rule or "", h.policy_source or "",
" ; ".join("%s %s" % e for e in h.flaps), " | ".join(h.notes)]
c.writerow(row)
# ----------------------------------------------------------------- entry
def parse_args(argv):
p = argparse.ArgumentParser(
description="Walk the real path a flow takes, hop by hop, inspecting every device "
"on it -- including the L2 switches a traceroute cannot see -- and say which hop hurts. Read-only.")
p.add_argument("--src", required=True, help="the source address -- the host that cannot reach, or is losing packets to, the destination")
p.add_argument("--dst", required=True, help="the destination address")
p.add_argument("--dport", required=True, help="destination TCP/UDP port, e.g. 1433")
p.add_argument("--proto", default="tcp", choices=["tcp", "udp"], help="default tcp")
p.add_argument("--gateway", help="the SOURCE host's default gateway. The walk starts here, so the path measured is the one the source's traffic takes")
p.add_argument("--gateway-os", help="its platform, if it is not in the inventory")
p.add_argument("--inventory", help="YAML inventory. Needs NET_USER and NET_PASS")
p.add_argument("--offline-dir", help="Replay a --save-dir. No devices, no credentials")
p.add_argument("--save-dir", help="Save every command's output for later offline runs")
p.add_argument("--util-warn", type=float, default=70.0,
help="Flag an interface at or above this percent. Default 70")
p.add_argument("--csv", help="Write the whole path to a CSV")
args = p.parse_args(argv)
if not args.offline_dir and not args.gateway:
p.error("--gateway is required (or use --offline-dir)")
if args.offline_dir and args.inventory is None and args.gateway is None:
p.error("--offline-dir still needs --gateway to know where to start")
return args
def main(argv=None):
args = parse_args(argv)
inv = load_inventory(args.inventory) if args.inventory else {}
if args.gateway and args.gateway_os and args.gateway not in inv:
inv[args.gateway] = {"host": args.gateway, "os": args.gateway_os,
"port": 22, "device_name": args.gateway,
"mgmt": args.gateway, "addresses": [args.gateway]}
t = Transport(args)
w = Walker(t, inv, args)
try:
gw, gw_os = w.resolve_active_gateway(args.gateway)
w.src_ip = args.src
w.walk_to_source(gw, gw_os, args.src)
w.walk_to_dst(gw, gw_os, args.src, args.dst, args.dport, args.proto)
w.walk_to_dest_host(args.dst)
w.check_links()
finally:
t.close()
w.hops.sort(key=lambda h: h.order)
report(w, args)
if args.csv:
write_csv(args.csv, w, args)
print("\nCSV written: %s" % args.csv)
return 1 if any(f[0] in ("HIGH", "MEDIUM") for f in w.findings) else 0
if __name__ == "__main__":
try:
sys.exit(main())
except KeyboardInterrupt:
sys.stderr.write("\ninterrupted\n")
sys.exit(2)
except SystemExit:
raise
except OSError as exc:
sys.stderr.write("failed: %s\n" % exc)
sys.exit(2)
--srctextrequiredthe source host
--dsttextrequiredthe destination host
--dporttextrequiredThe destination port that matters, e.g. 1433. The firewall lookup uses it, so a wrong port returns the wrong rule
--gatewaytextrequiredthe source host's default gateway. The walk starts here, so the path measured is the one the source's traffic takes. Name either member of an HSRP/VRRP pair: the script finds the active and moves there
--prototexttcp or udp. Default tcp
--gateway-ostextThe gateway's platform, if it is not in the inventory
--inventorytextYAML inventory. Needs NET_USER and NET_PASS in the environment
--offline-dirtextReplay a --save-dir. No devices and no credentials
--save-dirtextSave every command's output. Worth doing every run: it makes the walk repeatable and lets you show the text a finding came from
--util-warnnumberFlag an interface at or above this percent. Default 70
--csvtextWrite the whole path to a CSV, one row per hop. UTF-8 with BOM for Excel
What This Solves
The application team says the database connection times out. The network team says the network is fine. Both are right about the half they can see, and neither of them has a drawing of the path between them that is younger than two years. Somebody now has to log into six devices, run the same four commands on each, and read a traceroute that does not show half of the equipment in the way. That is the afternoon this replaces. Point it at a source, a destination and the port that matters: python3 network_path_analyzer.py --src 10.20.30.40 --dst 10.50.60.70 --dport 1433 --gateway 10.20.30.1 --inventory devices.yaml It walks the path, enters every device on it, and reports what each one is doing to that flow. It answers two tickets, AND they are the same walk "They cannot talk at all." The answer is usually a policy nobody remembers writing, and you get its rule name. "They drop packets, intermittently, and nobody can reproduce it." The answer is usually a saturated uplink, a port throwing CRCs, or a link that flapped at 3am. You get all three, per hop, with the age of the counters so you can tell a problem from a souvenir. Why it is NOT a traceroute wrapper Three facts make the obvious approach wrong, and they are the whole reason this tool has the shape it has. A traceroute from your laptop measures the path from YOUR LAPTOP. If the source host sits in another VLAN, another site or another VRF, that is simply a different path from the one that is failing. So you name the source's default gateway and the walk begins there, inside the source's own subnet. LAYER 2 SWITCHES NEVER APPEAR IN A TRACEROUTE. They do not decrement TTL, so no amount of probing will ever reveal them. They are found the way an engineer finds them by hand: the ARP entry gives the MAC, the MAC table gives the port, the CDP or LLDP neighbour on that port gives the next switch, and a port with no neighbour is where the host is plugged in. That chain is also the only thing that can tell you the access port is throwing CRC errors, which no traceroute will ever show you. Firewalls do appear in internal traceroutes often enough these days, but an appearance tells you nothing about your flow. The probe is UDP or ICMP and matches a DIFFERENT RULE than TCP/1433. A firewall that cheerfully forwards your traceroute can be dropping the port you care about. So the traceroute finds the firewall, and a policy lookup answers the question with the real five-tuple.
Is It Safe to Run
It never configures anything. There is no code path that enters configuration mode, and nothing it sends changes state on a device. It is not, however, all show commands, and saying otherwise would cost you an afternoon on your first run. The full list: show ip arp / show arp read show mac address-table read show cdp neighbors detail read show lldp neighbors detail read show ip route / show route read show ip cef exact-route read show interfaces read show version / show system info read show logging read show standby brief / show vrrp brief read traceroute NOT a show command show security flow session (Junos) read show session all filter (PAN-OS) read show conn (ASA) read show security match-policies (Junos) read test routing fib-lookup (PAN-OS) test class test security-policy-match (PAN-OS) test class packet-tracer (ASA/FTD) privileged exec, usually 15 diagnose firewall iprope lookup (FortiOS) diagnose class Read Authorisation before you request an account. Juniper is the one platform where even the policy lookup is a show, so an SRX can be walked end to end with nothing but read access.
Before You Start
Python 3.8 or newer. pip install "netmiko>=4.0" pyyaml Neither is needed for --offline-dir, which replays saved output using only the standard library. You need three things. THE FLOW. Source address, destination address, and the destination port. The port is not optional and it is not cosmetic: the policy lookup uses it, so a wrong port returns a different rule and a confident wrong answer. THE SOURCE'S DEFAULT GATEWAY. You know it; the tool cannot work it out, and asking you beats guessing. If that gateway is an HSRP or VRRP pair it does not matter which member you name -- see below. AN ACCOUNT. Yours or a dedicated one. The account: yours, or a dedicated one Either works and the tool cannot tell the difference. Your own credentials are the quickest way to run it once. For anything you expect to repeat -- a runbook, a scheduled check, several engineers -- create a GENERIC SERVICE ACCOUNT in TACACS+ or RADIUS, dedicated to this tool. A dedicated account is better for three reasons that have nothing to do with this script. The device logs then show which commands came from automation rather than from a person. The account can be scoped to exactly the commands listed above instead of inheriting whatever privileges an engineer happens to carry. And it can be disabled without taking anybody's login with it. Whichever you use, the script reads it from the environment and never from an argument, because an argument is visible to every user on the box in "ps auxww" and is written to your shell history: Linux / macOS export NET_USER=svc-netpath read -s NET_PASS; export NET_PASS PowerShell $env:NET_USER = "svc-netpath" $env:NET_PASS = "..." NET_ENABLE is optional and falls back to NET_PASS.
Your First Run
python3 network_path_analyzer.py --src 10.20.30.40 --dst 10.50.60.70 --dport 1433 --gateway 10.20.30.1 --inventory devices.yaml --save-dir ./walk-evidence --csv path.csv That is the whole thing. USE --save-dir EVERY TIME. It writes every command's output to disk, which buys you two things that matter during an incident. You can re-run the entire analysis later with no devices at all -- python3 network_path_analyzer.py --src 10.20.30.40 --dst 10.50.60.70 --dport 1433 --gateway 10.20.30.1 --offline-dir ./walk-evidence -- and when somebody disputes a finding, you can show them the raw text it came from instead of re-running commands on a device that has since changed. If you name the standby of an HSRP or VRRP pair, the tool notices, finds the active member, MOVES THERE, and says so in the report. The path is measured from where the traffic actually goes, not from wherever you happened to log in. If it cannot tell which member is active it says that too, rather than quietly walking from the wrong one.
What the Output Means
============================================================================== network-path-analyzer 10.10.10.50 -> 10.20.20.80 TCP/1433 ============================================================================== Gateway given : 10.10.10.2 PATH ------------------------------------------------------------------------------ source 10.10.10.50 - note attached to sw-access-01 GigabitEthernet1/0/12 (access port, no CDP/LLDP neighbour) L2 sw-access-01 10.10.99.5 switch WS-C3850-48P [ios/modelled] egress GigabitEthernet1/0/12 up 1G full util 0.3% inErr 871 CRC 844 runts 12 frame 27 resets 4 inQdrop 19 flap Oct 19 06:41:07.003 (cleared 3d04h) L3 gw-active (active HSRP) 10.10.10.3 switch C9300-48U [ios/modelled] ingress GigabitEthernet1/0/1 up 1G util 12.0% egress Port-channel1 up bw 20G mtu 9216 util 21.0% (cleared 11w4d) routed in global table member TenGigabitEthernet1/1/1 (P) up 10G full mtu 9216 util 21.0% (cleared 11w4d) member TenGigabitEthernet1/1/2 (P) up 10G full mtu 9216 util 97.0% (cleared 11w4d) note ECMP: 2 equal-cost paths; exact-route puts THIS flow on Port-channel1. The others were not examined L3 core-01 10.0.0.1 router ASR-920-24SZ-M [ios/modelled] ingress TenGigabitEthernet0/0/0 up 10G full util 42.0% inErr 2914 CRC 2890 frame 24 (cleared 3d04h) egress TenGigabitEthernet0/1/0 up 10G util 92.0% drops 48219 routed in vrf PROD (discovered) FW fw-dc-01 10.0.0.9 firewall PA-3220 [panos/modelled] ingress ethernet1/1 zone TRUST up util - egress ethernet1/3 zone DMZ up util - routed in virtual-router (default) policy DENY rule Deny-SQL-From-Users (from the policy simulation) L3 dist-01 10.20.20.1 switch C9300-24T [ios/modelled] ingress Vlan220 up bw 1G util 1.1% egress GigabitEthernet1/0/24 (Vlan220) up 1G full util 1.1% (cleared 6w2d) routed in global table note ingress Vlan220 is an SVI; physical port unknown (no ARP entry for 10.0.0.9) L2 sw-dc-access-01 10.10.99.8 switch C9300-48T [ios/modelled] egress GigabitEthernet1/0/9 up 1G full util 2.3% (cleared 9w1d) destination 10.20.20.80 - note attached to sw-dc-access-01 GigabitEthernet1/0/9 (access port, no CDP/LLDP neighbour) HIGH (6) ------------------------------------------------------------------------------ [sw-access-01] GigabitEthernet1/0/12: physical (crc 844, runts 12, frame 27) CRC, runts or framing errors point at the cable, the optic or a speed/duplex mismatch. Load has nothing to do with it. counters last cleared 3d04h [gw-active (active HSRP)] Port-channel1 member TenGigabitEthernet1/1/2 is at 97.0% the port-channel aggregate hides this: a flow hashes to ONE member, and this one is saturated [core-01] TenGigabitEthernet0/0/0: physical (crc 2890, frame 24) CRC, runts or framing errors point at the cable, the optic or a speed/duplex mismatch. Load has nothing to do with it. counters last cleared 3d04h [core-01] TenGigabitEthernet0/1/0 is at 92.0% saturation here explains loss and latency downstream [fw-dc-01] traffic is denied here rule Deny-SQL-From-Users denies 10.10.10.50 -> 10.20.20.80:1433. This is where the flow stops. [gw-active (active HSRP)] MTU mismatch on gw-active (active HSRP) Port-channel1 <-> core-01 TenGigabitEthernet0/0/0 Port-channel1 is 9216 and TenGigabitEthernet0/0/0 is 1500. Large packets are dropped in one direction; small ones, including ping, are not MEDIUM (4) ------------------------------------------------------------------------------ [sw-access-01] GigabitEthernet1/0/12: punt (inq_drops 19) input-queue drops are the control plane falling behind, which is a different fault from a full link. counters last cleared 3d04h [sw-access-01] GigabitEthernet1/0/12: flapping (resets 4) interface resets and lost carrier count the times the link was bounced. counters last cleared 3d04h [sw-access-01] GigabitEthernet1/0/12 last flapped Oct 19 06:41:07.003 a recent flap on the path explains intermittency. Log: Oct 19 06:41:07.003 Interface GigabitEthernet1/0/12, changed state to up; Oct 19 06:41:02.774 Interface GigabitEthernet1/0/12, changed state to down; Oct 19 02:14:55.118 Interface GigabitEthernet1/0/12, changed state to up [core-01] TenGigabitEthernet0/1/0: congestion (drops 48219) output drops are the queue overflowing. Not an error: the interface is simply full at times. counter age unknown -- a total is not a rate until you know when it was last cleared INFO (1) ------------------------------------------------------------------------------ [gw-standby] entered the standby, moved to the active gw-standby is the HSRP standby. Everything below was measured from 10.10.10.3, which is where the traffic actually goes. WHAT THIS WALK DOES NOT PROVE ------------------------------------------------------------------------------ - One direction only. The return path from 10.20.20.80 can differ, and asymmetric routing is a common cause of what looks like one-way loss. - The traceroute probe is not your flow: under ECMP it can hash elsewhere. Where the platform offers it, the egress was re-resolved with the real 5-tuple and the note says so. - A firewall that permits the traceroute probe can still deny your port: the probe matches a different rule. - Counters are cumulative since the last clear. A large total is not a rate. - Every parser in this build is marked 'modelled' until it has been run against that platform's real output. A hop whose output was not recognised says so above rather than reporting a clean result. - The traceroute probe is not your flow. It is UDP or ICMP while the flow is TCP/1433, so under ECMP it can hash to a different path than the application takes. AUTHORISATION USED ------------------------------------------------------------------------------ ios show plus 'traceroute', which is not a show command panos show plus the test class: 'test routing fib-lookup' and 'test security-policy-match' The path block Devices are listed in travel order, source first. The tag says what the device is doing in this path: (blank) the endpoint L2 a switch, found through the MAC table. A traceroute cannot see these at all L3 a router or L3 switch FW a firewall, where the policy lookup runs Next to the address is the model and the role, because a path reads very differently once you can see it goes access switch, core router, firewall. An interface at 92% on an access switch and on a core uplink are different conversations. A ROUTING DEVICE GETS ONE ENTRY, NOT TWO. The gateway is reached twice during the walk -- once going down toward the source through its MAC table, once as the origin of the traceroute -- and both of its interfaces matter. They are printed as "toward source" and "toward dest" on the same device. Finding its downstream port in a MAC table does not make it a switch. Both ends of every device Each transit device prints two interfaces, because it has two in the path: ingress Te0/0/0 up 10G full util 42.0% inErr 2914 CRC 2890 egress Te0/1/0 up 10G util 92.0% drops 48219 This is not symmetry for its own sake. RECEIVE ERRORS ARE COUNTED ON THE INTERFACE THAT RECEIVES THE FRAME -- CRC, runts, framing and input errors all land on the ingress. A report that shows only the egress is showing the end of each cable where those errors do not appear, and a duplex mismatch, which is late collisions on one end and CRC on the other, cannot be diagnosed from one side at all. The ingress costs nothing to find: a traceroute hop answers from the interface the probe arrived on, so the hop's own address names it. Three things about those lines are deliberate: ONE NAME PER INTERFACE. Every ABBREVIATION is expanded, because show ip interface brief says Gi1/0/12 and show interfaces says GigabitEthernet1/0/12 -- the same port under two names reads as two ports. A name already in full form is left exactly as the device spells it: ethernet1/3 and ge-0/0/0 ARE the full names on PAN-OS and Junos, and "correcting" their capitalisation would send the device a command about an interface it does not have. AN SVI IS NOT AN INGRESS. Traffic does not arrive on a Vlan interface. It arrives on a PHYSICAL PORT whose access or trunk configuration puts it in that VLAN, and the SVI is only where the packet is routed once it is already inside the switch. So where a side resolves to an SVI, the tool finds the port behind it -- ARP for the neighbour on that side gives a MAC, the MAC table gives the port -- and prints the port with the VLAN as context: egress GigabitEthernet1/0/24 (Vlan220) up 1G full util 1.1% This is not tidiness. An SVI has no cable, no optic, no duplex and no CRC counter, so it can never show the fault the reader is hunting for. Where the port cannot be resolved the tool says so and names the SVI, because an admission is worth more than an interface that cannot be at fault. A LOGICAL INTERFACE IS MARKED AS ONE. An SVI, a tunnel, a loopback or a port-channel has no link, so it has no negotiated speed and no duplex. What the platform prints there is the CONFIGURED bandwidth, so it appears as bw 1G and the duplex is omitted rather than invented. EACH PLATFORM'S OWN WORD FOR THE TABLE. PAN-OS has virtual routers, Junos has routing instances, FortiOS has VDOMs. routed in global table under a Palo Alto is wrong, and an L2 switch gets no such line at all -- it bridged the frame and consulted no table. What it checks that a per-device walk cannot Reading each device in turn finds what is wrong ON a device. Most of what breaks a path is BETWEEN them, or in a part of the forwarding decision that one show interfaces never reaches. Five of those, in the order they bite: THE INGRESS COMES FROM A REVERSE FIB LOOKUP, not from the address the hop answered with. A router may source a TTL-exceeded from a loopback, from the egress, or from whatever icmp source-address says, and inside an MPLS LSP a P router does no IP lookup at all. A lookup back toward the SOURCE gives the real ingress -- and when the two disagree, THAT IS ASYMMETRIC ROUTING, reported as a finding. It is the commonest reason a stateful firewall drops a flow that every routing table says should work. A PORT-CHANNEL IS OPENED. A flow takes exactly ONE member. A Po at 21% can have a member at 97%, and the aggregate counter -- the only one most tools print -- says the uplink is fine while that flow is being tail-dropped. The members are read individually, and a (s)uspended or (D)own member is a finding, because a flow hashed to it is a flow in the bin. THE TWO ENDS OF EACH LINK ARE COMPARED. A duplex mismatch never appears as the word "mismatch": it is late collisions on one end and CRC on the other, and either alone reads as a bad cable. Only the pair names it. Speed is compared the same way -- except across a port-channel, whose bandwidth is the SUM of its members and so never matches the single interface facing it. THE PATH MTU IS COMPUTED, and where it steps down is named. "Ping works, the application hangs" is an MTU black hole, and jumbo configured on one side of a link only is the usual way it arrives. For TCP/1433 returning large result sets it is a weekly occurrence. ECMP IS COUNTED, not just resolved. exact-route says which path THIS flow takes. It does not say four others exist -- so a clean report could be clean about one fifth of the topology and read as clean about all of it. The destination's own L2 segment The walk goes down to the port hostB is plugged into, not just to its gateway, for the same reason it does at the source end: a host is not plugged into its gateway, and the access switch between them never appears in a traceroute because a switch does not decrement TTL. On a FABRIC it stops and says so, because the MAC table does not give a local port there at all: FabricPath Ports/SWID.SSID.LID * 220 aabb.ccdd.eeff dynamic 0 F F 1.2.65535 -> "learned over FabricPath from switch-id 1.2, not on a local port. The host is behind that switch" VXLAN/EVPN * 220 aabb.ccdd.eeff dynamic 0 F F nve1(10.1.1.42) -> "learned over VXLAN from VTEP 10.1.1.42, not on a local port. The host is behind that VTEP" The entry is CLASSIFIED before the port is read, which is the whole point: a switch-id is three dotted numbers and an nve has an interface name, so a tool that just greps for something port-shaped will happily report 1.2.65535 as a port and walk into a topology it invented. Naming the switch-id or the VTEP says which leaf to go and look at, which is the useful answer anyway. The interface line The egress interface shown is the one THIS FLOW uses, which is not necessarily the one that answered the traceroute. It comes from a route lookup for the destination and, where the platform supports it, is re-resolved with the real five-tuple through exact-route. When those two disagree, ECMP is hashing your flow somewhere other than the probe, and a note says so. What is read from each interface: status up, down, administratively down duplex and HALF on a modern link is a finding of its own speed normalised, so 1G and 10G are directly comparable utilisation the worse direction against nominal bandwidth in_err crc frame the physical layer runts giants overrun ignored the receiver could not keep up out_err collisions the transmit side late_coll a DUPLEX MISMATCH, not congestion resets how many times the link was bounced drops output queue overflow: congestion, not an error inq_drops flushes the control plane falling behind, a different fault last_flap when the link last changed state up_since how long it has been up counters_age when those counters were last cleared last_in last_out traffic recency, the only clue on a quiet link A FIELD THAT COULD NOT BE READ PRINTS AS A DASH, NEVER AS ZERO. "util -" means unknown, not idle. "No CRC column found" and "zero CRC errors" are opposite findings and a tool that confuses them will call a dying link healthy. Which routing table the answer came from Every hop prints it, because a route lookup that quietly used the global table while the traffic lives in a VRF looks exactly like a correct answer: routed via global table routed via vrf PROD (discovered) routed via vrf PROD (from inventory) routed via TABLE AMBIGUOUS The table is RESOLVED, not assumed. First whatever the inventory says for that device. Failing that, the global table. And if the destination is not in the global table, the device is asked which VRFs it has and each one is tried. IF MORE THAN ONE TABLE ANSWERS, THE TOOL REFUSES TO CHOOSE. The same prefix legitimately exists in several VRFs, and which one carries your flow depends on the ingress interface. Picking one would be a coin flip presented as a finding, so the hop reports the ambiguity and names the tables, and you resolve it with a vrf: field in the inventory. The two things that make those numbers useful A counter total is not a rate. "48,000 output drops" means nothing until you know whether the counters were cleared on Tuesday or in 2019, so the age is read and printed beside them and every finding quotes it. And the counters are INTERPRETED, not listed. They do not all mean the same kind of thing, and treating them as one undifferentiated pile of "errors" is how the wrong team gets called: CRC, runts, framing the cable, the optic, or a speed/duplex mismatch. Load has nothing to do with it late collisions a duplex mismatch, almost always half on one end output drops the queue overflowing. NOT an error -- the link is simply full at times input-queue drops the control plane falling behind, which is a different fault from a full link overrun, ignored the receiver could not take the packets: buffers or CPU, not the cable interface resets the number of times the link was bounced Where the last flap comes from On NX-OS, EOS and Junos the interface itself reports when it last changed state. On IOS and IOS-XE it does not -- "show interfaces" carries no last-flap field at all -- so the walker reads the syslog buffer for LINK and LINEPROTO events on that port and reports the most recent ones. That buffer only holds what fits. "No link up/down events in the log" is therefore not proof the port has never flapped, and the report says exactly that instead of implying a clean history. The firewall line policy DENY rule Deny-SQL-From-Users (from the policy simulation) The part in brackets is the part to read, because two different questions were asked and they are not equally strong: from the live session a session matching your five-tuple exists right now and this is the rule it matched. The strongest answer available from the policy simulation no session exists, so the firewall was asked what it WOULD do. This is the answer you get for traffic that is being blocked -- which is precisely when there is no session to find policy NOT DETERMINED neither worked. Read the note. Usually the account cannot run the simulation command. ABSENCE OF A SESSION IS NOT PERMISSION The confidence bracket Every device shows [platform/confidence]. Confidence is "modelled" until that platform's output format has been verified against a real device of that type. A modelled parser that does not recognise what it receives reports that on the hop, with the raw text, instead of a clean result -- because a plausible wrong answer in a path analysis sends somebody to the wrong device in the middle of an incident. Findings HIGH is something that explains the ticket: an interface down, a link at 92%, a rule denying the flow, half duplex. MEDIUM is something worth a look: errors, drops, a recent flap. INFO is context, such as having been moved from a standby to the active member. What this walk does NOT prove Printed on every run, and worth reading before you forward the output to anybody. The short version: one direction, one moment in time, and a probe that is not your flow.
Exit Codes for Cron and CI
0 the path was walked and nothing was flagged 1 findings -- read the report 2 the run failed: bad arguments, auth rejected, gateway unreachable 1 is a result, not an error. A pipeline that treats any non-zero as failure will page you about a successful walk.
Every Flag
--src ADDR the source host --dst ADDR the destination host --dport PORT the destination port, e.g. 1433. Used by the policy lookup, so the wrong port returns the wrong rule --proto tcp|udp default tcp --gateway ADDR the source's default gateway. The walk starts here. Name either member of an HSRP/VRRP pair --gateway-os NAME its platform, if it is not in the inventory --inventory FILE YAML inventory. Needs NET_USER and NET_PASS --offline-dir DIR replay a --save-dir. No devices, no credentials --save-dir DIR save every command's output. Do this every time --util-warn N flag an interface at or above this percent. Default 70 --csv PATH write the whole path to a CSV, one row per hop The inventory file is optional Start without it. The normal run is the gateway and the flow: python3 network_path_analyzer.py --src 10.20.30.40 --dst 10.50.60.70 --dport 1433 --gateway 10.20.30.1 Everything the tool needs about each device, it asks the device: the platform from the CDP or LLDP detail of the hop before it, and where neither reaches, from SSH autodetection the hostname from its own "show version", so the report reads sw-access-01 rather than 10.10.99.5 the VRF resolved per hop against the device's own tables the model from the same output AND IT IS NOT WHERE THE RESULTS GO. The inventory is input and only input. The findings go to the terminal, to --csv as one row per hop, and to --save-dir as the raw output of every command. Nothing is ever written back to it. What it is actually for Four things that genuinely cannot be derived, and nothing else: port when SSH does not listen on 22 mgmt when the address a traceroute returns refuses SSH and the session has to go to a management address instead vrf when a destination is reachable in more than one table on that device, so the tool will not choose addresses the interface addresses a traceroute will show for a device whose management address you have listed defaults: os: ios port: 22 devices: - host: 10.99.0.1 # where the script connects device_name: core-01 addresses: [10.0.0.1] # how the script recognises this device vrf: PROD Credentials never go in it. A TRACEROUTE DOES NOT RETURN MANAGEMENT ADDRESSES. It returns each router's ingress interface -- 10.0.0.1 -- while your inventory, and the CMDB it came from, holds 10.99.0.1. Listing the transit addresses under addresses: is what connects the two. It only matters for devices you have chosen to list. How to grow it, if you ever need to Run without one. Every hop the report shows as "could not be identified" is a candidate, and its address is already in the output. Those are usually devices you do not own, or ones that refuse SSH from where you are -- in which case no inventory entry will help either, and the honest answer is the one you already have. VALID os VALUES ios iosxe iosxr nxos junos eos panos fortios asa
When Something Fails
"the gateway X is not in the inventory and no --gateway-os was given" Add it, or pass --gateway-os ios. The tool will not guess a platform for the device it is about to log into. "has no ARP entry for <source>" The host is silent, in a different VLAN, or behind a different gateway. The upstream half still runs; only the L2 segment is skipped, and the report says so rather than drawing a path it did not verify. The L2 walk stops early at a switch Its MAC table output was not recognised, which on a modelled parser is expected the first time you meet a platform. The raw reason is on the hop. Send it in and that parser becomes verified. Several hops say "not inspected: unknown platform" Neither CDP/LLDP nor the inventory identified them. Both are commonly disabled toward providers and on WAN edges -- which is exactly where a traceroute crosses equipment you do not own. List the ones you do own. "policy NOT determined" Almost always the account. Compare that platform's line in Authorisation with what the account is allowed to run. It is not an indication that the firewall is permitting the traffic. The path looks nothing like what I expected Check whether you were moved from a standby to an active member -- that is an INFO finding -- and read the ECMP note. A probe that hashes differently from your flow draws a path your application does not take. Counters look alarming but the link is fine Check the age beside them. A total accumulated since 2019 on a port that was recabled last year is a souvenir, not a fault.
Tags
traceroutepathtroubleshootingfirewallhsrpcdplldputilizationpacket-loss