Monitoring · python · 19 copies
Juniper SRX Health Check
Read-only health check for the whole Juniper SRX range — branch, mid-range and high-end, standalone or in a chassis cluster — written against Junos OS 21.4R3 and later, that writes a self-contained HTML report you can forward or print. Twenty-two checks per device: alarms, Routing Engine and SPU load, fans, storage, FPC and PIC slots, interfaces and LACP bundles, the flow session table, IKE and IPsec state, licences, and the whole cluster — status, control and fabric links, heartbeats, the redundancy group history and the cluster LED, all of which one node reports for both. Polling both nodes adds the confirmation that both are forwarding at once. Two versions ship here — netmiko with a YAML inventory for many devices, and a dependency-free SecureCRT script for the session you already have open.
- Junos
Dependencies
pip install "netmiko>=4.3" "PyYAML>=6.0"#!/usr/bin/env python3
"""
Juniper SRX Health Check
READ THE DOCUMENTATION BEFORE YOU RUN THIS:
https://permitany.com/scripts/junos-srx-healthcheck
It covers what the script does, whether it is safe to run, the
inventory file and credentials it needs, how to read the output,
and what to do when something fails.
pip install "netmiko>=4.3" "PyYAML>=6.0"
"""
from __future__ import annotations
import argparse
import json
import os
import re
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass, field, asdict
from datetime import datetime, timezone
from pathlib import Path
try:
import yaml
except ImportError:
yaml = None
from netmiko import ConnectHandler
from netmiko.exceptions import NetmikoAuthenticationException, NetmikoTimeoutException
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
# Written and verified against Junos OS 21.4R3 / 22.4R3 / 23.4R2 on SRX.
# Every command below is operational mode and read-only. Nothing in this file
# enters configuration mode.
JUNOS_TARGET = "Junos OS 21.4R3 and later (SRX branch, mid-range and high-end)"
# ---------------------------------------------------------------------------
# Inventory loading and connection
# ---------------------------------------------------------------------------
NETMIKO_DRIVER = {
"ios": "cisco_ios",
"iosxe": "cisco_ios",
"iosxr": "cisco_xr",
"nxos": "cisco_nxos",
"junos": "juniper_junos",
"eos": "arista_eos",
"panos": "paloalto_panos",
"fortios": "fortinet",
}
def load_inventory(path: str) -> list[dict]:
"""
Read devices.yaml (or .json) and return a list of device dicts.
Applies the `defaults:` block to every device, validates the `os` field,
and fails with a readable message rather than a stack trace.
"""
p = Path(path)
if not p.exists():
raise SystemExit(
f"Inventory file not found: {path}\n"
f"Create it with:\n\n"
f" devices:\n"
f" - host: 192.168.1.1\n"
f" os: ios\n"
)
raw = p.read_text(encoding="utf-8")
try:
if p.suffix.lower() in (".yaml", ".yml"):
if yaml is None:
raise SystemExit("PyYAML is not installed. Run: pip install PyYAML")
data = yaml.safe_load(raw)
else:
data = json.loads(raw)
except SystemExit:
raise
except Exception as exc:
raise SystemExit(f"Could not parse {path}: {exc}\n"
f"Check for tabs (use spaces) and missing spaces after colons.")
if isinstance(data, dict):
defaults = data.get("defaults", {}) or {}
devices = data.get("devices", []) or []
else:
defaults, devices = {}, data
if not devices:
raise SystemExit(f"No devices found in {path}. The file needs a `devices:` list.")
out = []
for i, d in enumerate(devices, 1):
if not isinstance(d, dict):
raise SystemExit(f"Device #{i} in {path} is not a mapping. "
f"Each entry needs at least `host:` and `os:`.")
merged = {**defaults, **d}
if "host" not in merged:
raise SystemExit(f"Device #{i} in {path} has no `host:` field.")
os_key = merged.get("os")
if not os_key:
raise SystemExit(f"Device '{merged['host']}' has no `os:` field. "
f"Valid values: {', '.join(sorted(NETMIKO_DRIVER))}")
if os_key not in NETMIKO_DRIVER:
raise SystemExit(
f"Device '{merged['host']}' has os '{os_key}', which is not supported.\n"
f"Valid values: {', '.join(sorted(NETMIKO_DRIVER))}\n"
f"Note: 'cisco' is a vendor, not an OS - use ios, iosxe, iosxr, or nxos."
)
merged.setdefault("vrfs", [])
out.append(merged)
return out
def credentials() -> tuple[str, str, str]:
"""Read credentials from the environment and fail clearly if missing."""
user = os.environ.get("NET_USER", "")
pwd = os.environ.get("NET_PASS", "")
enable = os.environ.get("NET_ENABLE", "")
if not user or not pwd:
raise SystemExit(
"Set NET_USER and NET_PASS in your environment before running.\n\n"
" Linux/macOS: export NET_USER='admin'\n"
" export NET_PASS='your-password'\n\n"
" PowerShell: $env:NET_USER = 'admin'\n"
" $env:NET_PASS = 'your-password'\n"
)
return user, pwd, enable
def connect(dev: dict):
"""Open an SSH session to one device using netmiko."""
user, pwd, enable = credentials()
params = {
"device_type": NETMIKO_DRIVER[dev["os"]],
"host": dev["host"],
"username": dev.get("username") or user,
"password": dev.get("password") or pwd,
"port": dev.get("port", 22),
"conn_timeout": 20,
"fast_cli": False,
}
secret = dev.get("secret") or enable
if secret:
params["secret"] = secret
conn = ConnectHandler(**params)
if secret and dev["os"] in ("ios", "iosxe", "nxos"):
conn.enable()
return conn
# ---------------------------------------------------------------------------
# Severity model
#
# OK / WARNING / CRITICAL are verdicts. UNKNOWN is NOT a verdict: it means the
# command did not answer, or this platform does not implement it, so the check
# could not be decided. "Could not verify" and "verified as bad" are different
# facts and this script never merges them. N/A means the check does not apply
# (cluster checks on a standalone box).
#
# A value that could not be read prints as a dash. Never as zero.
# ---------------------------------------------------------------------------
OK = "OK"
INFO = "INFO"
WARN = "WARNING"
CRIT = "CRITICAL"
UNKNOWN = "UNKNOWN"
NA = "N/A"
RANK = {CRIT: 0, WARN: 1, UNKNOWN: 2, INFO: 3, OK: 4, NA: 5}
DASH = "\u2014"
@dataclass
class Check:
name: str
status: str
summary: str
detail: list[str] = field(default_factory=list)
headers: list[str] = field(default_factory=list)
rows: list[list[str]] = field(default_factory=list)
# One severity per row of `rows`, same order and same length - or empty
# when the check does not grade element by element. It is filled by the
# check itself, from the SAME condition that decided its verdict: a
# second grader reading the Status column on its own would be a second
# truth, and it could contradict the verdict. Empty is a correct answer,
# not an unfinished one.
row_status: list[str] = field(default_factory=list)
@dataclass
class DeviceResult:
host: str
label: str
status: str = "ok"
error: str = ""
hostname: str = DASH
model: str = DASH
serial: str = DASH
version: str = DASH
uptime: str = DASH
booted: str = DASH
reboot_reason: str = DASH
role: str = DASH
# Which node of the cluster this box is - the heading of its column in
# every table of the report. See finalise() for where it comes from: the
# CLI prompt first, because a clustered SRX names the answering node in it
# and a prompt cannot be out of date.
node: str = ""
node_source: str = ""
node_note: str = ""
cluster_id: str = ""
cluster_group: str = ""
rg_primaries: dict = field(default_factory=dict)
rg_failovers: dict = field(default_factory=dict)
# Every node row of every redundancy group, as the cluster reported it:
# {rg: [[node, priority, status, preempt, manual, monitor], ...]}. This is
# what lets the report print one row per redundancy group with a column
# per node instead of repeating the whole table under each box.
rg_nodes: dict = field(default_factory=dict)
checks: list[Check] = field(default_factory=list)
def worst(self) -> str:
real = [c.status for c in self.checks if c.status != NA]
if self.status != "ok":
return UNKNOWN
return min(real, key=lambda s: RANK[s]) if real else UNKNOWN
# ---------------------------------------------------------------------------
# Commands. Operational mode only, every one of them read-only.
# ---------------------------------------------------------------------------
BASE_COMMANDS = {
"version": "show version",
"hardware": "show chassis hardware",
"uptime": "show system uptime",
"re": "show chassis routing-engine",
"chassis_alarms": "show chassis alarms",
"system_alarms": "show system alarms",
"storage": "show system storage",
"cores": "show system core-dumps",
"ifaces": "show interfaces terse",
"sessions": "show security flow session summary",
"ipsec": "show security ipsec security-associations",
"ike": "show security ike security-associations",
"license": "show system license",
"commits": "show system commit",
"fpc": "show chassis fpc detail",
"pics": "show chassis fpc pic-status",
"env": "show chassis environment",
"lacp": "show lacp interfaces",
"lldp": "show lldp neighbors",
}
CLUSTER_COMMANDS = {
"cl_status": "show chassis cluster status",
"cl_info": "show chassis cluster information",
"cl_ifaces": "show chassis cluster interfaces",
"cl_stats": "show chassis cluster statistics",
}
ERRORS_COMMAND = "show interfaces extensive"
# ---------------------------------------------------------------------------
# High-end only: the data plane
#
# On an SRX5400/5600/5800 the Routing Engine can be idle while the SPUs are
# saturated, so `show chassis routing-engine` and the session summary are not
# enough. These two commands look at the services processing cards.
#
# `show security monitoring fpc <n>` exists since Junos 9.2 and reports FPC,
# PIC, CPU Utilization (%), Memory Utilization (%), Current flow session, Max
# flow session, SPU current cp session and SPU max cp session.
#
# `show security monitoring performance spu summary` exists since 10.2 and
# gives SPU utilisation per second over the last 60 seconds. Its `fpc` and
# `pic` arguments are NOT available on SRX300/320/340, SRX1500/1600/2300,
# SRX4100/4120/4200/4300/4600 or vSRX - which is the other half of why these
# checks run on high-end only and are reported as N/A everywhere else.
#
# Juniper documents the field NAMES but not an example of the output, so the
# column layout here is unverified. The parser tries a key-value shape and
# then a header-plus-rows shape, and when neither fits it returns UNKNOWN.
# It never falls back to a number it is not sure about.
# ---------------------------------------------------------------------------
HIGH_END = re.compile(r"srx\s*(5400|5600|5800)", re.I)
SPU_PERF_COMMAND = "show security monitoring performance spu summary"
def spu_fpc_command(slot) -> str:
return f"show security monitoring fpc {slot}"
def is_high_end(version_text: str | None) -> bool:
"""SRX5400/5600/5800 only. An unreadable `show version` is not high-end."""
model = grab(r"^Model:\s*(\S+)", version_text)
return bool(model and HIGH_END.search(model))
# `show chassis fpc`, documented for SRX5600/5800 among others. The slot
# number and the state share one column, which Juniper's field table calls
# "Slot or Slot State". These are the nine documented states; the parser only
# accepts a line whose second token is one of them, which is what stops it
# reading a number out of some unrelated table.
FPC_STATES = {
"online": "in service",
"empty": "no FPC in the slot",
"offline": "configured offline",
"dead": "held in reset because of errors",
"dormant": "held in reset",
"diag": "ignored while diagnostics run",
"present": "detected but not in service",
"probed": "probe complete, waiting for the PFE to restart",
"probe-wait": "waiting to be probed",
}
# Not a fault: a slot can be legitimately empty, and offline is a decision.
FPC_BENIGN = {"empty", "offline"}
FPC_DETAIL_SLOT = re.compile(r"^Slot\s+(\d+)\s+information\s*:", re.I | re.M)
def parse_chassis_fpc_detail(text: str) -> list[dict]:
"""
The same slot states, out of `show chassis fpc detail`.
`detail` prints one indented block per slot instead of one row per slot,
so the tabular parser below finds nothing in it. This one reads the blocks
and also picks up the extra per-FPC fields - temperature, the DRAM the CPU
was given, and CPU and memory utilisation where the platform prints them.
Those extra fields are REPORTED AND NEVER GRADED. Junos publishes no
threshold for any of them, and a number invented here would read like a
vendor limit without being one. The state is the verdict; the rest is
context, and a field that was not printed is a dash and not a zero.
"""
rows = []
bounds = [(m.start(), m.group(1)) for m in FPC_DETAIL_SLOT.finditer(text)]
for i, (pos, slot) in enumerate(bounds):
end = bounds[i + 1][0] if i + 1 < len(bounds) else len(text)
blk = text[pos:end]
state = grab(r"^\s*State\s+([A-Za-z][A-Za-z-]*)\s*$", blk)
if not state or state.lower() not in FPC_STATES:
continue
note = ""
for phrase in ("Hardware Not Supported", "Hardware Not In Right Slot"):
if phrase.lower() in blk.lower():
note = phrase
rows.append({
"slot": slot,
"state": state.lower(),
"note": note,
"temp": grab(r"^\s*Temperature\s+(\d+)\s*degrees C", blk),
"cpu": grab(r"^\s*CPU utilization\s+(\d+)\s*percent", blk),
"mem": grab(r"^\s*Memory utilization\s+(\d+)\s*percent", blk),
"dram": grab(r"^\s*Total CPU DRAM\s+(\d+)\s*MB", blk),
})
return rows
def parse_chassis_fpc(text: str) -> list[dict]:
"""
Slot, state and the hardware note, from `show chassis fpc detail`.
The block layout of `detail` is tried first. A device that answers in the
older one-row-per-slot table still parses, because the fallback below is
the parser that was written for that shape - the state check is the same
check either way.
Juniper's page leaves the sample output blank, so the column spacing is
unverified. The guard is the STATE VOCABULARY rather than the position: a
line counts only when its second token is one of the nine documented
states. Returns [] on a shape it does not recognise, and the callers turn
that into UNKNOWN rather than assuming slot 0 exists.
"""
detail = parse_chassis_fpc_detail(text)
if detail:
return detail
rows = []
lines = text.splitlines()
for i, line in enumerate(lines):
m = re.match(r"^\s*(\d{1,2})\s+([A-Za-z][A-Za-z-]*)\b(.*)$", line)
if not m:
continue
state = m.group(2).lower()
if state not in FPC_STATES:
continue
tail = m.group(3) + " " + (lines[i + 1] if i + 1 < len(lines) else "")
note = ""
for phrase in ("Hardware Not Supported", "Hardware Not In Right Slot"):
if phrase.lower() in tail.lower():
note = phrase
rows.append({"slot": m.group(1), "state": state, "note": note,
"temp": None, "cpu": None, "mem": None, "dram": None})
return rows
def online_fpc_slots(fpc_text: str | None) -> list[str]:
if not fpc_text:
return []
return [r["slot"] for r in parse_chassis_fpc(fpc_text)
if r["state"] == "online"]
# A Junos box answers an unimplemented or unauthorised command with one of
# these. Treating that as "healthy" is exactly the lie this script must not
# tell, so it becomes UNKNOWN instead.
JUNOS_ERROR = re.compile(
r"^(error:|unknown command|syntax error|\s*\^$|.*permission denied)", re.I | re.M
)
NOT_CLUSTERED = re.compile(
r"(not enabled|Chassis cluster is not enabled|not configured for chassis cluster)", re.I
)
def read_prompt(conn) -> str:
"""
The CLI prompt block, read for ONE reason: on a chassis cluster Junos
prints the node that is answering on the line just above the prompt -
`{primary:node0}` - and that is the only source of the node ID which
cannot be out of date.
A bare carriage return, with no command behind it, is what brings it
back. netmiko's find_prompt() is deliberately not used: it returns only
the LAST line, which is the one WITHOUT the node in it.
Never allowed to break a run. A prompt that cannot be read only means the
node is worked out from the next source in finalise().
"""
try:
return str(conn.send_command_timing("", strip_prompt=False,
strip_command=False) or "")
except KeyboardInterrupt:
raise
except Exception: # noqa: BLE001 - the prompt is never worth a failed run
return ""
def send(conn, cmd: str, timeout: int) -> tuple[str | None, str]:
"""Send one read-only command. Returns (output, error). Never raises."""
try:
out = conn.send_command(cmd, read_timeout=timeout)
except KeyboardInterrupt:
raise
except Exception as exc: # noqa: BLE001 - one command must not abort the run
return None, f"{type(exc).__name__}: {exc}"
if not out or not out.strip():
return None, "the device returned no output"
# "chassis cluster is not enabled" arrives as an error on some releases.
# It is an ANSWER, not a failure: the box is standalone. Checking it first
# is what keeps a standalone SRX out of the undetermined column.
if NOT_CLUSTERED.search(out):
return out, ""
if JUNOS_ERROR.search(out):
first = next((l.strip() for l in out.splitlines() if l.strip()), "")
return None, f"command rejected: {first[:110]}"
return out, ""
def grab(pat: str, text: str | None, group: int = 1) -> str | None:
if not text:
return None
m = re.search(pat, text, re.I | re.M)
return m.group(group).strip() if m else None
def grab_int(pat: str, text: str | None, group: int = 1) -> int | None:
v = grab(pat, text, group)
if v is None:
return None
try:
return int(v)
except ValueError:
return None
def show(v) -> str:
"""A value that could not be read prints as a dash, never as zero."""
return DASH if v is None else str(v)
# ---------------------------------------------------------------------------
# The checks. Each one takes the raw command output and returns a Check.
# None of them ever guesses: a counter it cannot find becomes a dash and the
# check becomes UNKNOWN.
# ---------------------------------------------------------------------------
# A chassis cluster puts the node that is answering into the CLI prompt:
#
# {primary:node0}
# admin@srx-dc-node0>
#
# The word in braces is the ROLE, and it changes at a failover. The `nodeN`
# beside it is the fixed node ID, which is the part worth reading. All six
# redundancy states are accepted and not just primary and secondary, because
# a node sitting in hold or ineligible still has to land in the right column.
# A standalone SRX has no braces in its prompt, so this returns nothing and
# the next source is tried.
NODE_PROMPT = re.compile(
r"\{(?:primary-hold|primary|secondary-hold|secondary|hold|ineligible"
r"|disabled)[^}]*:\s*(node\d+)\s*\}", re.I)
NODE_BANNER = re.compile(r"^\s*(node[01])\s*:\s*$", re.I | re.M)
def node_from_prompt(text: str | None) -> str:
"""The node ID out of the CLI prompt, or "" when the prompt has none."""
m = NODE_PROMPT.search(text or "")
return m.group(1).lower() if m else ""
def sole_node_banner(raw: dict) -> str:
"""
Which node answered, when the device said so itself and said it ONCE.
A clustered SRX prefixes its own operational output with a `node0:` or
`node1:` banner. If exactly one banner appears across the outputs read
here, that is the node this session is on. If TWO appear, the output is
the cluster-wide view and it identifies nothing about which node replied -
so nothing is inferred, which is the whole point: a wrong node label would
put the right finding in the wrong column.
"""
seen = set()
for key in ("cl_info", "version", "hardware", "re"):
text = (raw.get(key) or (None, ""))[0]
if text:
seen |= {m.lower() for m in NODE_BANNER.findall(text)}
return next(iter(seen)) if len(seen) == 1 else ""
def finalise(res: DeviceResult, raw: dict) -> None:
"""
The two things that can only be settled once every check has run.
Where the node ID comes from, in this order:
1. THE CLI PROMPT. A clustered SRX writes the answering node into it,
reading it costs no extra command, and it is the only source that
cannot be out of date. This is what makes the two-column report come
out right on the first run, with nobody editing an inventory.
2. The `node:` field in the inventory, as an explicit manual override
for a prompt this script cannot read - a customised prompt, or a
session that does not show the cluster braces.
3. The single node banner the device prints over its own output, and
only when exactly ONE appears: output that names both nodes says
nothing about which one answered.
4. Nothing, and the column is headed with the device label instead. A
wrong node label would put the right finding in the wrong column, so
this step never guesses.
When the prompt and the inventory disagree the PROMPT wins - a device
cannot be wrong about its own identity and an inventory can be stale -
and the report says so instead of discarding the declaration quietly.
"""
if res.role == DASH:
res.role = "standalone"
declared = res.node
from_prompt = node_from_prompt((raw.get("_prompt") or (None, ""))[0])
if from_prompt:
res.node, res.node_source = from_prompt, "prompt"
if declared and declared != from_prompt:
res.node_note = (
f"The inventory declares {res.label} as {declared}, but the "
f"firewall's own CLI prompt answers as {from_prompt}. The "
f"prompt is what this report used. Correct the `node:` field, "
f"or remove it and let the prompt speak.")
elif declared:
res.node_source = "inventory"
else:
guess = sole_node_banner(raw)
if guess:
res.node, res.node_source = guess, "node banner"
def uptime_days(uptime_text: str | None, re_text: str | None) -> int | None:
"""Days since boot, from `show system uptime` or `show chassis routing-engine`."""
m = re.search(r"System booted:.*?\((?:(\d+)w)?(?:(\d+)d)?\s*([\d:]+)?\s*ago\)",
uptime_text or "", re.I)
if m and (m.group(1) or m.group(2)):
return int(m.group(1) or 0) * 7 + int(m.group(2) or 0)
if m:
return 0
return grab_int(r"^\s*Uptime\s+(\d+) days", re_text)
def check_identity(res: DeviceResult, raw: dict) -> Check:
ver, _ = raw["version"]
up, _ = raw["uptime"]
re_out, _ = raw["re"]
res.hostname = show(grab(r"^Hostname:\s*(\S+)", ver))
res.model = show(grab(r"^Model:\s*(\S+)", ver))
res.version = show(
grab(r"^Junos:\s*(\S+)", ver) or grab(r"JUNOS .*?\[(\S+?)\]", ver)
)
booted = grab(r"System booted:\s*(.+?)\s*$", up)
days = uptime_days(up, re_out)
res.uptime = f"{days} day(s)" if days is not None else DASH
reason = grab(r"Last reboot reason\s+(.+?)\s*$", re_out)
res.booted = show(booted)
res.reboot_reason = show(reason)
# No detail list. Everything this check reads - model, Junos, uptime, boot
# time, boot reason - is printed ONCE, in the equipment table at the top of
# the report. It used to print the same four fields again in a block of its
# own, which is where half the repetition in the report came from.
detail: list[str] = []
if ver is None and up is None:
return Check("Software and uptime", UNKNOWN,
"neither `show version` nor `show system uptime` answered",
detail)
if days is None:
return Check("Software and uptime", UNKNOWN,
"could not read the uptime, which is not the same thing as "
"finding no reboot", detail)
if days < 1:
return Check("Software and uptime", WARN,
f"rebooted less than 24 hours ago (up {days} day(s))", detail)
return Check("Software and uptime", OK,
f"up {days} day(s) on Junos {res.version}", detail)
CHASSIS_LINE = re.compile(r"^Chassis\s+(\S+)(?:\s+(.*?))?\s*$")
# A serial number on Junos is an unbroken run of letters and digits. The
# Version and Part number columns are blank on the Chassis line of every SRX
# checked, so the first token after "Chassis" is the serial - but it is only
# accepted when it LOOKS like one. On a box that prints something else there,
# the serial becomes a dash rather than a confident wrong answer.
SERIAL_SHAPE = re.compile(r"^[A-Za-z0-9]{6,}$")
def check_hardware(res: DeviceResult, raw: dict) -> Check:
"""
Model and chassis serial number, from `show chassis hardware`.
INFO, always. A serial number cannot be a finding. It is collected because
every RMA, licence request and support case starts with it, and because on
a cluster it is the only field in the summary that tells the two nodes
apart at a glance - the hostname can be anything, the serial is the box.
"""
name = "Chassis hardware"
headers = ["Node", "Serial number", "Description"]
text, err = raw.get("hardware", (None, "not collected"))
if text is None:
return Check(name, UNKNOWN,
"`show chassis hardware` did not answer, so the chassis "
"serial number is undetermined", [err], headers, [])
node, rows = "", []
for line in text.splitlines():
m = re.match(r"^\s*(node[01])\s*:\s*$", line, re.I)
if m:
node = m.group(1).lower()
continue
m = CHASSIS_LINE.match(line.rstrip())
if not m:
continue
serial, desc = m.group(1), (m.group(2) or "").strip()
if not SERIAL_SHAPE.match(serial):
serial, desc = DASH, line.strip()[len("Chassis"):].strip()
rows.append([node or DASH, serial, desc or DASH])
if not rows:
return Check(name, UNKNOWN,
"the command answered but no Chassis line could be read, "
"so the serial number is undetermined", [], headers, [])
res.serial = rows[0][1]
if res.model in (DASH, "") and rows[0][2] != DASH:
res.model = rows[0][2]
# No detail and no table. The serial number is printed ONCE, in the
# equipment table at the top of the report, which is where it is decided.
# What is kept here is the count, because a command that reported more
# chassis than this script put in the summary is worth saying out loud.
detail: list[str] = []
extra = ""
if len(rows) > 1:
extra = f", {len(rows)} chassis reported"
detail.append(
f"This command reported {len(rows)} chassis. The equipment table "
f"shows the first, which is the same node whose hostname "
f"`show version` reported first.")
return Check(name, INFO, f"chassis serial {res.serial}{extra}",
detail, [], [])
def parse_alarm_table(text: str) -> list[list[str]]:
rows = []
for line in text.splitlines():
m = re.match(r"^(\d{4}-\d{2}-\d{2} \S+(?: \S+)?)\s+(Major|Minor)\s+(.+?)\s*$",
line.strip(), re.I)
if m:
rows.append([m.group(1), m.group(2).title(), m.group(3)])
return rows
def check_alarms(raw: dict) -> Check:
rows: list[list[str]] = []
missing = []
for key, label in (("chassis_alarms", "chassis"), ("system_alarms", "system")):
text, err = raw[key]
if text is None:
missing.append(f"{label}: {err}")
continue
for r in parse_alarm_table(text):
rows.append([label] + r)
headers = ["Source", "Raised", "Class", "Description"]
if missing and not rows:
return Check("Chassis and system alarms", UNKNOWN,
"could not read the alarm tables", missing, headers, rows)
if not rows:
if missing:
return Check("Chassis and system alarms", UNKNOWN,
"one alarm table was clean and the other did not answer, "
"so the device is not cleared", missing, headers, rows)
return Check("Chassis and system alarms", OK,
"no alarms currently active",
["No alarms currently active."], headers, rows)
major = sum(1 for r in rows if r[2].lower() == "major")
sev = [CRIT if r[2].lower() == "major" else WARN for r in rows]
return Check("Chassis and system alarms", CRIT if major else WARN,
f"{len(rows)} alarm(s) active, {major} of them Major",
missing, headers, rows, sev)
def check_routing_engine(raw: dict) -> Check:
text, err = raw["re"]
if text is None:
return Check("Routing Engine load", UNKNOWN,
"`show chassis routing-engine` did not answer", [err])
idle = grab_int(r"^\s*Idle\s+(\d+) percent", text)
mem = (grab_int(r"Memory utilization\s+(\d+) percent", text)
or grab_int(r"Max\s+\d+\s*MB used\s*\(\s*(\d+) percent\)", text))
temp = grab_int(r"^\s*Temperature\s+(\d+) degrees C", text)
detail = [
f"CPU idle: {show(idle)}" + ("%" if idle is not None else ""),
f"Memory used: {show(mem)}" + ("%" if mem is not None else ""),
f"Temperature: {show(temp)}" + (" C" if temp is not None else ""),
"Temperature is reported, not graded: the safe range differs per SRX "
"model, and an over-temperature condition raises a chassis alarm, which "
"this report checks separately.",
]
if idle is None and mem is None:
return Check("Routing Engine load", UNKNOWN,
"the command answered but neither the CPU nor the memory "
"counter could be parsed on this release", detail)
problems = []
status = OK
if idle is not None and idle < 10:
status = CRIT
problems.append(f"CPU idle {idle}% - the Routing Engine is saturated")
elif idle is not None and idle < 20:
status = WARN
problems.append(f"CPU idle {idle}%")
if mem is not None and mem >= 95:
status = CRIT
problems.append(f"memory {mem}% used")
elif mem is not None and mem >= 85 and status != CRIT:
status = WARN
problems.append(f"memory {mem}% used")
if status == OK:
return Check("Routing Engine load", OK,
f"CPU idle {show(idle)}%, memory {show(mem)}% used", detail)
if idle is None or mem is None:
detail.append("Only one of the two counters could be read, so this "
"verdict covers only that one.")
return Check("Routing Engine load", status, "; ".join(problems), detail)
def check_storage(raw: dict) -> Check:
text, err = raw["storage"]
if text is None:
return Check("Filesystem usage", UNKNOWN,
"`show system storage` did not answer", [err])
# devfs and the other pseudo filesystems sit at 100% on every healthy
# Junos box. Grading them would report every SRX in the estate as
# critical, which is the kind of confident lie that gets a tool ignored.
PSEUDO = ("devfs", "procfs", "linprocfs", "fdescfs", "tmpfs")
rows = []
for line in text.splitlines():
m = re.match(r"^(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\d+)%\s+(\S+)\s*$", line)
if not m:
continue
fs, mount = m.group(1), m.group(6)
if fs.lower() in PSEUDO or mount.rstrip("/").endswith("/dev"):
continue
rows.append([mount, fs, m.group(2), m.group(5) + "%"])
headers = ["Mounted on", "Filesystem", "Size", "Used"]
if not rows:
return Check("Filesystem usage", UNKNOWN,
"the command answered but no filesystem line could be parsed",
[err or "unexpected column layout"], headers, rows)
worst = max(int(r[3].rstrip("%")) for r in rows)
hot = [r[0] for r in rows if int(r[3].rstrip("%")) >= 80]
sev = [CRIT if int(r[3].rstrip("%")) >= 90
else WARN if int(r[3].rstrip("%")) >= 80 else OK for r in rows]
if worst >= 90:
return Check("Filesystem usage", CRIT,
f"{', '.join(hot)} at {worst}% - a full /var stops logging "
f"and can block a commit", [], headers, rows, sev)
if worst >= 80:
return Check("Filesystem usage", WARN,
f"{', '.join(hot)} at {worst}%", [], headers, rows, sev)
return Check("Filesystem usage", OK,
f"{len(rows)} filesystem(s), busiest at {worst}%",
[], headers, rows, sev)
def check_core_dumps(raw: dict) -> Check:
text, err = raw["cores"]
if text is None:
return Check("Core dumps", UNKNOWN,
"`show system core-dumps` did not answer", [err])
files = [l.strip() for l in text.splitlines()
if "core" in l.lower() and "No such file" not in l
and re.search(r"\s\d{2}:\d{2}\s|\s\d{4}\s", l)]
if not files:
return Check("Core dumps", OK, "no core dumps on the device", [])
return Check("Core dumps", WARN,
f"{len(files)} core dump(s) present - a daemon crashed at some "
f"point and the evidence is still on disk",
files[:12] + (["..."] if len(files) > 12 else []))
PHYSICAL = re.compile(r"^(ge|xe|et|fe|reth|ae|fab|lo0|em|fxp|irb|st0)", re.I)
def check_interfaces(raw: dict) -> Check:
text, err = raw["ifaces"]
if text is None:
return Check("Interface state", UNKNOWN,
"`show interfaces terse` did not answer", [err])
down, admin_down, total = [], [], 0
for line in text.splitlines():
m = re.match(r"^(\S+)\s+(up|down)\s+(up|down)\b", line)
if not m:
continue
name, admin, link = m.group(1), m.group(2), m.group(3)
if "." in name or not PHYSICAL.match(name):
continue
total += 1
if admin == "down":
admin_down.append(name)
elif link == "down":
down.append(name)
headers = ["Interface", "Condition"]
rows = [[n, "admin up, link DOWN"] for n in down]
rows += [[n, "administratively down"] for n in admin_down]
if total == 0:
return Check("Interface state", UNKNOWN,
"the command answered but no physical interface line could "
"be parsed", [], headers, rows)
dataplane = [n for n in down if n.lower().startswith(("reth", "fab"))]
# The same test, row by row. An administratively down port is INFO and
# never marked: it is a decision somebody took, which is why it is not
# graded in the verdict either.
sev = [CRIT if n.lower().startswith(("reth", "fab")) else WARN
for n in down] + [INFO] * len(admin_down)
detail = [f"{total} physical interface(s) seen, {len(admin_down)} "
f"administratively down - which is a choice, not a fault"]
if dataplane:
return Check("Interface state", CRIT,
f"{', '.join(dataplane)} is enabled but has no link, and "
f"that is a cluster data or fabric path", detail, headers,
rows, sev)
if down:
return Check("Interface state", WARN,
f"{len(down)} interface(s) enabled with no link: "
f"{', '.join(down[:6])}", detail, headers, rows, sev)
return Check("Interface state", OK,
f"every enabled interface has link ({total} seen)",
detail, headers, rows, sev)
DASH_NOTE = ("A dash means the counter was not present in the output, not that "
"it was zero. Those are opposite findings.")
def check_interface_errors(raw: dict) -> Check:
text, err = raw.get("errors", (None, "not collected - pass --check-errors"))
if text is None:
return Check("Interface error counters", UNKNOWN,
"not collected, so errors are undetermined rather than "
"absent", [err])
blocks = re.split(r"^Physical interface:\s*", text, flags=re.M)[1:]
headers = ["Interface", "Input errors", "Framing", "CRC/Align",
"Output errors", "Carrier transitions"]
rows, bad, sev, parsed = [], [], [], 0
for b in blocks:
name = b.split(",", 1)[0].strip()
if not PHYSICAL.match(name):
continue
inp = b.split("Input errors:", 1)[1][:400] if "Input errors:" in b else ""
outp = b.split("Output errors:", 1)[1][:400] if "Output errors:" in b else ""
in_err = grab_int(r"Errors:\s*(\d+)", inp)
framing = grab_int(r"Framing errors:\s*(\d+)", inp)
crc = grab_int(r"CRC/Align errors\s+(\d+)", b)
out_err = grab_int(r"Errors:\s*(\d+)", outp)
carrier = grab_int(r"Carrier transitions:\s*(\d+)", outp)
if all(v is None for v in (in_err, framing, crc, out_err, carrier)):
continue
parsed += 1
rows.append([name, show(in_err), show(framing), show(crc),
show(out_err), show(carrier)])
if any(v for v in (in_err, framing, crc, out_err) if v):
bad.append(name)
sev.append(WARN)
elif carrier and carrier > 2:
bad.append(name)
sev.append(WARN)
else:
sev.append(OK)
if parsed == 0:
return Check("Interface error counters", UNKNOWN,
"the command answered but no error counter could be located",
[DASH_NOTE], headers, rows)
if bad:
return Check("Interface error counters", WARN,
f"{len(bad)} interface(s) with non-zero errors or repeated "
f"carrier transitions: {', '.join(sorted(set(bad))[:6])}",
[DASH_NOTE], headers, rows, sev)
return Check("Interface error counters", OK,
f"no errors on {parsed} interface(s)", [DASH_NOTE], headers,
rows, sev)
def check_flow_sessions(raw: dict) -> Check:
text, err = raw["sessions"]
if text is None:
return Check("Flow session table", UNKNOWN,
"`show security flow session summary` did not answer", [err])
in_use = (grab_int(r"Sessions-in-use:\s*(\d+)", text)
or grab_int(r"Valid sessions:\s*(\d+)", text))
maximum = grab_int(r"Maximum-sessions:\s*(\d+)", text)
failed = grab_int(r"Failed-sessions:\s*(\d+)", text)
detail = [f"Sessions in use: {show(in_use)}",
f"Maximum sessions: {show(maximum)}",
f"Failed sessions: {show(failed)}"]
if in_use is None or not maximum:
return Check("Flow session table", UNKNOWN,
"could not read both the in-use and the maximum session "
"count, so the utilisation is undetermined", detail)
pct = round(in_use * 100 / maximum, 1)
detail.append(f"Utilisation: {pct}%")
if pct >= 90:
return Check("Flow session table", CRIT,
f"session table {pct}% full ({in_use}/{maximum}) - new flows "
f"are dropped once it fills", detail)
if pct >= 80:
return Check("Flow session table", WARN,
f"session table {pct}% full ({in_use}/{maximum})", detail)
return Check("Flow session table", OK,
f"{in_use} of {maximum} sessions in use ({pct}%)", detail)
def check_vpn(raw: dict) -> Check:
ipsec, ipsec_err = raw["ipsec"]
ike, ike_err = raw["ike"]
if ipsec is None and ike is None:
return Check("IPsec VPN", UNKNOWN,
"neither the IKE nor the IPsec security-association table "
"answered", [ipsec_err, ike_err])
tunnels = grab_int(r"Total active tunnels:\s*(\d+)", ipsec)
if tunnels is None and ipsec:
gws = set(re.findall(r"\b(?:\d{1,3}\.){3}\d{1,3}\b", ipsec))
tunnels = len(gws) if gws else None
ike_up = len(re.findall(r"^\s*\d+\s+UP\b", ike or "", re.M))
ike_down = len(re.findall(r"^\s*\d+\s+DOWN\b", ike or "", re.M))
detail = [f"IPsec tunnels up: {show(tunnels)}",
f"IKE SAs up: {ike_up if ike else DASH}",
f"IKE SAs down: {ike_down if ike else DASH}"]
if ike_down:
return Check("IPsec VPN", CRIT,
f"{ike_down} IKE SA(s) in DOWN state", detail)
if ike_up and tunnels == 0:
return Check("IPsec VPN", CRIT,
f"{ike_up} IKE SA(s) up but no IPsec SA - phase 1 completed "
f"and phase 2 did not, which is a proposal or a proxy-ID "
f"mismatch", detail)
if tunnels is None and ike_up == 0 and (ipsec or ike):
return Check("IPsec VPN", NA,
"no IKE or IPsec state on this device", detail)
if tunnels is None:
return Check("IPsec VPN", UNKNOWN,
"the IPsec table answered in a layout this script could not "
"parse, so the tunnel count is undetermined", detail)
if tunnels == 0 and ike_up == 0:
return Check("IPsec VPN", NA, "no VPN state on this device", detail)
return Check("IPsec VPN", OK,
f"{tunnels} IPsec tunnel(s) up, {ike_up} IKE SA(s) up", detail)
def check_license(raw: dict) -> Check:
text, err = raw["license"]
if text is None:
return Check("Licences", UNKNOWN, "`show system license` did not answer",
[err])
headers = ["Feature", "Used", "Installed", "Needed", "Expiry"]
rows, needed, expiring, sev = [], [], [], []
today = datetime.now(timezone.utc).date()
# The Expiry column is OPTIONAL in this pattern, and that is the whole
# point of it. Juniper's published examples of `show system license` always
# print `permanent` or a countdown there, and not one of them shows a
# feature with `installed 0` - so what that column contains for a feature
# in use WITHOUT a licence is undocumented. It may be a dash; it may be
# empty.
#
# The risk is not symmetric, which is what decides it. Tolerating an empty
# column costs nothing when the box prints a dash. Requiring the column
# and getting an empty one drops the ENTIRE ROW, `needed` is never seen,
# and a feature running unlicensed is reported as OK - a clean pass over
# something nobody looked at, which is worse than undetermined and is the
# one lie this script must not tell. Somebody closes a licence audit with
# that report.
#
# So the house rule applies: in front of a format that could not be
# verified, degrade the FIELD, never discard the row. An empty Expiry is
# treated exactly like a dash - not a date, so never an expiry finding -
# and the row is still counted and still graded on `needed`.
for line in text.splitlines():
m = re.match(r"^\s{1,4}(\S+)\s+(\d+)\s+(\d+)\s+(\d+)\s*(.*?)\s*$", line)
if not m:
continue
feat, used, inst, need, exp = m.groups()
exp = exp or DASH
rows.append([feat, used, inst, need, exp])
row_sev = OK
if int(need) > 0:
needed.append(feat)
row_sev = WARN
d = re.match(r"(\d{4})-(\d{2})-(\d{2})", exp)
if d:
try:
left = (datetime(int(d.group(1)), int(d.group(2)), int(d.group(3)),
tzinfo=timezone.utc).date() - today).days
except ValueError:
left = None
if left is not None and left <= 30:
expiring.append(f"{feat} expires in {left} day(s)")
row_sev = WARN
sev.append(row_sev)
if not rows:
return Check("Licences", UNKNOWN,
"the command answered but no licence row could be parsed",
["The column layout differs between releases. That makes "
"this undetermined, not a pass."], headers, rows)
if needed:
return Check("Licences", WARN,
f"{len(needed)} feature(s) in use without a licence: "
f"{', '.join(needed[:5])}", expiring, headers, rows, sev)
if expiring:
return Check("Licences", WARN,
f"{len(expiring)} licence(s) expiring within 30 days",
expiring, headers, rows, sev)
return Check("Licences", OK,
f"{len(rows)} licence row(s), none short and none expiring "
f"within 30 days", [], headers, rows, sev)
def check_commit(raw: dict) -> Check:
text, err = raw["commits"]
if text is None:
return Check("Last configuration commit", UNKNOWN,
"`show system commit` did not answer", [err])
m = re.search(r"^\s*0\s+(\d{4}-\d{2}-\d{2} [\d:]+ \S+)\s+by (\S+) via (\S+)",
text, re.M)
if not m:
return Check("Last configuration commit", UNKNOWN,
"the commit history answered in an unexpected layout", [])
return Check("Last configuration commit", INFO,
f"{m.group(1)} by {m.group(2)} via {m.group(3)}",
["Reported for context and never graded: a recent commit is not "
"a fault. It is here because a change made an hour before the "
"symptom is the first thing worth knowing."])
# ---------------------------------------------------------------------------
# PIC status, environment, LACP and LLDP
# ---------------------------------------------------------------------------
def check_pic_status(raw: dict) -> Check:
"""
Every PIC and whether it is Online, from `show chassis fpc pic-status`.
A PIC that is not Online is a WARNING and deliberately not a CRITICAL. A
PIC can be offline because somebody took it offline, and calling that
"verified as bad" is exactly the conflation this script refuses to make
elsewhere. It is still worth saying out loud, because an offline PIC is
capacity nobody is using.
"""
name = "PIC status"
headers = ["Slot", "PIC", "State", "Type"]
text, err = raw.get("pics", (None, "not collected"))
if text is None:
if err.startswith("command rejected"):
return Check(name, NA,
"this platform does not report PIC status", [err],
headers, [])
return Check(name, UNKNOWN,
"`show chassis fpc pic-status` did not answer", [err],
headers, [])
slot, rows, bad, sev = DASH, [], [], []
for line in text.splitlines():
m = re.match(r"^\s*Slot\s+(\d+)\s+(\S+)\s*(.*)$", line, re.I)
if m:
slot = m.group(1)
continue
m = re.match(r"^\s*PIC\s+(\d+)\s+(\S+)\s*(.*)$", line, re.I)
if not m:
continue
state = m.group(2)
rows.append([slot, m.group(1), state, m.group(3).strip() or DASH])
sev.append(OK if state.lower() == "online" else WARN)
if state.lower() != "online":
bad.append(f"slot {slot} PIC {m.group(1)} is {state}")
detail = ["A PIC that is not Online is reported as a warning, not as a "
"fault: it can legitimately have been taken offline, and "
"\"could not be ruled benign\" is not the same as \"verified as "
"bad\"."]
if not rows:
return Check(name, NA,
"no PIC was reported, so there is nothing to grade on "
"this platform", detail, headers, [])
if bad:
return Check(name, WARN, "; ".join(bad), detail, headers, rows, sev)
return Check(name, OK, f"{len(rows)} PIC(s) online", detail, headers, rows,
sev)
# `show chassis environment` is a Class / Item / Status / Measurement table and
# THE SHAPE VARIES BY PLATFORM. A branch SRX names each fan on its own
# ("SRX345 Chassis fan 1 OK Spinning at normal speed"); a high-end chassis
# reports trays ("Upper Fan Tray", "Bottom Fan Tray"); on vSRX the command
# returns next to nothing. So the parser reads by CLASS BLOCK and guards on the
# status VOCABULARY rather than on column position, and when there is no Fans
# block at all the fan check is not applicable - which is not the same as a
# failed fan, and is never reported as one.
ENV_CLASSES = ("power", "temp", "fans", "misc")
ENV_STATUS = ("OK", "Testing", "Failed", "Absent", "Check", "Normal")
ENV_FAIL = {"failed", "absent"}
ENV_RULE = re.compile(r"^-+$")
def parse_environment(text: str) -> dict:
out = {c: [] for c in ENV_CLASSES}
cls = None
for raw_line in text.splitlines():
line = raw_line.rstrip()
if not line.strip():
continue
if re.match(r"^\s*Class\s+Item\b", line, re.I):
continue
if re.match(r"^\s*node[01]\s*:\s*$", line, re.I):
continue
if ENV_RULE.match(line.strip()):
continue
lead = re.match(r"^(\S+)\s*(.*)$", line)
if lead and lead.group(1).lower() in ENV_CLASSES:
cls = lead.group(1).lower()
entry = lead.group(2).strip()
elif line[0] in " \t" and cls:
entry = line.strip()
else:
continue
if not entry:
continue
m = re.match(r"^(.*?)\s\s+(" + "|".join(ENV_STATUS) + r")\b\s*(.*)$",
entry, re.I)
if m:
out[cls].append({"item": m.group(1).strip(),
"status": m.group(2),
"measurement": m.group(3).strip()})
continue
m = re.match(r"^(.*?)\s\s+(\S.*)$", entry)
if m:
# A row whose status word is not one Junos documents. It goes in
# with the status as a dash, which is what makes it undetermined
# instead of healthy.
out[cls].append({"item": m.group(1).strip(), "status": DASH,
"measurement": m.group(2).strip()})
else:
out[cls].append({"item": entry, "status": DASH, "measurement": ""})
return out
def check_environment(raw: dict) -> Check:
"""
Fans first, temperatures as context.
A fan reported Failed or Absent is CRITICAL: the chassis is on a thermal
clock from that moment. A fan in Testing, or with a status word this script
does not recognise, is UNDETERMINED - it is neither verified good nor
verified bad, and reporting it either way would be a lie.
TEMPERATURES ARE NOT GRADED. There is no threshold here because Juniper
publishes none per model, and an over-temperature condition raises a
chassis alarm, which this report grades in its own check. A number in
degrees invented in this file would read like a vendor limit without
being one.
"""
name = "Fans and environment"
headers = ["Class", "Item", "Status", "Measurement"]
text, err = raw.get("env", (None, "not collected"))
if text is None:
if err.startswith("command rejected"):
return Check(name, NA,
"this platform does not implement `show chassis "
"environment`", [err], headers, [])
return Check(name, UNKNOWN,
"`show chassis environment` did not answer", [err],
headers, [])
parsed = parse_environment(text)
rows = [[c.title(), e["item"], e["status"], e["measurement"] or DASH]
for c in ("fans", "temp", "power", "misc") for e in parsed[c]]
# One severity per row, from the same two tests that decide the verdict
# below: a fan in ENV_FAIL is critical, and a fan that is neither in
# ENV_FAIL nor OK is undetermined. Temperature, power and the rest are
# not graded by this check, so their rows carry INFO and are never
# marked - the same reason the verdict ignores them.
sev = [(CRIT if e["status"].lower() in ENV_FAIL
else OK if e["status"].lower() == "ok" else UNKNOWN)
if c == "fans" else INFO
for c in ("fans", "temp", "power", "misc") for e in parsed[c]]
detail = ["Temperatures are reported and NOT graded. Juniper publishes no "
"safe range per SRX model, and an over-temperature condition "
"raises a chassis alarm, which this report grades separately. A "
"threshold in degrees invented here would read like a vendor "
"limit, and it would not be one."]
fans = parsed["fans"]
if not fans:
return Check(name, NA,
"this platform reported no fan block, so there is nothing "
"to grade",
detail + ["The output of this command varies by platform: "
"a branch SRX names each fan, a high-end "
"chassis reports fan trays, and vSRX reports "
"next to nothing. No fan block means the check "
"does not apply - it is not a failed fan."],
headers, rows, sev)
failed = [f for f in fans if f["status"].lower() in ENV_FAIL]
undet = [f for f in fans
if f["status"].lower() not in ENV_FAIL
and f["status"].lower() != "ok"]
if failed:
return Check(name, CRIT,
"; ".join(f"{f['item']} is {f['status']}"
for f in failed)
+ " - a failed or absent fan puts the chassis on a "
"thermal clock", detail, headers, rows, sev)
if undet:
return Check(name, UNKNOWN,
"; ".join(f"{f['item']} reports "
f"{f['status'] if f['status'] != DASH else 'a status this script does not recognise'}"
for f in undet)
+ " - neither verified good nor verified bad",
detail, headers, rows, sev)
return Check(name, OK,
f"{len(fans)} fan(s) or fan tray(s) reporting OK",
detail, headers, rows, sev)
# Mux states, longest first on purpose: "Collecting distributing" has to be
# tested before "Distributing" or the longer one is never seen. Only the
# longest of them is forwarding.
LACP_MUX = ("Collecting distributing", "Collecting", "Distributing",
"Attached", "Waiting", "Detached")
def parse_lacp(text: str) -> dict:
"""
One entry per aggregate: the Actor/Partner flag rows and the protocol row
for every member link.
The two fields that decide the verdict are the partner's `Def` column and
the Mux state. Juniper documents `Defaulted` as meaning no LACP PDU
arrived before the Current-state timer expired for the second time - in
plain terms, THE OTHER END IS NOT SPEAKING LACP, or is not configured for
it. The Mux walks Detached / Waiting / Attached / Collecting distributing,
and only the last of those passes traffic.
"""
out: dict = {}
ae = None
for line in text.splitlines():
m = re.match(r"^\s*Aggregated interface:\s*(\S+)", line, re.I)
if m:
ae = m.group(1)
out.setdefault(ae, {"flags": [], "mux": []})
continue
if ae is None:
continue
m = re.match(r"^\s*(\S+)\s+(Actor|Partner)\s+(\S+)\s+(\S+)\s+(\S+)\s+"
r"(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s+(\S+)\s*$", line, re.I)
if m:
out[ae]["flags"].append({
"link": m.group(1), "role": m.group(2).title(),
"exp": m.group(3), "def": m.group(4), "dist": m.group(5),
"col": m.group(6), "syn": m.group(7), "aggr": m.group(8),
"timeout": m.group(9), "activity": m.group(10)})
continue
stripped = line.rstrip()
for mux in LACP_MUX:
if stripped.lower().endswith(mux.lower()):
head = stripped[:len(stripped) - len(mux)].split()
if len(head) >= 2:
out[ae]["mux"].append({
"link": head[0], "receive": head[1],
"transmit": " ".join(head[2:]) or DASH, "mux": mux})
break
return out
def check_lacp(raw: dict) -> Check:
"""
A half-formed bundle is the quiet failure. The aggregate stays up on the
members that did come up, so nothing alarms and no interface is down - the
capacity is simply gone, and it stays gone until somebody counts the
members by hand. That is why a defaulted partner or a member outside
Collecting distributing is CRITICAL here rather than a warning.
"""
name = "LACP bundles"
headers = ["Bundle", "Link", "Partner defaulted", "Receive state",
"Mux state"]
text, err = raw.get("lacp", (None, "not collected"))
if text is None:
if err.startswith("command rejected"):
return Check(name, NA,
"no aggregated Ethernet interface is configured, so "
"there is no LACP to check", [err], headers, [])
return Check(name, UNKNOWN, "`show lacp interfaces` did not answer",
[err], headers, [])
parsed = parse_lacp(text)
detail = ["A bundle that came up on some of its members is capacity lost "
"without anything going down, which is why this is graded "
"critical. `Defaulted` on the partner side means no LACP PDU is "
"arriving at all - the other end is not speaking LACP."]
if not parsed:
return Check(name, NA,
"no aggregated Ethernet interface is configured, so there "
"is no LACP to check", detail, headers, [])
rows, bad, sev = [], [], []
for ae, info in sorted(parsed.items()):
mux_by = {m["link"]: m for m in info["mux"]}
partner = {f["link"]: f for f in info["flags"]
if f["role"] == "Partner"}
for link in sorted(set(list(mux_by) + list(partner))):
defaulted = partner[link]["def"] if link in partner else DASH
recv = mux_by[link]["receive"] if link in mux_by else DASH
mux = mux_by[link]["mux"] if link in mux_by else DASH
rows.append([ae, link, defaulted, recv, mux])
if defaulted.lower() == "yes" or recv.lower() == "defaulted":
sev.append(CRIT)
bad.append(f"{ae} member {link}: the partner information is "
f"DEFAULTED - no LACP PDU is arriving, so the other "
f"end is not speaking LACP")
elif mux != DASH and mux.lower() != "collecting distributing":
sev.append(CRIT)
bad.append(f"{ae} member {link}: Mux state is {mux}, not "
f"Collecting distributing - this member is not "
f"forwarding")
else:
sev.append(OK)
if not rows:
return Check(name, UNKNOWN,
"the command answered but no member link could be parsed, "
"so the bundles are undetermined", detail, headers, [])
if bad:
return Check(name, CRIT, "; ".join(bad), detail, headers, rows, sev)
return Check(name, OK,
f"{len(parsed)} bundle(s), {len(rows)} member(s), all "
f"collecting and distributing", detail, headers, rows, sev)
LLDP_PORT = re.compile(r"^(ge|xe|et|fe|ae|reth|irb|em|fxp)[-\d]", re.I)
def check_lldp(raw: dict) -> Check:
"""
The neighbour table, INFO and never graded.
It is collected because it is the fastest way to confirm what is actually
on the other end of a port - and the fastest way to notice that what you
expected is not there. If LLDP is not configured the command answers with
an error or with nothing, and the check is not applicable.
"""
name = "LLDP neighbours"
headers = ["Local interface", "Parent", "Chassis ID", "Port info",
"System name"]
text, err = raw.get("lldp", (None, "not collected"))
if text is None:
if err.startswith("command rejected"):
return Check(name, NA, "LLDP is not configured on this device",
[err], headers, [])
return Check(name, UNKNOWN, "`show lldp neighbors` did not answer",
[err], headers, [])
rows = []
for line in text.splitlines():
if re.match(r"^\s*Local\s+\w", line, re.I):
continue
parts = line.split()
if len(parts) < 3 or not LLDP_PORT.match(parts[0]):
continue
while len(parts) < 5:
parts.append(DASH)
rows.append([parts[0], parts[1], parts[2], parts[3],
" ".join(parts[4:])])
detail = ["Reported for context and never graded: a neighbour table cannot "
"be a finding. It is here because it is the fastest way to "
"confirm what is on the other end of a port."]
if not rows:
return Check(name, NA,
"no LLDP neighbour was reported, so there is nothing to "
"show", detail, headers, [])
return Check(name, INFO, f"{len(rows)} LLDP neighbour(s)", detail,
headers, rows)
# ---------------------------------------------------------------------------
# High-end data plane checks
# ---------------------------------------------------------------------------
def check_fpc_state(raw: dict) -> Check:
"""
The state of every FPC slot. This one is not high-end only: any SRX with
FPCs answers it.
The finding that matters is `Present` with a hardware note. It means the
chassis can see the card and is NOT using it - wrong slot, or not
supported by this Junos - and an operator can spend months believing that
capacity is in service. `Dead` is the same shape of problem with a
different cause. `Empty` and `Offline` are not findings: a slot can
legitimately be empty, and offline is a decision somebody took.
"""
name = "FPC / SPC slots"
headers = ["Slot", "State", "Means", "Temp C", "CPU %", "Memory %",
"CPU DRAM MB", "Hardware note"]
text, err = raw.get("fpc", (None, "not collected"))
if text is None:
# A rejected command means the platform has no FPC slots - that is an
# answer. A timeout means nobody looked, which is not.
if err.startswith("command rejected"):
return Check(name, NA,
"this platform has no FPC slots to report", [err],
headers, [])
return Check(name, UNKNOWN,
"`show chassis fpc detail` did not answer", [err],
headers, [])
rows = parse_chassis_fpc(text)
if not rows:
return Check(name, UNKNOWN,
"the command answered in a layout this script could not "
"parse, so the slot states are undetermined",
["Juniper's documentation for this command leaves the "
"sample output blank, so the column spacing here is "
"unverified. An unrecognised shape is reported as "
"undetermined and never as an empty chassis."],
headers, [])
table = [[r["slot"], r["state"], FPC_STATES[r["state"]],
show(r.get("temp")), show(r.get("cpu")), show(r.get("mem")),
show(r.get("dram")), r["note"] or DASH] for r in rows]
crit = [r for r in rows
if r["state"] == "dead" or (r["state"] == "present" and r["note"])]
warn = [r for r in rows if r not in crit
and r["state"] not in FPC_BENIGN and r["state"] != "online"]
online = [r for r in rows if r["state"] == "online"]
sev = [CRIT if r in crit else WARN if r in warn else OK for r in rows]
detail = [f"{len(online)} slot(s) online out of {len(rows)} reported.",
"Temperature, CPU, memory and DRAM come from `detail` and are "
"shown without a verdict: Junos publishes no threshold for them. "
"A dash means the field was not in the output, never that it "
"was zero."]
if crit:
return Check(name, CRIT,
"; ".join(f"slot {r['slot']} is {r['state']}"
+ (f" ({r['note']})" if r["note"] else "")
+ " - the chassis is not using this card"
for r in crit), detail, headers, table, sev)
if warn:
return Check(name, WARN,
"; ".join(f"slot {r['slot']} is {r['state']} "
f"({FPC_STATES[r['state']]})" for r in warn),
detail, headers, table, sev)
if not online:
return Check(name, WARN,
"no slot reports Online - every slot is empty or offline",
detail, headers, table, sev)
return Check(name, OK, f"{len(online)} slot(s) online, none dead, dormant, "
f"in diagnostics or unsupported", detail, headers,
table, sev)
SPU_FIELDS = [
("FPC", "fpc"),
("PIC", "pic"),
("CPU Utilization (%)", "cpu"),
("Memory Utilization (%)", "mem"),
("Current flow session", "cur_flow"),
("Max flow session", "max_flow"),
("SPU current cp session", "cur_cp"),
("SPU max cp session", "max_cp"),
]
NOT_HIGH_END = ("this check runs on SRX5400/5600/5800 only, where the data "
"plane runs on separate services cards")
# Said plainly rather than buried: these two checks have never been run
# against the output of a real high-end chassis. They are written to the field
# names Juniper publishes, and they return UNKNOWN when the shape does not
# fit.
SPU_UNVERIFIED = ("These SPU checks have never been seen against real "
"SRX5400/5600/5800 output. They are written to the field "
"names Juniper publishes, and they report undetermined "
"rather than a number when the layout does not fit.")
def parse_spu_fpc(text: str) -> list[dict]:
"""
Pull the SPU rows out of `show security monitoring fpc <n>`.
Two strategies, because the published documentation gives the field names
and not an example of the output.
1. A key-value shape: the label, then a number ON THE SAME LINE. The
whitespace class deliberately excludes the newline - allowing it would
let a label with nothing after it swallow the first number of the next
line, which is how a parser invents a value.
2. A header line naming the columns, then rows of numbers underneath. The
columns are mapped by the ORDER they appear in the header, and a row
only counts when it carries exactly as many numbers as there are known
columns. A row that does not line up is skipped rather than guessed.
Returns [] when neither fits, and the caller turns that into UNKNOWN.
"""
rows: list[dict] = []
kv = {}
for label, key in SPU_FIELDS:
m = re.search(re.escape(label) + r"[^\S\n]*[:=]?[^\S\n]*(\d+)", text)
if m:
kv[key] = int(m.group(1))
if "mem" in kv:
return [kv]
lines = text.splitlines()
for i, line in enumerate(lines):
if "Memory Utilization" not in line:
continue
present = [(line.index(lbl), key) for lbl, key in SPU_FIELDS
if lbl in line]
order = [key for _, key in sorted(present)]
if "mem" not in order:
continue
for raw_row in lines[i + 1:]:
nums = re.findall(r"\d+", raw_row)
if len(nums) != len(order):
continue
rows.append({k: int(v) for k, v in zip(order, nums)})
if rows:
break
return rows
def check_spu_fpc(raw: dict, high_end: bool) -> Check:
name = "SPU cards (high-end data plane)"
headers = ["FPC", "PIC", "CPU %", "Memory %", "Flow sessions",
"CP sessions"]
if not high_end:
return Check(name, NA, NOT_HIGH_END, [], headers, [])
keys = sorted(k for k in raw if k.startswith("spu_fpc_"))
if not keys:
# No slot was polled because `show chassis fpc` could not be read.
# Falling back to slot 0 would be reading a slot this script has no
# evidence exists, and presenting it as the state of the chassis.
return Check(name, UNKNOWN,
"the FPC slots could not be enumerated, so no SPU card "
"was read - see the FPC / SPC slots check above, or name "
"the slots yourself with --spu-fpc",
[SPU_UNVERIFIED], headers, [])
rows, problems, missing, sev = [], [], [], []
status = OK
for key in keys:
slot = key.split("_")[-1]
text, err = raw[key]
if text is None:
missing.append(f"FPC {slot}: {err}")
continue
parsed = parse_spu_fpc(text)
if not parsed:
missing.append(f"FPC {slot}: the command answered in a layout this "
f"script could not parse")
continue
for r in parsed:
flow = DASH
row_sev = OK
if r.get("max_flow"):
pct = round(r.get("cur_flow", 0) * 100 / r["max_flow"], 1)
flow = f"{r.get('cur_flow')}/{r['max_flow']} ({pct}%)"
if pct >= 90:
status = CRIT
row_sev = CRIT
problems.append(f"FPC {slot} PIC {show(r.get('pic'))}: SPU "
f"flow table {pct}% full")
elif pct >= 80 and status != CRIT:
status = WARN
row_sev = WARN
problems.append(f"FPC {slot} PIC {show(r.get('pic'))}: SPU "
f"flow table {pct}% full")
cp = DASH
if r.get("max_cp"):
cp = f"{r.get('cur_cp')}/{r['max_cp']}"
mem = r.get("mem")
if mem is not None and mem > 80:
if status != CRIT:
status = WARN
if row_sev != CRIT:
row_sev = WARN
problems.append(
f"FPC {slot} PIC {show(r.get('pic'))}: memory {mem}%. "
f"Juniper documents this one: \"If this number exceeds 80 "
f"percent, there might be a software problem (memory "
f"leak)\".")
rows.append([show(r.get("fpc", slot)), show(r.get("pic")),
show(r.get("cpu")), show(mem), flow, cp])
sev.append(row_sev)
detail = list(missing)
detail.append("The 80% memory threshold is Juniper's own, quoted above. "
"The 80/90% flow-table thresholds are this script's "
"judgement, not a vendor figure.")
detail.append("CPU utilisation is shown and NOT graded: Juniper publishes "
"no threshold for it, and inventing one here would dress up "
"a guess as a finding.")
detail.append(SPU_UNVERIFIED)
if not rows:
return Check(name, UNKNOWN,
"no SPU slot could be read, so the data plane is "
"undetermined", detail, headers, rows)
if missing and status == OK:
return Check(name, UNKNOWN,
f"{len(rows)} SPU row(s) read and healthy, but "
f"{len(missing)} slot(s) could not be read",
detail, headers, rows, sev)
if status == OK:
return Check(name, OK, f"{len(rows)} SPU row(s), memory under 80% and "
f"flow tables under 80%", detail, headers, rows,
sev)
return Check(name, status, "; ".join(problems), detail, headers, rows, sev)
def check_spu_performance(raw: dict, high_end: bool) -> Check:
name = "SPU utilisation over the last minute"
if not high_end:
return Check(name, NA, NOT_HIGH_END)
text, err = raw.get("spu_perf", (None, "not collected"))
if text is None:
return Check(name, UNKNOWN, f"`{SPU_PERF_COMMAND}` did not answer",
[err])
# The command reports utilisation per second for the last 60 seconds. The
# samples are the only thing worth reading, and a run of at least ten
# percentages is the conservative signature of that list. Anything else is
# a layout this script does not recognise, and it says so.
samples = [int(v) for v in re.findall(r"\b(\d{1,3})\b", text)
if int(v) <= 100]
if len(samples) < 10:
return Check(name, UNKNOWN,
"the command answered in a layout this script could not "
"parse, so SPU utilisation is undetermined",
[SPU_UNVERIFIED,
"Juniper documents the field names for this command but "
"not an example of its output. Run it by hand and read "
"it yourself rather than trusting a parser that did not "
"understand it."])
return Check(name, INFO,
f"peak {max(samples)}% over the last minute",
["Reported and never graded. Juniper publishes no threshold "
"for SPU CPU utilisation, so a verdict here would be this "
"script's opinion presented as a fact.",
"The peak is the only figure given. An average is not, and "
"that is deliberate: the layout of this command is not "
"documented, so the per-second samples cannot be told apart "
"from a slot or thread number on the same line with any "
"certainty. A mean computed over the wrong set of integers "
"would look precise and be wrong.",
"A peak of 100% is worth looking at by hand - run the "
"command yourself and read the whole minute.",
SPU_UNVERIFIED])
# ---------------------------------------------------------------------------
# Chassis cluster
#
# ONE NODE IS ENOUGH TO SEE THE CLUSTER. `show chassis cluster status` and
# `show chassis cluster information` return the state of BOTH nodes from
# either one of them, because the cluster tables are replicated over the
# control link. Every cluster check in this file therefore runs identically in
# both versions of the script, including the SecureCRT one, which only ever
# has one session.
#
# What a single node sees when there IS a split-brain is the SIGNATURE of it:
# the peer shows as `lost` and the control link is down. That is already
# CRITICAL from one node and it stays CRITICAL.
#
# What polling both nodes adds is one extra fact, and only one: the
# CONFIRMATION that the two nodes are both forwarding at the same time for the
# same redundancy group. The signature and the confirmation are different
# facts, and this script reports them as different facts.
# ---------------------------------------------------------------------------
CLUSTER_BAD = {"lost", "unknown", "unavailable"}
CLUSTER_DISABLED = {"ineligible", "disabled"}
CLUSTER_HOLD = {"hold", "secondary-hold", "primary-hold"}
def parse_cluster_status(text: str) -> dict:
"""Returns {cluster_id, groups: {rg: {failover, nodes: [[node, prio, status,
preempt, manual, monitor]]}}}"""
out = {"cluster_id": grab(r"^Cluster ID:\s*(\d+)", text) or "", "groups": {}}
rg = None
for line in text.splitlines():
m = re.match(r"^Redundancy group:\s*(\d+)\s*,\s*Failover count:\s*(\d+)",
line.strip(), re.I)
if m:
rg = m.group(1)
out["groups"][rg] = {"failover": m.group(2), "nodes": []}
continue
n = re.match(r"^(node[01])\s+(\d+)\s+(\S+)\s+(\S+)\s+(\S+)\s*(.*)$",
line.strip(), re.I)
if n and rg is not None:
out["groups"][rg]["nodes"].append([
n.group(1), n.group(2), n.group(3).lower(), n.group(4),
n.group(5), (n.group(6) or "None").strip() or "None",
])
return out
def check_cluster_status(res: DeviceResult, raw: dict) -> Check:
text, err = raw.get("cl_status", (None, "not collected"))
headers = ["RG", "Node", "Priority", "Status", "Preempt", "Manual",
"Monitor failures"]
if text is None:
return Check("Chassis cluster status", UNKNOWN,
"`show chassis cluster status` did not answer, so whether "
"this box is clustered is undetermined", [err], headers, [])
if NOT_CLUSTERED.search(text):
res.role = "standalone"
return Check("Chassis cluster status", NA,
"chassis cluster is not enabled on this device",
["Every cluster check below is marked N/A for the same "
"reason."], headers, [])
parsed = parse_cluster_status(text)
res.cluster_id = parsed["cluster_id"]
rows, problems, status, sev = [], [], OK, []
if not parsed["groups"]:
return Check("Chassis cluster status", UNKNOWN,
"the command answered but no redundancy group could be "
"parsed", [], headers, [])
roles = []
for rg, g in sorted(parsed["groups"].items(), key=lambda kv: int(kv[0])):
prim = [n for n in g["nodes"] if n[2] == "primary"]
res.rg_primaries[rg] = sorted(n[0] for n in prim)
res.rg_nodes[rg] = [list(n) for n in g["nodes"]]
for n in g["nodes"]:
rows.append([rg] + n)
row_sev = OK
if n[2] in CLUSTER_BAD:
status = CRIT
row_sev = CRIT
problems.append(f"RG{rg}: {n[0]} is {n[2]} - the peer is not "
f"being heard, which is how split-brain starts")
elif n[2] in CLUSTER_DISABLED:
status = CRIT
row_sev = CRIT
problems.append(f"RG{rg}: {n[0]} is {n[2]} and cannot take over")
elif n[2] in CLUSTER_HOLD and status != CRIT:
status = WARN
row_sev = WARN
problems.append(f"RG{rg}: {n[0]} is {n[2]}, a transient state "
f"that should not persist")
if n[5].lower() not in ("none", "n/a", ""):
status = CRIT
row_sev = CRIT
problems.append(f"RG{rg}: {n[0]} reports monitor failures "
f"{n[5]}")
sev.append(row_sev)
if len(prim) > 1:
status = CRIT
problems.append(f"RG{rg}: {len(prim)} nodes claim primary at once - "
f"this is split-brain")
elif not prim:
status = CRIT
problems.append(f"RG{rg}: no node is primary")
else:
roles.append(f"RG{rg} primary on {prim[0][0]}")
sec = [n for n in g["nodes"] if n[2] == "secondary"]
if (sec and int(sec[0][1]) > int(prim[0][1])
and prim[0][3].lower() == "yes"):
if status != CRIT:
status = WARN
problems.append(
f"RG{rg}: the secondary has the higher priority "
f"({sec[0][1]} against {prim[0][1]}) and preempt is on, so a "
f"failover is pending the next time it is eligible")
# The failover count goes into the device summary whatever its value,
# including zero: here it IS a counter the device reported, so zero is
# a real zero and not a field that could not be read.
res.rg_failovers[rg] = g["failover"]
if int(g["failover"]) > 0:
problems.append(f"RG{rg}: {g['failover']} failover(s) counted since "
f"boot - cumulative, so not a fault on its own")
res.role = f"cluster {res.cluster_id}: " + ", ".join(roles) if roles else \
f"cluster {res.cluster_id}"
if status == OK:
return Check("Chassis cluster status", OK,
"; ".join(roles) or "cluster healthy", problems, headers,
rows, sev)
return Check("Chassis cluster status", status,
"; ".join(p for p in problems if "cumulative" not in p)
or "cluster state is not healthy", problems, headers, rows,
sev)
def parse_cluster_information(text: str) -> dict:
"""
`show chassis cluster information` gives two things nothing else does: the
HISTORY of redundancy group state changes (Time / From / To / Reason) and
the chassis cluster LED.
Version note, verified in Juniper's documentation: the command exists from
Junos 12.1X47-D10, so it is available across the whole range this script
covers. The `Redundancy mode` field (active-active / active-backup) was
only ADDED IN 24.4R1, so nothing here depends on it being present - it is
read when it is there and prints as a dash when it is not.
"""
out = {"groups": {}, "leds": [],
"mode": grab(r"^\s*Redundancy mode:\s*(\S+)", text)}
node, rg = "", None
for line in text.splitlines():
m = re.match(r"^\s*(node[01])\s*:\s*$", line, re.I)
if m:
node, rg = m.group(1).lower(), None
continue
m = re.match(r"^\s*Redundancy Group\s+(\d+)\s*,\s*Current State:\s*"
r"([^,]+?)\s*,\s*Weight:\s*(\d+)", line, re.I)
if m:
rg = (node, m.group(1))
out["groups"][rg] = {"state": m.group(2).strip().lower(),
"weight": m.group(3), "events": []}
continue
m = re.match(r"^\s*((?:\w{3}\s+\d{1,2}|\d{4}-\d{2}-\d{2})\s+[\d:]+)\s+"
r"(\S+)\s+(\S+)\s+(.+?)\s*$", line)
if m and rg is not None and m.group(2).lower() != "from":
out["groups"][rg]["events"].append(
[m.group(1), m.group(2), m.group(3), m.group(4)])
continue
m = re.match(r"^\s*Current LED color:\s*(\S+)", line, re.I)
if m:
out["leds"].append({"node": node or DASH, "color": m.group(1),
"reason": ""})
continue
m = re.match(r"^\s*Last LED change reason:\s*(.+?)\s*$", line, re.I)
if m and out["leds"]:
out["leds"][-1]["reason"] = m.group(1)
return out
def check_cluster_information(res: DeviceResult, raw: dict) -> Check:
"""
The redundancy group history and the chassis cluster LED.
The LED is graded WARNING and NOT CRITICAL, deliberately. The LED is a
SUMMARY of an underlying failure, and every underlying failure already has
its own check here: a monitored interface that went down, a redundancy
group whose weight dropped, a peer that went lost, a control link that
failed. Those are the CRITICALs. Grading the LED critical as well would
count the same single failure twice and inflate the report, which makes
the network read worse than it is - and a count nobody trusts is a count
nobody acts on.
The "Last LED change reason" is quoted verbatim, because it is the
device's own words for why it went amber and it is usually the shortest
description of the fault anyone will get.
"""
name = "Cluster failover history and LED"
headers = ["Node", "RG", "When", "From", "To", "Reason"]
text, err = raw.get("cl_info", (None, "not collected"))
if text is None:
return Check(name, UNKNOWN,
"`show chassis cluster information` did not answer, so "
"the cluster LED and the failover history are "
"undetermined", [err], headers, [])
if NOT_CLUSTERED.search(text):
return Check(name, NA,
"chassis cluster is not enabled on this device", [],
headers, [])
parsed = parse_cluster_information(text)
ordered = sorted(parsed["groups"].items(),
key=lambda kv: (kv[0][0], int(kv[0][1])))
rows = [[(n or DASH), rg] + ev for (n, rg), g in ordered
for ev in g["events"]]
detail = [f"Redundancy mode: {show(parsed['mode'])}",
"`Redundancy mode` is only printed from Junos 24.4R1. A dash "
"there means this release does not report the field, not that "
"the cluster does not have a mode."]
for (n, rg), g in ordered:
prefix = f"{n} " if n else ""
detail.append(f"{prefix}RG{rg}: current state {g['state']}, weight "
f"{g['weight']}, {len(g['events'])} state change(s) in "
f"the history")
for led in parsed["leds"]:
detail.append(f"{led['node']} LED: {led['color']} - last LED change "
f"reason: {led['reason'] or DASH}")
detail.append("The LED is graded as a warning and not as a fault on "
"purpose: it summarises a failure that already has its own "
"check above, and grading it critical as well would count "
"the same failure twice.")
if not parsed["leds"] and not parsed["groups"]:
return Check(name, UNKNOWN,
"the command answered in a layout this script could not "
"parse, so the LED and the history are undetermined",
detail, headers, rows)
amber = [l for l in parsed["leds"] if l["color"].lower() != "green"]
if amber:
return Check(name, WARN,
"; ".join(f"{l['node']} cluster LED is {l['color']} - "
f"last LED change reason: "
f"\"{l['reason'] or DASH}\"" for l in amber),
detail, headers, rows)
if not parsed["leds"]:
return Check(name, UNKNOWN,
"the redundancy group history was read but no LED colour "
"was reported, so the LED is undetermined", detail,
headers, rows)
history = f", {len(rows)} state change(s) in the history" if rows else ""
return Check(name, OK,
f"cluster LED green on every node reported{history}",
detail, headers, rows)
def check_cluster_links(raw: dict) -> Check:
text, err = raw.get("cl_ifaces", (None, "not collected"))
headers = ["Element", "Name", "Child / group", "Status"]
if text is None:
return Check("Cluster control and fabric links", UNKNOWN,
"`show chassis cluster interfaces` did not answer", [err],
headers, [])
if NOT_CLUSTERED.search(text):
return Check("Cluster control and fabric links", NA,
"chassis cluster is not enabled on this device", [],
headers, [])
control = grab(r"Control link status:\s*(\S+)", text)
fabric = grab(r"Fabric link status:\s*(\S+)", text)
rows, problems, status, sev = [], [], OK, []
rows.append(["Control link", "-", "-", show(control)])
rows.append(["Fabric link", "-", "-", show(fabric)])
# A link whose status line was not in the output is UNDETERMINED and not
# healthy - the same distinction the verdict below makes when neither of
# them could be parsed.
for link in (control, fabric):
sev.append(CRIT if (link or "").lower() == "down"
else UNKNOWN if link is None else OK)
for m in re.finditer(r"^\s*(\d+)\s+(\S+)\s+(Up|Down)\s+\S+\s+\S+\s*$",
text, re.M | re.I):
rows.append(["Control interface", m.group(2), "-", m.group(3)])
sev.append(CRIT if m.group(3).lower() == "down" else OK)
if m.group(3).lower() == "down":
status = CRIT
problems.append(f"control interface {m.group(2)} is down")
for m in re.finditer(r"^\s*(fab[01])\s+(\S+)\s+(Up|Down)\s*/\s*(Up|Down)",
text, re.M | re.I):
rows.append(["Fabric child", m.group(1), m.group(2),
f"{m.group(3)} / {m.group(4)}"])
sev.append(CRIT if "down" in (m.group(3).lower(), m.group(4).lower())
else OK)
if "down" in (m.group(3).lower(), m.group(4).lower()):
status = CRIT
problems.append(f"{m.group(1)} child {m.group(2)} is "
f"{m.group(3)}/{m.group(4)} - cold sync and session "
f"sync ride this link")
reths = 0
for m in re.finditer(r"^\s*(reth\d+)\s+(Up|Down)\s+(\S+)", text, re.M | re.I):
reths += 1
rows.append(["Redundant ethernet", m.group(1), f"RG {m.group(3)}",
m.group(2)])
sev.append(WARN if (m.group(2).lower() == "down"
or not m.group(3).isdigit()) else OK)
if m.group(2).lower() == "down":
if status != CRIT:
status = WARN
problems.append(f"{m.group(1)} is down - no child link is up in "
f"either node")
if not m.group(3).isdigit():
if status != CRIT:
status = WARN
problems.append(f"{m.group(1)} is not assigned to a redundancy "
f"group, so nothing will fail it over")
if control is None and fabric is None:
return Check("Cluster control and fabric links", UNKNOWN,
"the command answered but neither link status could be "
"parsed", [], headers, rows, sev)
if (control or "").lower() == "down":
status = CRIT
problems.insert(0, "the control link is DOWN - each node will declare "
"itself primary, which is split-brain")
if (fabric or "").lower() == "down":
status = CRIT
problems.insert(0, "the fabric link is DOWN - no session sync and no "
"cold sync between nodes")
if status == OK:
return Check("Cluster control and fabric links", OK,
f"control {show(control)}, fabric {show(fabric)}, "
f"{reths} reth interface(s) up", problems, headers, rows,
sev)
return Check("Cluster control and fabric links", status,
"; ".join(problems), problems, headers, rows, sev)
def check_cluster_statistics(raw: dict) -> Check:
text, err = raw.get("cl_stats", (None, "not collected"))
if text is None:
return Check("Cluster heartbeat and fabric probes", UNKNOWN,
"`show chassis cluster statistics` did not answer", [err])
if NOT_CLUSTERED.search(text):
return Check("Cluster heartbeat and fabric probes", NA,
"chassis cluster is not enabled on this device", [])
sent = grab_int(r"Heartbeat packets sent:\s*(\d+)", text)
recv = grab_int(r"Heartbeat packets received:\s*(\d+)", text)
errs = grab_int(r"Heartbeat packet errors:\s*(\d+)", text)
# Every child link, not just the first: a two-link fabric with one dead
# child still syncs sessions, and reading only link 0 would miss it.
children = []
for blk in re.split(r"^\s*Child link\s+\d+", text, flags=re.M)[1:]:
children.append((grab_int(r"Probes sent:\s*(\d+)", blk[:300]),
grab_int(r"Probes received:\s*(\d+)", blk[:300])))
probes_sent = sum(s for s, _ in children if s) if children else None
probes_recv = sum(r for _, r in children if r) if children else None
dead = [i for i, (s, r) in enumerate(children) if s and not r]
detail = [f"Heartbeat packets sent: {show(sent)}",
f"Heartbeat packets received: {show(recv)}",
f"Heartbeat packet errors: {show(errs)}",
f"Fabric probes sent: {show(probes_sent)}",
f"Fabric probes received: {show(probes_recv)}"]
if sent is None and probes_sent is None:
return Check("Cluster heartbeat and fabric probes", UNKNOWN,
"the command answered but no heartbeat or probe counter "
"could be located", detail)
if errs:
return Check("Cluster heartbeat and fabric probes", WARN,
f"{errs} heartbeat packet error(s) on the control link",
detail)
if dead:
return Check("Cluster heartbeat and fabric probes", CRIT,
f"fabric child link {', '.join(str(i) for i in dead)} is "
f"sending probes and getting none back - that link is not "
f"passing traffic", detail)
if errs is None and probes_recv is None:
return Check("Cluster heartbeat and fabric probes", UNKNOWN,
"neither the error counter nor the probe counter could be "
"read, so this is undetermined", detail)
return Check("Cluster heartbeat and fabric probes", OK,
"no heartbeat errors and fabric probes are returning", detail)
def correlate_split_brain(results: list[DeviceResult]) -> list[str]:
"""
The CONFIRMATION of a split-brain, which is a different fact from its
signature.
A single node already sees the signature - the peer reported as `lost`
while the control link is down - and the two cluster checks above grade
that CRITICAL from one session. This function adds the other half: if two
polled devices belong to the same cluster and each of them names a
DIFFERENT node as primary for the SAME redundancy group, then both nodes
are forwarding for that group at the same time. That is the confirmation,
and it is the only part that needs both nodes.
Nodes are grouped by the `cluster:` label in the inventory when it is
present. Falling back to the Cluster ID would be wrong on its own: the ID
is only locally significant, so two unrelated clusters are very often both
cluster 1, and comparing them would invent a split-brain that does not
exist. When the grouping had to be inferred, the finding says so.
"""
findings = []
groups: dict[tuple[str, bool], list[DeviceResult]] = {}
for r in results:
if r.status != "ok" or not r.cluster_id:
continue
# group_key() is the single definition of the grouping, shared with
# the report, so a finding can never land under a heading that was
# worked out a different way.
groups.setdefault(group_key(r), []).append(r)
for (name, inferred), members in groups.items():
if len(members) < 2:
continue
rgs: set[str] = set()
for m in members:
rgs |= set(m.rg_primaries)
for rg in sorted(rgs, key=lambda x: int(x)):
claims: dict[str, list[str]] = {}
for m in members:
for node in m.rg_primaries.get(rg, []):
claims.setdefault(node, []).append(m.label)
if len(claims) > 1:
caveat = ("" if not inferred else
" These nodes were grouped by their Cluster ID "
"because the inventory gave no `cluster:` label; if "
"they are in fact two separate clusters that both use "
"this ID, ignore this and label them.")
findings.append(
f"{name}, RG{rg}: SPLIT-BRAIN. "
+ "; ".join(f"{', '.join(v)} reports {k} as primary"
for k, v in sorted(claims.items()))
+ ". Two nodes forwarding for the same redundancy group "
"means the control link is not carrying heartbeats."
+ caveat
)
return findings
# ---------------------------------------------------------------------------
# The HTML report
#
# Self-contained on purpose: no CDN, no web font, no <script>. It has to open
# from a laptop with no internet in a meeting room, and it has to print.
# The SecureCRT variant of this script renders the SAME structure from the
# same function, so a report looks identical whichever one produced it.
# ---------------------------------------------------------------------------
CSS = """
:root { --ink:#1b1b1b; --soft:#5b5b5b; --line:#c9c9c9; --panel:#f6f6f4; }
* { box-sizing:border-box; }
body { margin:0; padding:28px 32px 64px; background:#fff; color:var(--ink);
font:14px/1.55 "Segoe UI", Helvetica, Arial, sans-serif; max-width:1040px; }
h1 { font-size:22px; margin:0 0 4px; }
h2 { font-size:17px; margin:34px 0 10px; padding-bottom:5px;
border-bottom:2px solid var(--ink); }
h3 { font-size:14px; margin:18px 0 6px; }
p { margin:6px 0; }
.sub { color:var(--soft); font-size:12.5px; margin:0 0 18px; }
.meta { display:table; width:100%; font-size:12.5px; border-collapse:collapse;
margin:0 0 18px; }
.meta div { display:table-row; }
.meta span { display:table-cell; padding:2px 14px 2px 0; }
.meta span:first-child { color:var(--soft); width:170px; }
.verdict { border:2px solid var(--ink); padding:12px 16px; margin:0 0 20px; }
.verdict strong { display:block; font-size:16px; letter-spacing:.06em;
text-transform:uppercase; margin-bottom:3px; }
.v-critical { border-color:#8c1c13; background:#fbf0ef; }
.v-critical strong { color:#8c1c13; }
.v-warning { border-color:#8a6100; background:#fcf7ea; }
.v-warning strong { color:#8a6100; }
.v-ok { border-color:#1f6b3a; background:#f0f7f2; }
.v-ok strong { color:#1f6b3a; }
.v-unknown { border-color:#4a4a4a; background:var(--panel); }
table { border-collapse:collapse; width:100%; margin:8px 0 14px;
font-size:12.5px; }
th, td { border:1px solid var(--line); padding:5px 8px; text-align:left;
vertical-align:top; }
th { background:var(--panel); font-weight:600; }
td.num { text-align:right; font-variant-numeric:tabular-nums; }
code, .mono { font-family:Consolas, "DejaVu Sans Mono", monospace;
font-size:12px; }
.tag { display:inline-block; min-width:76px; text-align:center; padding:1px 6px;
border:1px solid var(--ink); font-size:11px; letter-spacing:.04em;
text-transform:uppercase; }
.t-critical { border-color:#8c1c13; color:#8c1c13; background:#fbf0ef; }
.t-warning { border-color:#8a6100; color:#8a6100; background:#fcf7ea; }
.t-ok { border-color:#1f6b3a; color:#1f6b3a; background:#f0f7f2; }
.t-unknown { border-color:#4a4a4a; color:#4a4a4a; background:var(--panel); }
.t-info, .t-na { border-color:var(--line); color:var(--soft); }
/* A graded cell is TINTED, not labelled. The pill that used to sit in front
of the text said WARNING next to `1 needed, expires -`, which is the same
fact twice and cost half the column. The tint says it without a word, and
the left bar says it again as SHAPE - so the mark survives greyscale
printing and a reader who does not separate red from amber. */
td.c-critical { background:#fbf0ef; border-left:3px solid #8c1c13; }
td.c-warning { background:#fcf7ea; border-left:3px solid #8a6100; }
td.c-unknown { background:#ebebe8; border-left:3px solid #4a4a4a; }
ul { margin:6px 0 10px 20px; padding:0; }
li { margin:2px 0; }
.device { border:1px solid var(--line); padding:4px 18px 16px; margin:0 0 22px; }
.device h2 { margin-top:16px; }
.note { color:var(--soft); font-size:12px; }
footer { margin-top:34px; border-top:1px solid var(--line); padding-top:10px;
color:var(--soft); font-size:11.5px; }
@media print {
body { padding:0; font-size:11.5pt; max-width:none; }
h2 { page-break-after:avoid; }
.device, table { page-break-inside:avoid; }
.verdict, th, .tag, td[class] { -webkit-print-color-adjust:exact;
print-color-adjust:exact; }
}
"""
# Escaping is done here instead of with the standard library's html module,
# and the reason is the SecureCRT variant. The interpreter embedded in
# SecureCRT ships a REDUCED standard library, and what is in it has changed
# across versions - VanDyke's own forum has `telnetlib` failing inside
# SecureCRT while it imported fine in IDLE on the same machine. `html` is a
# module nothing in a terminal emulator needs, so it is exactly the kind that
# gets left out, and the failure would be an ImportError on line 20 of a
# script whose whole selling point is that it needs no dependency.
# Five replacements are not worth that risk. The order is not optional: the
# ampersand goes first, or the escapes that follow get escaped again.
_ESCAPES = (("&", "&"), ("<", "<"), (">", ">"),
('"', """), ("'", "'"))
def esc(v) -> str:
s = str(v)
for ch, rep in _ESCAPES:
s = s.replace(ch, rep)
return s
def sev_slug(status: str) -> str:
"""`N/A` -> `na`. One normaliser, so the pill and the tinted cell can
never end up keyed on different spellings of the same severity."""
return status.lower().replace("/", "").replace(" ", "")
def tag(status: str) -> str:
return f'<span class="tag t-{sev_slug(status)}">{esc(status)}</span>'
def table(headers: list[str], rows: list[list[str]]) -> str:
if not headers or not rows:
return ""
head = "".join(f"<th>{esc(h)}</th>" for h in headers)
body = "".join(
"<tr>" + "".join(f"<td>{esc(c)}</td>" for c in r) + "</tr>" for r in rows
)
return f"<table><thead><tr>{head}</tr></thead><tbody>{body}</tbody></table>"
def failover_summary(d: DeviceResult) -> str:
"""
Failovers per redundancy group, as the device counted them.
A dash means no redundancy group reported a count - a standalone SRX, or a
cluster status that could not be read. It never means zero.
"""
if not d.rg_failovers:
return DASH
return ", ".join(f"RG{rg} {d.rg_failovers[rg]}"
for rg in sorted(d.rg_failovers, key=int))
def bullets(items: list[str]) -> str:
items = [i for i in items if i]
if not items:
return ""
return "<ul>" + "".join(f"<li>{esc(i)}</li>" for i in items) + "</ul>"
# ---------------------------------------------------------------------------
# How the report is organised, and the one rule behind it
#
# THE UNIT OF THE REPORT IS THE CLUSTER, NOT THE BOX. Half of these checks
# describe the cluster and not a node, so organising the page by device meant
# printing one single fact under each node and calling it two findings. The
# nodes are COLUMNS: one row per field, one row per check, one column per node.
# A standalone SRX is a group of one and renders with one column through the
# same code path - there is no special case for it.
#
# GOLDEN RULE: a value is printed WHERE IT IS DECIDED, and nowhere else.
# The equipment table identifies, the findings table judges, the detail
# explains. If a number helps understand a finding it is quoted INSIDE that
# finding; it is never re-tabulated somewhere else on the page.
#
# Two consequences worth naming, because both were defects:
# * `Chassis hardware` and `Software and uptime` have no detail block of
# their own. Everything they read - model, serial, Junos, uptime, boot
# time, boot reason - lives in the equipment table at the top. They still
# RUN, they still count, and if either comes back undetermined or graded
# it still appears as a finding. What they no longer do is print the same
# four fields a second time.
# * A cluster check is printed PER NODE like every other check, even when
# both nodes say the same thing. It used to collapse into one cell
# spanning the node columns, which saved a line and cost the reader the
# grammar of the table: a column headed node0 that sometimes held node0's
# answer and sometimes held the pair's. A table is read down a column.
# ---------------------------------------------------------------------------
SUMMARY_CHECKS = ("Software and uptime", "Chassis hardware")
SPLIT_BRAIN_CHECK = "Cluster split-brain"
# Rows of the equipment table where a difference between the nodes is a
# FINDING and not a curiosity. Hostname, serial number and uptime differ by
# design; these two do not.
DIFF_MATTERS = {
"Junos": "The nodes are not running the same Junos release. A chassis "
"cluster is only supported on matching releases, and an upgrade "
"left half done is the usual cause.",
"Model": "The nodes are not the same model. A chassis cluster requires "
"two identical models.",
}
def group_key(r: "DeviceResult") -> tuple[str, bool]:
"""
Which group a device belongs to, and whether that had to be inferred.
The inventory `cluster:` label when it is there. The Cluster ID when it is
not - and that is flagged as inferred, because a Cluster ID is only
locally significant, so two unrelated clusters are very often both
cluster 1, and comparing them would invent a split-brain that does not
exist. A box that is not clustered is its own group of one.
"""
if r.cluster_group:
return (r.cluster_group, False)
if r.cluster_id:
return (f"cluster {r.cluster_id}", True)
return (r.label, False)
def cluster_groups(devices: list) -> list:
"""[(name, inferred, members)], members ordered by node then label."""
order, groups = [], {}
for r in devices:
name, inferred = group_key(r)
if name not in groups:
groups[name] = [inferred, []]
order.append(name)
groups[name][1].append(r)
out = []
for name in order:
inferred, members = groups[name]
members.sort(key=lambda d: (d.node or "zzz", d.label))
out.append((name, inferred, members))
return out
def node_head(d) -> str:
"""The column heading for one node."""
return d.node or d.label
def nodes_word(members: list) -> str:
return "cluster / both nodes" if len(members) == 2 else "cluster / all nodes"
def group_title(name: str, members: list, many_groups: bool) -> str:
"""
The heading of a group, written for whoever opens the report.
It used to read `dc - 2 nodes`: the `cluster:` label from the inventory
and a node count. Neither says anything. The label is ours and means
nothing to the reader, and the count is already the number of columns in
every table underneath. What they need in the first line is WHAT THIS
EQUIPMENT IS - so a pair, or any box that reported a Cluster ID, is a
`Cluster SRX`, and a box that is not clustered is a `Standalone SRX`.
The label comes back only when one run carries MORE THAN ONE group, which
is the only time two headings would otherwise be identical.
"""
clustered = len(members) > 1 or any(d.cluster_id for d in members)
kind = "Cluster SRX" if clustered else "Standalone SRX"
return f"{esc(kind)} — {esc(name)}" if many_groups else esc(kind)
def rich_table(headers: list[str], rows: list[list[str]]) -> str:
"""Like table(), but the cells are already HTML and are NOT escaped."""
if not headers or not rows:
return ""
head = "".join(f"<th>{esc(h)}</th>" for h in headers)
body = "".join("<tr>" + "".join(cells) + "</tr>" for cells in rows)
return f"<table><thead><tr>{head}</tr></thead><tbody>{body}</tbody></table>"
def warn_pill(label: str) -> str:
"""The warning pill the report already has in its CSS, reused inline."""
return f' <span class="tag t-warning">{esc(label)}</span>'
# ---------------------------------------------------------------------------
# The equipment table: one row per field, one column per node
# ---------------------------------------------------------------------------
def equipment_table(members: list) -> tuple[str, list[str]]:
fields = [
("Hostname", lambda d: d.hostname),
("Model", lambda d: d.model),
("Serial number", lambda d: d.serial),
("Junos", lambda d: d.version),
("Uptime", lambda d: d.uptime),
("Booted", lambda d: d.booted),
("Last reboot reason", lambda d: d.reboot_reason),
]
# The row that maps a column back to the inventory, and ONLY when it says
# something the hostname row does not. Most of the time the inventory
# label and the hostname are the same string, and printing it twice is
# exactly the repetition this rewrite exists to remove. When they DIFFER
# it is worth a row of its own: either the inventory is out of date or the
# box is being polled by address.
if any(d.label != d.hostname for d in members):
fields.insert(0, ("Polled as", lambda d: d.label))
rows, notes = [], []
for name, get in fields:
values = [get(d) for d in members]
real = [v for v in values if v not in (DASH, "", None)]
differs = (name in DIFF_MATTERS and len(members) > 1
and len(set(real)) > 1)
if differs:
notes.append(DIFF_MATTERS[name])
cells = [f"<td>{esc(name)}</td>"]
for v in values:
mark = warn_pill("differs") if differs else ""
shown = v if v not in ("", None) else DASH
cells.append(f"<td>{esc(shown)}{mark}</td>")
rows.append(cells)
notes += [d.node_note for d in members if d.node_note]
heads = ["Field"] + [node_head(d) for d in members]
return rich_table(heads, rows), notes
# ---------------------------------------------------------------------------
# Redundancy groups: one row per group, one column per node
# ---------------------------------------------------------------------------
def rg_cell(members: list, rg: str, node: str) -> str:
"""
What the cluster reports for one node in one redundancy group.
When the nodes report the same thing there is one answer and it is printed
plainly. When they DISAGREE - which is exactly what a split-brain looks
like from the two sides - every view is printed with the name of the node
that reported it, and the cell is flagged. A node's report of itself comes
first, because it is the authoritative one.
"""
views: dict = {}
for d in members:
for n in d.rg_nodes.get(rg, []):
if n[0].lower() != node.lower():
continue
views.setdefault(f"{n[2]} (prio {n[1]})", []).append(d)
if not views:
return DASH
if len(views) == 1:
return esc(next(iter(views)))
mine = [t for t, who in views.items() if any(d.node == node for d in who)]
bits = [esc(mine[0])] if mine else []
for t, who in views.items():
if mine and t == mine[0]:
continue
bits += [f"{esc(node_head(d))} reports it {esc(t)}" for d in who]
return "; ".join(bits) + warn_pill("nodes disagree")
def redundancy_table(members: list) -> str:
nodes, rgs = [], set()
for d in members:
for rg, ns in d.rg_nodes.items():
rgs.add(rg)
for n in ns:
if n[0].lower() not in nodes:
nodes.append(n[0].lower())
if not rgs:
return ""
nodes.sort()
rows = []
for rg in sorted(rgs, key=int):
counts: dict = {}
for d in members:
if rg in d.rg_failovers:
counts.setdefault(d.rg_failovers[rg], []).append(d)
if not counts:
fail = DASH
elif len(counts) == 1:
fail = esc(next(iter(counts)))
else:
fail = ", ".join(f"{esc(node_head(d))} {esc(v)}"
for v, who in counts.items() for d in who)
cells = [f"<td>RG{esc(rg)}</td>", f'<td class="num">{fail}</td>']
for node in nodes:
cells.append(f"<td>{rg_cell(members, rg, node)}</td>")
rows.append(cells)
return rich_table(["Redundancy group", "Failovers"] + nodes, rows)
# ---------------------------------------------------------------------------
# One row per CHECK, one cell per node
#
# This is what removed the repetition. The four cluster checks used to be
# printed in full under every node: 22 blocks describing four facts. They are
# one row each now, with their verdict in every node's column - the same shape
# as every other check, so the table has one grammar and not two.
# ---------------------------------------------------------------------------
def check_rows(members: list, predicate) -> list:
"""[(worst status, check name, [cell per node])], worst first."""
names, byname = [], {}
for d in members:
for c in d.checks:
if c.name not in byname:
byname[c.name] = {}
names.append(c.name)
byname[c.name][id(d)] = c
out = []
for name in names:
cells, picked = [], []
for d in members:
c = byname[name].get(id(d))
if c is not None and predicate(c):
cells.append(c.summary)
picked.append(c)
else:
cells.append("")
if not picked:
continue
worst = min((c.status for c in picked), key=lambda s: RANK[s])
out.append((worst, name, cells))
out.sort(key=lambda r: RANK[r[0]])
return out
def node_cells(members: list, cells: list, render) -> list[str]:
"""
One cell per node, always - including a cluster check whose nodes agree.
A cluster check used to collapse into a single cell spanning the node
columns and reading "cluster / both nodes - OK". It is per node now, and
the reason is the reason the rest of the page is a grid: a table is read
DOWN a column, and a column headed node0 that sometimes holds node0's
answer and sometimes holds the pair's makes the reader parse two different
grammars in the same table. If both cells say the same thing, both cells
say it - two short words is a cheaper price than an exception.
The collapsing of identical content INSIDE a detail block is a different
thing and stays: there it folds two full tables into one.
"""
return [f"<td>{render(c) if c else DASH}</td>" for c in cells]
def findings_table(members: list, predicate, extra: list = ()) -> str:
rows = list(extra) + check_rows(members, predicate)
rows.sort(key=lambda r: RANK[r[0]])
if not rows:
return ""
body = []
for worst, name, cells in rows:
tds = [f"<td>{tag(worst)}</td>", f"<td>{esc(name)}</td>"]
tds += node_cells(members, cells, esc)
body.append(tds)
heads = ["Severity", "Check"] + [node_head(d) for d in members]
return rich_table(heads, body)
def coverage_table(members: list) -> str:
"""
Every check that ran, as a severity per node and nothing else.
No summaries here, on purpose: the wording of a finding belongs in the
findings table and repeating it is the mistake this structure exists to
remove. This table answers one question - what was looked at.
"""
names, byname = [], {}
for d in members:
for c in d.checks:
if c.name not in byname:
byname[c.name] = {}
names.append(c.name)
byname[c.name][id(d)] = c
body = []
for name in names:
statuses = []
for d in members:
c = byname[name].get(id(d))
statuses.append(c.status if c else "")
tds = [f"<td>{esc(name)}</td>"]
tds += node_cells(members, statuses, tag)
body.append(tds)
return rich_table(["Check"] + [node_head(d) for d in members], body)
# ---------------------------------------------------------------------------
# The detail: one block per FAILING check, nodes as columns
#
# A detail block exists for a check that came back CRITICAL, WARNING or
# UNDETERMINED, and for nothing else. That one rule removed two thirds of the
# report. What is healthy is already answered by its row in `Every check`, and
# explaining a passing check is pure volume: on a healthy cluster 16 of the 18
# blocks this section used to print were describing OK or INFO.
#
# Inside the block the nodes are COLUMNS, like every other table on the page.
# One row per element - per fan, per PIC, per slot, per interface - and one
# short state per node. The same element never appears twice because it exists
# on two nodes; that was the defect that made `Fans and environment` 14 rows.
# The OK node keeps its column, because "node0 Failed / node1 OK" is the
# comparison that makes the row readable, and it costs one short cell.
# ---------------------------------------------------------------------------
DETAIL_SHOW = (CRIT, WARN, UNKNOWN)
# How each check's own table is folded into one cell per node.
# check name: ([key column indices], "cell format over the row")
# The key columns identify the element and stay as their own columns; the
# format builds the one short state that goes in the node column. Anything not
# listed falls back to column 0 as the key and the rest joined - which is
# right for a two-column table and merely terse for a wider one.
DETAIL_PIVOT = {
"Chassis and system alarms": ([3], "{1} / {2} / {0}"),
"Filesystem usage": ([0], "{3} of {2}"),
"Interface state": ([0], "{1}"),
"Interface error counters": ([0],
"in {1} / framing {2} / crc {3} / out {4} / "
"carrier {5}"),
"Licences": ([0], "{3} needed, expires {4}"),
"PIC status": ([0, 1], "{2} / {3}"),
"Fans and environment": ([0, 1], "{2} / {3}"),
"LACP bundles": ([0, 1], "{4} / partner defaulted {2} / receive {3}"),
"LLDP neighbours": ([0], "{4} {3} ({2})"),
"FPC / SPC slots": ([0], "{1} / {2} / {3} C / {6} MB DRAM / {7}"),
"SPU cards (high-end data plane)": ([0, 1],
"cpu {2} / memory {3} / sessions {4}"),
"Chassis cluster status": ([0, 1], "{3}, priority {2}, monitor {6}"),
"Cluster failover history and LED": ([1, 2], "{3} to {4}: {5}"),
"Cluster control and fabric links": ([0, 1, 2], "{3}"),
}
# One line of what to do, per check. This is the only prose left in a detail
# block, it is the only thing the reader cannot work out from the table, and
# it is NOT the finding - the finding is already a row in Findings. Nothing is
# invented here: each line is the action the check's own threshold implies.
DETAIL_ACTION = {
"Software and uptime": "Confirm the reboot was planned, and that both "
"nodes run the same Junos release.",
"Chassis hardware": "Quote the serial number on the support case or RMA.",
"Chassis cluster status": "A node reported lost or disabled cannot take "
"over: check the control link before anything "
"else.",
"Cluster failover history and LED": "Clear the underlying failure the LED "
"reason names; the LED follows it.",
"Cluster control and fabric links": "Control link down is split-brain "
"waiting to happen - repair it first. "
"Fabric down loses session sync.",
"Cluster heartbeat and fabric probes": "Probes sent and none returning "
"means that child link is not "
"passing traffic - check the cable "
"and the switch in between.",
"Chassis and system alarms": "Clear the alarm at source; a Major alarm is "
"the chassis asking for hardware attention.",
"Routing Engine load": "Find the process holding the CPU or the memory "
"before it starts dropping control traffic.",
"Fans and environment": "Replace the failed or absent fan. The chassis is "
"on a thermal clock until it is back.",
"Filesystem usage": "Clear old logs and software images: a full /var "
"stops logging and can block a commit.",
"Core dumps": "Collect the dumps for JTAC, then remove them - a daemon "
"crashed and the evidence is still on disk.",
"Interface state": "An enabled port with no link is a cable, an optic or "
"the far end. On a reth or fab member it is a data "
"path.",
"Interface error counters": "Errors point at the physical layer: optic, "
"patch lead, or a duplex mismatch.",
"LACP bundles": "A member outside Collecting distributing is capacity "
"lost with nothing down. Fix the far end's LACP.",
"LLDP neighbours": "Confirm the neighbour on each port is the one you "
"expect.",
"Flow session table": "Raise the session limit or find what is opening "
"the flows: new sessions drop once the table fills.",
"FPC / SPC slots": "A card reported Present with a hardware note is "
"capacity the chassis can see and is not using.",
"PIC status": "Confirm the PIC was taken offline on purpose. Offline is a "
"decision, not a fault.",
"SPU cards (high-end data plane)": "A saturated SPU drops traffic while "
"the Routing Engine still looks idle.",
"SPU utilisation over the last minute": "Check the peak, not the average: "
"the data plane is sized for the "
"peak.",
"IPsec VPN": "An IKE SA up with no IPsec SA is a phase 2 proposal or "
"proxy-ID mismatch.",
"Licences": "Order the licence for the feature in use, or stop using "
"the feature.",
"Last configuration commit": "Check the last change against when the "
"symptom started.",
SPLIT_BRAIN_CHECK: "Repair the control link. Until it carries heartbeats "
"again both nodes forward and the pair cannot fail "
"over.",
}
def pivot_detail(name: str, members: list, byname: dict) -> str:
"""
One check's table, folded so that the nodes are columns.
Every member of the group gets a column, including the ones that came back
healthy - that is what turns two tables into one. An element the node did
not report is a dash, which here means "not in that node's output" and
never zero. A key that repeats inside one node (two alarms from the same
source) keeps its own row, matched across nodes by order.
A cell whose row came back critical, warning or undetermined is TINTED in
that severity's colour, so the failing element is found INSIDE the table
and not only in the verdict line above it. The whole cell, not a pill in
front of the text: the pill repeated a word the cell's own text already
carried - `WARNING 1 needed, expires -` - and spent half the column on
it. The severity follows the row OF ITS OWN NODE: it travels with the
cell, not with the index of the merged row, which is why it is stored
alongside the text. OK, INFO, N/A and a check that does not grade row by
row are left plain: the tint POINTS, and a table with every cell painted
points at nothing.
"""
spec = DETAIL_PIVOT.get(name)
headers: list[str] = []
for d in members:
c = byname[name].get(id(d))
if c is not None and c.headers and c.rows:
headers = c.headers
break
if not headers:
return ""
keycols = [i for i in (spec[0] if spec else [0]) if i < len(headers)]
fmt = spec[1] if spec else None
order: list = []
data: dict = {}
for d in members:
c = byname[name].get(id(d))
if c is None:
continue
counts: dict = {}
for i, raw_row in enumerate(c.rows):
row_sev = c.row_status[i] if i < len(c.row_status) else ""
row = list(raw_row) + [DASH] * (len(headers) - len(raw_row))
k = tuple(row[i] for i in keycols)
n = counts.get(k, 0)
counts[k] = n + 1
key = k + (n,)
if key not in data:
data[key] = {}
order.append(key)
rest = " / ".join(row[j] for j in range(len(headers))
if j not in keycols)
try:
cell = fmt.format(*row) if fmt else rest
except (IndexError, KeyError):
cell = rest
data[key][id(d)] = (cell, row_sev)
if not order:
return ""
rows = []
for key in order:
tds = [f"<td>{esc(v)}</td>" for v in key[:-1]]
for d in members:
cell, cell_sev = data[key].get(id(d), (DASH, ""))
cls = (f' class="c-{sev_slug(cell_sev)}"'
if cell_sev in DETAIL_SHOW else "")
tds.append(f"<td{cls}>{esc(cell)}</td>")
rows.append(tds)
heads = [headers[i] for i in keycols] + [node_head(d) for d in members]
return rich_table(heads, rows)
def detail_blocks(members: list) -> list[str]:
parts, names, byname = [], [], {}
for d in members:
for c in d.checks:
byname.setdefault(c.name, {})[id(d)] = c
for d in members:
for c in d.checks:
if c.name in names or c.name in SUMMARY_CHECKS:
continue
if any(x.status in DETAIL_SHOW for x in byname[c.name].values()):
names.append(c.name)
names.sort(key=lambda n: min(RANK[c.status] for c in byname[n].values()
if c.status in DETAIL_SHOW))
for name in names:
parts.append(f"<h3>{esc(name)}</h3>")
verdicts = []
for d in members:
c = byname[name].get(id(d))
if c is None or c.status not in DETAIL_SHOW:
continue
who = "" if len(members) == 1 else f" {esc(node_head(d))}"
verdicts.append(f"{tag(c.status)}{who}")
parts.append('<p class="note">' + " ".join(verdicts) + "</p>")
action = DETAIL_ACTION.get(name)
if action:
parts.append(f'<p class="note"><strong>Do</strong> {esc(action)}'
"</p>")
pivoted = pivot_detail(name, members, byname)
if pivoted:
parts.append(pivoted)
continue
# No table of its own: the counters this check read ARE its detail, so
# they are printed - but only for the nodes that did not pass.
for d in members:
c = byname[name].get(id(d))
if c is None or c.status not in DETAIL_SHOW or not c.detail:
continue
if len(members) > 1:
parts.append(f'<p class="note">{esc(node_head(d))}</p>')
parts.append(bullets(c.detail))
return parts
def split_brain_block(split: list[str], members: list) -> list[str]:
"""
Split-brain in Detail, as one more CRITICAL and with no special billing.
It used to have a section of its own ON TOP of its row in Findings, which
made the report read as though it were about split-brain whatever it had
actually found. It is one finding out of 22. The detection did not change;
only the billing did.
"""
who = f" {esc(nodes_word(members))}" if len(members) > 1 else ""
return [f"<h3>{esc(SPLIT_BRAIN_CHECK)}</h3>",
f'<p class="note">{tag(CRIT)}{who}</p>',
'<p class="note"><strong>Do</strong> '
f"{esc(DETAIL_ACTION[SPLIT_BRAIN_CHECK])}</p>",
bullets(split)]
# ---------------------------------------------------------------------------
# The top of the report: the severity table and nothing else
#
# There were four paragraphs of plain English here, and they were 230 words of
# the 2,300 the report had. They are gone. The reader opens the report and
# sees the count per severity, which is the question they came with; the
# readable label in the third column is what keeps it legible for somebody who
# does not know what a redundancy group is - one line per row, instead of a
# paragraph per idea.
# ---------------------------------------------------------------------------
SEVERITY_MEANING = [
(CRIT, "Needs action now (critical)"),
(WARN, "Should be planned (warning)"),
(UNKNOWN, "Could not be checked (undetermined)"),
(OK, "Verified healthy (ok)"),
(INFO, "Information only (info)"),
(NA, "Does not apply (not applicable)"),
]
def severity_table(tally: dict) -> str:
"""
The count per severity, with the pill on all six rows.
Here the pill does not POINT - it CLASSIFIES. This table is the legend of
the six states, so every row carries its own colour, and it is the same
colour the reader meets again in the rest of the report: the severity
cell is rendered as the pill the page already uses, in the classes the
stylesheet already has. Nothing new was styled for it.
There was a verdict box above this table saying `CRITICAL - Action
required`, and it repeated what the first row of the table already said.
The box is gone and what it was doing lives in the rows themselves.
"""
rows = []
for key, label in SEVERITY_MEANING:
rows.append([f"<td>{tag(key)}</td>", f"<td>{tally[key]}</td>",
f"<td>{esc(label)}</td>"])
return rich_table(["Severity", "Count", "What it means"], rows)
def render_html(report: dict) -> str:
devices: list = report["devices"]
tally = report["tally"]
groups = cluster_groups(devices)
# THE REPORT OPENS WITH THE EQUIPMENT, not with metadata about the run.
# There was a block here listing Generated, Devices polled, Checks per
# device, Written against and Produced by. Two of those are answered by
# the tables below - the devices ARE the columns and the checks ARE the
# rows - and the other three are about the tool rather than the firewall,
# so they moved to one line in the footer. The date is not lost: an
# archived report without one is worthless, it just is not the headline.
parts = [
"<!DOCTYPE html>",
'<html lang="en"><head><meta charset="utf-8">',
'<meta name="viewport" content="width=device-width, initial-scale=1">',
f"<title>SRX health check {esc(report['generated'])}</title>",
f"<style>{CSS}</style></head><body>",
"<h1>Juniper SRX Health Check</h1>",
'<p class="sub">Read-only report. No configuration was changed on any '
"device.</p>",
]
# With one group the severity table belongs INSIDE it, straight after the
# equipment: you see which firewall this is, then how it came back. With
# more than one it is a summary OF ALL of them, so it cannot sit inside
# any single one and goes above them instead.
if len(groups) > 1:
parts.append(severity_table(tally))
for name, inferred, members in groups:
parts.append('<div class="device">')
parts.append(f"<h2>{group_title(name, members, len(groups) > 1)}</h2>")
if inferred and len(members) > 1:
parts.append('<p class="note">These nodes were grouped by the '
"Cluster ID they report, because the inventory gave "
"no <code>cluster:</code> label, and a Cluster ID is "
"only locally significant. If they are in fact two "
"separate clusters that both use this ID, label them "
"and run it again.</p>")
parts.append("<h3>Equipment</h3>")
eq, notes = equipment_table(members)
parts.append(eq)
for note in notes:
parts.append(f'<p class="note">{esc(note)}</p>')
if len(groups) == 1:
parts.append(severity_table(tally))
rgt = redundancy_table(members)
if rgt:
parts.append("<h3>Redundancy groups</h3>")
parts.append('<p class="note">A redundancy group is a set of '
"interfaces that fails over together: one node owns "
"it and the other stands by. The failover count is "
"cumulative since boot, so a number there is history "
"and not a fault on its own.</p>")
parts.append(rgt)
split = [s for s in report["split_brain"] if s.startswith(name + ",")]
extra = []
if split:
extra.append((CRIT, SPLIT_BRAIN_CHECK,
["Both nodes report themselves as primary for the "
"same redundancy group at the same time, so both "
"are forwarding. The control link between them is "
"down."] * len(members)))
parts.append("<h3>Findings</h3>")
found = findings_table(members, lambda c: c.status in (CRIT, WARN),
extra)
parts.append(found or '<p class="note">No check came back critical or '
"warning.</p>")
undet = findings_table(members, lambda c: c.status == UNKNOWN)
unreachable = [d for d in members if d.status != "ok"]
if undet or unreachable:
parts.append("<h3>Could not verify</h3>")
parts.append('<p class="note">These are not findings. A check '
"that did not answer says nothing about the device, "
"and reporting it as a pass or as a fault would both "
"be wrong.</p>")
if unreachable:
parts.append(table(["Node", "Why it could not be polled"],
[[node_head(d), d.error]
for d in unreachable]))
if undet:
parts.append(undet)
parts.append("<h3>Every check</h3>")
parts.append('<p class="note">What was looked at, and how it came '
"back. The first two report into Equipment above; the "
"wording of each finding is in Findings.</p>")
parts.append(coverage_table(members))
parts.append("<h3>Detail</h3>")
parts.append('<p class="note">One block per check that came back '
"critical, warning or undetermined. A check that passed "
"has its row in Every check and needs nothing said about "
"it.</p>")
if split:
parts.extend(split_brain_block(split, members))
parts.extend(detail_blocks(members))
parts.append("</div>")
# One line, in the grey the stylesheet already gives the footer. The date
# is here because an archived report without one is worthless. `Written
# against` is here because it is a real warning - these commands change
# between releases - and a warning belongs in the footnote, not on the
# cover. PermitANY is here as attribution and as text only: the report
# belongs to the engineer who ran it, so the footer is the honest place
# for our name and the header would be appropriation.
parts.append(
f"<footer>Generated {esc(report['generated'])} · "
f"{esc(report['tool'])} · PermitANY · Written against "
f"{esc(report['junos_target'])}</footer></body></html>")
return "".join(p for p in parts if p)
# ---------------------------------------------------------------------------
# Collecting from one device
# ---------------------------------------------------------------------------
def spu_slots(args, fpc_text: str | None) -> list[str]:
"""
Which FPC/SPC slots to read.
Default is `auto`, which means ask the chassis: the slots come from
`show chassis fpc` and only the ones reporting Online are used. If that
output cannot be parsed the answer is an EMPTY list, deliberately - there
is no fallback to slot 0. Reading a slot this script has no evidence
exists and presenting the result as the state of the chassis is the kind
of confident guess the whole tool is built to avoid. The SPU checks then
come out as undetermined, which is the honest answer.
Pass --spu-fpc 0,1 to name the slots by hand instead.
"""
want = str(args.spu_fpc).strip().lower()
if want in ("auto", ""):
return online_fpc_slots(fpc_text)
return [s.strip() for s in want.split(",") if s.strip()]
def extra_commands(version_text: str | None, fpc_text: str | None,
args) -> dict:
"""
The commands that depend on WHAT the device turned out to be.
`show version` and `show chassis fpc` are read first and on their own for
this reason. The SPU commands only exist on the high-end chassis, and they
need a slot number that only the chassis can supply. Asking the model
first means those checks are reported as not applicable on a branch SRX,
which is the truth, instead of undetermined, which is not.
"""
extra = {}
if args.check_errors:
extra["errors"] = ERRORS_COMMAND
if is_high_end(version_text):
for slot in spu_slots(args, fpc_text):
extra[f"spu_fpc_{slot}"] = spu_fpc_command(slot)
extra["spu_perf"] = SPU_PERF_COMMAND
return extra
def collect_live(dev: dict, args) -> dict:
conn = None
try:
conn = connect(dev)
raw = {"_prompt": (read_prompt(conn), "")}
for first in ("version", "fpc"):
raw[first] = send(conn, BASE_COMMANDS[first], args.read_timeout)
wanted = {k: v for k, v in BASE_COMMANDS.items()
if k not in ("version", "fpc")}
wanted.update(CLUSTER_COMMANDS)
wanted.update(extra_commands(raw["version"][0], raw["fpc"][0], args))
for key, cmd in wanted.items():
raw[key] = send(conn, cmd, args.read_timeout)
return raw
finally:
if conn:
try:
conn.disconnect()
except Exception: # noqa: BLE001 - closing must never raise
pass
def collect_replay(dev: dict, args) -> dict:
"""
Read captured command output from disk instead of a device.
One directory per host, one file per command key, so a run can be
reproduced and a parser can be tested without touching a production
firewall. A missing file is treated as "did not answer" - which is
exactly how a device that does not implement the command behaves.
"""
base = Path(args.replay) / dev["host"]
def read_capture(key: str):
f = base / f"{key}.txt"
if not f.exists():
return (None, f"no capture at {f}")
text = f.read_text(encoding="utf-8", errors="replace")
if NOT_CLUSTERED.search(text):
return (text, "")
if JUNOS_ERROR.search(text):
first = next((l.strip() for l in text.splitlines() if l.strip()), "")
return (None, f"command rejected: {first[:110]}")
return (text, "")
raw = {"_prompt": read_capture("prompt"),
"version": read_capture("version"), "fpc": read_capture("fpc")}
wanted = {k: v for k, v in BASE_COMMANDS.items()
if k not in ("version", "fpc")}
wanted.update(CLUSTER_COMMANDS)
wanted.update(extra_commands(raw["version"][0], raw["fpc"][0], args))
for key in wanted:
raw[key] = read_capture(key)
return raw
def poll_device(dev: dict, args) -> DeviceResult:
host = dev["host"]
res = DeviceResult(host=host, label=dev.get("device_name") or host,
cluster_group=str(dev.get("cluster") or ""),
node=str(dev.get("node") or "").strip().lower())
if dev.get("os") != "junos":
res.status = "error"
res.error = (f"os is '{dev.get('os')}' - this script is SRX only, so "
f"every device in the inventory needs os: junos")
return res
try:
raw = collect_replay(dev, args) if args.replay else collect_live(dev, args)
except NetmikoAuthenticationException as exc:
res.status = "error"
res.error = f"authentication failed - check NET_USER/NET_PASS ({exc})"
return res
except NetmikoTimeoutException as exc:
res.status = "error"
res.error = f"unreachable or SSH timed out ({exc})"
return res
except KeyboardInterrupt:
raise
except Exception as exc: # noqa: BLE001 - one device must not abort the run
res.status = "error"
res.error = f"{type(exc).__name__}: {exc}"
return res
res.checks = [check_identity(res, raw)]
res.checks.append(check_hardware(res, raw))
res.checks.append(check_cluster_status(res, raw))
res.checks.append(check_cluster_information(res, raw))
res.checks.append(check_cluster_links(raw))
res.checks.append(check_cluster_statistics(raw))
res.checks.append(check_alarms(raw))
res.checks.append(check_routing_engine(raw))
res.checks.append(check_environment(raw))
res.checks.append(check_storage(raw))
res.checks.append(check_core_dumps(raw))
res.checks.append(check_interfaces(raw))
if args.check_errors:
res.checks.append(check_interface_errors(raw))
res.checks.append(check_lacp(raw))
res.checks.append(check_lldp(raw))
res.checks.append(check_flow_sessions(raw))
high_end = is_high_end(raw["version"][0])
res.checks.append(check_fpc_state(raw))
res.checks.append(check_pic_status(raw))
res.checks.append(check_spu_fpc(raw, high_end))
res.checks.append(check_spu_performance(raw, high_end))
res.checks.append(check_vpn(raw))
res.checks.append(check_license(raw))
res.checks.append(check_commit(raw))
finalise(res, raw)
return res
# ---------------------------------------------------------------------------
# Terminal output and entry point
# ---------------------------------------------------------------------------
def default_report_path() -> str:
stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%SZ")
return f"srx-healthcheck-{stamp}.html"
def main() -> int:
ap = argparse.ArgumentParser(
description="Read-only health check for Juniper SRX (branch, mid-range, "
"high-end and chassis cluster) that writes an HTML report.",
epilog="Credentials come from NET_USER and NET_PASS, never from an "
"argument. Written against " + JUNOS_TARGET + ".",
)
ap.add_argument("--inventory", required=True,
help="Path to devices.yaml listing the SRX devices to poll")
ap.add_argument("--limit",
help="Only poll hosts whose name contains this text")
ap.add_argument("--report", default=None,
help="Where to write the HTML report. Default: "
"srx-healthcheck-<UTC timestamp>.html in the current "
"directory")
ap.add_argument("--no-report", action="store_true",
help="Print to the terminal only and write no HTML file")
ap.add_argument("--open", action="store_true", dest="open_report",
help="Open the finished HTML report in your browser")
ap.add_argument("--check-errors", action="store_true",
help="Also collect `show interfaces extensive` and grade the "
"input, framing, CRC and carrier-transition counters. "
"Slower on a chassis with many ports")
ap.add_argument("--spu-fpc", default="auto", metavar="LIST",
help="Which FPC/SPC slots to read on a high-end chassis. "
"Default `auto`, which takes the slots reporting "
"Online from `show chassis fpc`. Give a comma "
"separated list such as 0,1 to override. Ignored on "
"branch and mid-range, where the command does not "
"apply")
ap.add_argument("--replay", metavar="DIR",
help="Read captured command output from DIR/<host>/<key>.txt "
"instead of connecting. For testing and for replaying a "
"capture taken during an outage")
ap.add_argument("--read-timeout", type=int, default=90,
help="Seconds to wait per command. Default: 90")
ap.add_argument("--workers", type=int, default=8,
help="How many devices to poll at once. Default: 8")
ap.add_argument("--json-out",
help="Also write the same results as JSON to this file")
args = ap.parse_args()
devices = load_inventory(args.inventory)
if args.limit:
devices = [d for d in devices if args.limit in d["host"]]
if not devices:
raise SystemExit(f"No device name contains '{args.limit}'.")
if not args.replay:
credentials() # fail fast, before any connection is opened
mode = "replaying captures for" if args.replay else "polling"
print(f"SRX health check - {mode} {len(devices)} device(s). Read-only.")
print("=" * 78)
results: list[DeviceResult] = []
with ThreadPoolExecutor(max_workers=args.workers) as pool:
futures = {pool.submit(poll_device, d, args): d["host"] for d in devices}
for fut in as_completed(futures):
host = futures[fut]
try:
results.append(fut.result())
except KeyboardInterrupt:
raise
except Exception as exc: # noqa: BLE001
r = DeviceResult(host=host, label=host, status="error",
error=f"worker crashed: {exc}")
results.append(r)
results.sort(key=lambda r: r.label)
split = correlate_split_brain(results)
tally = {CRIT: 0, WARN: 0, UNKNOWN: 0, INFO: 0, OK: 0, NA: 0}
for r in results:
for c in r.checks:
tally[c.status] = tally.get(c.status, 0) + 1
for r in results:
if r.status != "ok":
print(f"\n[NOT POLLED] {r.label}")
print(f" {r.error}")
continue
print(f"\n{r.label} - {r.hostname} / {r.model} / s/n {r.serial} "
f"/ Junos {r.version} / {r.role}")
if r.rg_failovers:
print(f" failovers since boot: "
f"{failover_summary(r)}")
for c in r.checks:
if c.status == NA:
continue
print(f" [{c.status:<8}] {c.name:<34} {c.summary}")
if split:
print("\n" + "=" * 78)
for s in split:
print(f"[CRITICAL] {s}")
print("\n" + "=" * 78)
print(f"{tally[CRIT]} CRITICAL, {tally[WARN]} WARNING, "
f"{tally[UNKNOWN]} undetermined, {tally[OK]} OK "
f"across {len(results)} device(s).")
if tally[UNKNOWN]:
print("Undetermined is not a pass: those checks could not be decided.")
report = {
"generated": datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC"),
"tool": "junos-srx-healthcheck",
"junos_target": JUNOS_TARGET,
"checks_per_device": max((len(r.checks) for r in results), default=0),
"devices": results,
"split_brain": split,
"tally": tally,
}
if not args.no_report:
path = Path(args.report or default_report_path())
path.write_text(render_html(report), encoding="utf-8")
print(f"\nHTML report written to: {path.resolve()}")
print("Open it with a double click. It needs no internet connection.")
if args.open_report:
import webbrowser
webbrowser.open(path.resolve().as_uri())
if args.json_out:
payload = {k: v for k, v in report.items() if k != "devices"}
payload["devices"] = [asdict(r) for r in results]
Path(args.json_out).write_text(json.dumps(payload, indent=2),
encoding="utf-8")
print(f"JSON written to: {Path(args.json_out).resolve()}")
unreachable = sum(1 for r in results if r.status != "ok")
if tally[CRIT] or split:
return 2
if tally[WARN] or tally[UNKNOWN] or unreachable:
return 1
return 0
if __name__ == "__main__":
try:
sys.exit(main())
except KeyboardInterrupt:
print("\nInterrupted.", file=sys.stderr)
sys.exit(130)
This is the report the script writes to disk. It is a single self-contained file with no external dependencies, so it opens without an internet connection.
--inventorytextrequiredPath to your devices.yaml. Lists which SRX devices to poll, each with a host and os: junos. Give both nodes of a chassis cluster the same cluster label — that is what turns on the split-brain check. Full format in the documentation below.
--limittextOnly poll hosts whose name contains this text. The safe way to widen from one device to many.
--reporttextWhere to write the HTML report. Default: srx-healthcheck-<UTC timestamp>.html in the current directory.
--no-reportboolTerminal output only, no HTML file.
--openboolOpen the finished HTML report in your browser.
--check-errorsboolAlso collect show interfaces extensive and grade the input, framing, CRC and carrier-transition counters. Off by default because it is slow on a chassis with hundreds of ports.
--spu-fpctextWhich FPC/SPC slots to read on a high-end chassis (SRX5400/5600/5800). Default: auto, which takes the slots reporting Online from show chassis fpc. Pass a comma separated list such as 0,1 to override. Ignored on branch and mid-range, where the command does not apply.
--replaytextRead captured command output from DIR/<host>/<key>.txt instead of connecting to anything. Tests the parsers offline, and re-analyses a capture taken during an outage after the device has recovered.
--read-timeoutnumberSeconds to wait per command. Default: 90
--workersnumberHow many devices to poll at once. Default: 8
--json-outtextAlso write the same results as JSON to this file.
What This Solves
Somebody asks whether the SRX estate is healthy, and the honest answer is that nobody knows. Checking means logging into each firewall, running twenty-odd operational commands, remembering which number is bad on which model, and then writing it up by hand. So it gets done the week before an audit and not again for a year. This script does that walk and hands you an HTML file you can send to someone else. It answers three questions a single `show` command cannot: Is the cluster really a cluster? One node is enough to see the cluster. `show chassis cluster status` and `show chassis cluster information` return the state of BOTH nodes from either one of them, because the cluster tables are replicated over the control link - so every cluster check here runs the same in both versions of the script, including the SecureCRT one with its single session. What a single node sees when there IS a split-brain is the SIGNATURE of it: the peer reported as `lost` while the control link is down. That is graded CRITICAL from one node, and the cluster LED turns amber with the reason written on it. What polling both nodes adds is one further fact: the CONFIRMATION that both nodes are primary for the same redundancy group at the same moment, which means both are forwarding. The signature and the confirmation are different facts, and this script reports them as different facts. What is close to the edge, not just what is already broken? A session table at 91% is not an outage today. It is an outage on the busiest afternoon of the quarter. Flow sessions, /var, Routing Engine memory and licence expiry are all graded before they bite. What could NOT be checked? An undetermined check is reported as undetermined, never as a pass. A counter this script cannot find prints as a dash and not as a zero, because "0 errors" and "I could not read the error counter" are opposite findings and only one of them is good news. It is written against Junos OS 21.4R3 and later and covers the whole SRX range - branch (SRX300/320/340/380), mid-range (SRX1500/4000) and high-end (SRX5400/5600/5800) - standalone or in a chassis cluster. It runs 22 checks per device, 23 with --check-errors. On the high-end it also reads the SPU cards, because there the data plane can be saturated while the Routing Engine looks fine. There are two versions of it on this page. The main one connects to many devices from a YAML inventory. The SecureCRT one runs inside SecureCRT on the session you already have open, needs nothing installed, and is covered in Running It from SecureCRT below.
Is It Safe to Run
Yes. It is strictly read-only. It never enters configuration mode, it never commits anything, and it never writes to a device. The only file it creates is the HTML report, on your own machine. These are the only commands it sends, all of them Junos operational mode: Always show version show chassis hardware show system uptime show chassis routing-engine show chassis alarms show system alarms show chassis environment show system storage show system core-dumps show interfaces terse show lacp interfaces show lldp neighbors show security flow session summary show security ipsec security-associations show security ike security-associations show system license show system commit show chassis fpc detail show chassis fpc pic-status Always, and skipped as not applicable on a standalone SRX show chassis cluster status show chassis cluster information show chassis cluster interfaces show chassis cluster statistics Only with --check-errors show interfaces extensive Only on a high-end chassis (SRX5400/5600/5800) show security monitoring fpc <slot> show security monitoring performance spu summary The slot numbers in that last pair are not guessed. They come from `show chassis fpc detail`, and only the slots reporting Online are read. Three of the commands in that list may not apply to your box, and the check each one feeds then says so rather than failing: `show chassis environment` prints a different shape on every platform and next to nothing on vSRX; `show lacp interfaces` has nothing to say without an aggregated Ethernet interface; and `show lldp neighbors` errors out when LLDP is not configured. All three come back as not applicable, which is a third thing - not a pass and not a finding. The SecureCRT version sends the same list plus `show cli` and two `set cli screen-length` commands. Pagination is turned off before the first command whose output is parsed; the only thing sent ahead of it is `show cli`, which is three lines long and is what lets the original screen length be put back exactly. Those turn pagination off and then put it back exactly as it was. `set cli` is an operational-mode setting for your own terminal session - it is not configuration, it is not committed, and it changes nothing for anyone else. If a command is not implemented on your platform or your account may not run it, that check is reported as undetermined. It is never silently graded as healthy.
Before You Start
Three things, once. The SecureCRT version needs none of them, only SecureCRT 9.0 or later - skip to Running It from SecureCRT if that is the one you want. 1. Python 3.9 or newer. Check what you have: python --version 2. The two libraries this script imports: pip install "netmiko>=4.3" "PyYAML>=6.0" If pip is not found, use "python -m pip install" instead. 3. SSH access to the SRX from the machine you are running this on, with an account that can run show commands. Prove it by hand first - the script cannot do anything your own SSH session cannot: ssh admin@10.10.0.11 show chassis cluster status If that works for you manually, the script will work.
Your First Run
Start with ONE device. Not the whole estate, and not the internet edge during business hours. Pick one firewall you are comfortable with, read the report, then widen. Step 1 - create a file named devices.yaml next to the script, with a single device in it: devices: - host: 10.10.0.11 os: junos The os value is always junos. This script is SRX only, and any other value is rejected rather than guessed at. Step 2 - put your credentials in the environment, not in the file: export NET_USER='admin' export NET_PASS='your-password' On Windows PowerShell: $env:NET_USER = 'admin' $env:NET_PASS = 'your-password' Step 3 - run it: python junos_srx_healthcheck.py --inventory devices.yaml It connects, sends the show commands listed above, prints a summary, and writes srx-healthcheck-<timestamp>.html into the current directory. Nothing changed on the firewall. Once that looks right, add the rest. A single node already reports the whole cluster, so one entry is enough to grade the cluster. Put BOTH nodes in the inventory, with the same cluster label, when you also want the confirmation that the two of them are not forwarding at the same time: devices: - host: 10.10.0.11 os: junos cluster: hq device_name: srx-hq-node0 - host: 10.10.0.12 os: junos cluster: hq device_name: srx-hq-node1
What the Output Means
This is a real run against the two nodes of one healthy chassis cluster: both nodes up, both redundancy groups primary on node0, control and fabric links up. What it found is a failed fan on node0 with the Major chassis alarm it raises, a filesystem filling up on node1, a feature in use without a licence, a port enabled with no link, and one check it could not decide at all. That is what a normal Monday looks like, which is why it is the example and a split-brain is not. The run was produced with --replay against captured output, which is why it says "replaying captures" rather than "polling" - the Preview terminal shows the same run with the word a live poll prints. Everything else is exactly what you see against live kit, and it is the same run that produced the HTML in the Report tab. $ export NET_USER='admin' && export NET_PASS='********' $ python junos_srx_healthcheck.py --inventory devices.yaml SRX health check - replaying captures for 2 device(s). Read-only. ============================================================================== srx-dc-node0 - srx-dc-node0 / srx345 / s/n AE3213AA0033 / Junos 22.4R3-S3.3 / cluster 1: RG0 primary on node0, RG1 primary on node0 failovers since boot: RG0 1, RG1 2 [OK ] Software and uptime up 75 day(s) on Junos 22.4R3-S3.3 [INFO ] Chassis hardware chassis serial AE3213AA0033 [OK ] Chassis cluster status RG0 primary on node0; RG1 primary on node0 [OK ] Cluster failover history and LED cluster LED green on every node reported, 4 state change(s) in the history [OK ] Cluster control and fabric links control Up, fabric Up, 2 reth interface(s) up [OK ] Cluster heartbeat and fabric probes no heartbeat errors and fabric probes are returning [CRITICAL] Chassis and system alarms 2 alarm(s) active, 1 of them Major [OK ] Routing Engine load CPU idle 81%, memory 58% used [CRITICAL] Fans and environment SRX345 Chassis fan 1 is Failed - a failed or absent fan puts the chassis on a thermal clock [OK ] Filesystem usage 3 filesystem(s), busiest at 44% [OK ] Core dumps no core dumps on the device [WARNING ] Interface state 1 interface(s) enabled with no link: ge-0/0/3 [OK ] LACP bundles 1 bundle(s), 2 member(s), all collecting and distributing [INFO ] LLDP neighbours 2 LLDP neighbour(s) [OK ] Flow session table 2160 of 65536 sessions in use (3.3%) [OK ] FPC / SPC slots 1 slot(s) online, none dead, dormant, in diagnostics or unsupported [OK ] PIC status 2 PIC(s) online [OK ] IPsec VPN 2 IPsec tunnel(s) up, 2 IKE SA(s) up [WARNING ] Licences 1 feature(s) in use without a licence: idp-sig [INFO ] Last configuration commit 2026-11-01 18:22:41 UTC by netops via netconf srx-dc-node1 - srx-dc-node1 / srx345 / s/n AE3213AA0147 / Junos 22.4R3-S3.3 / cluster 1: RG0 primary on node0, RG1 primary on node0 failovers since boot: RG0 1, RG1 2 [OK ] Software and uptime up 75 day(s) on Junos 22.4R3-S3.3 [INFO ] Chassis hardware chassis serial AE3213AA0147 [OK ] Chassis cluster status RG0 primary on node0; RG1 primary on node0 [OK ] Cluster failover history and LED cluster LED green on every node reported, 4 state change(s) in the history [OK ] Cluster control and fabric links control Up, fabric Up, 2 reth interface(s) up [OK ] Cluster heartbeat and fabric probes no heartbeat errors and fabric probes are returning [WARNING ] Chassis and system alarms 1 alarm(s) active, 0 of them Major [OK ] Routing Engine load CPU idle 81%, memory 58% used [UNKNOWN ] Fans and environment SRX345 Chassis fan 2 reports Testing - neither verified good nor verified bad [WARNING ] Filesystem usage /.mount/var at 84% [OK ] Core dumps no core dumps on the device [OK ] Interface state every enabled interface has link (16 seen) [OK ] LACP bundles 1 bundle(s), 2 member(s), all collecting and distributing [INFO ] LLDP neighbours 2 LLDP neighbour(s) [OK ] Flow session table 2160 of 65536 sessions in use (3.3%) [OK ] FPC / SPC slots 1 slot(s) online, none dead, dormant, in diagnostics or unsupported [OK ] PIC status 2 PIC(s) online [OK ] IPsec VPN 2 IPsec tunnel(s) up, 2 IKE SA(s) up [WARNING ] Licences 1 feature(s) in use without a licence: idp-sig [INFO ] Last configuration commit 2026-11-01 18:22:41 UTC by netops via netconf ============================================================================== 2 CRITICAL, 5 WARNING, 1 undetermined, 26 OK across 2 device(s). Undetermined is not a pass: those checks could not be decided. HTML report written to: /home/you/srx-healthcheck-20261103-030412Z.html Open it with a double click. It needs no internet connection. Reading it: What the five results mean CRITICAL is verified bad: service is affected, or one more failure takes it out. WARNING is verified and outside the healthy range, so plan it. OK is verified healthy. INFO is context that is never graded, such as the last commit. UNKNOWN means the check could not be decided, which is not a pass - it is a gap in the evidence. There is also N/A, which only appears in the HTML report. It is a check that does not apply, such as the three cluster checks on a standalone SRX. The two halves of a split-brain Not in the run above - that cluster is healthy - and worth reading anyway, because it is the one failure a per-box check cannot see. When it happens it is a CRITICAL row in Findings and one block in Detail, with no section of its own: it is one finding out of 22. From ONE node you already see the signature, and it is graded CRITICAL there: [CRITICAL] Chassis cluster status RG0: node1 is lost - the peer is not being heard, which is how split-brain starts [CRITICAL] Cluster control and fabric links the control link is DOWN - each node will declare itself primary That works because the cluster tables are replicated: one node reports both nodes. The SecureCRT version, with its single session, grades exactly the same two checks. From BOTH nodes you additionally get the confirmation, printed on its own after every device: [CRITICAL] dc, RG1: SPLIT-BRAIN. srx-dc-node0 reports node0 as primary; srx-dc-node1 reports node1 as primary. That line says both nodes are primary for the same redundancy group at the same moment, so both are forwarding. Treat it as an outage even while traffic appears to work, because the two nodes are now writing different session tables. The cluster LED, and why it is a WARNING `show chassis cluster information` reports a chassis cluster LED with a colour and the device's own one-line reason for the last change, which is quoted verbatim because it is usually the shortest description of the fault anybody will get: [WARNING ] Cluster failover history and LED node0 cluster LED is Amber - last LED change reason: "Control link failure" Anything other than Green is a WARNING and deliberately not a CRITICAL. The LED is a SUMMARY of a failure that already has its own check here - a monitored interface that went down, a redundancy group whose weight dropped, a lost peer, a failed control link. Those are the CRITICALs. Grading the LED critical as well would count one failure twice and inflate the report, and a count nobody trusts is a count nobody acts on. The same command gives the state-change history per redundancy group - when, from what, to what and why - and the failover count per group goes into the device summary line. Fans, and why there is no temperature threshold A fan reported Failed or Absent is CRITICAL: the chassis is on a thermal clock from that moment. A fan in Testing, or with a status word this script does not recognise, is UNDETERMINED - neither verified good nor verified bad. Temperatures are printed and NOT graded, and that is a decision rather than an omission. Juniper publishes no safe range per SRX model, an over-temperature condition raises a chassis alarm which this report grades in its own check, and a number in degrees invented here would read like a vendor limit without being one. The same reasoning covers the per-FPC temperature, CPU and memory figures Half-formed LACP bundles, the quiet failure A bundle that came up on some of its members stays up. No interface is down, nothing alarms, and the missing capacity is simply gone until somebody counts the members by hand. So this one is CRITICAL, and it looks like this when it happens: [CRITICAL] LACP bundles ae0 member ge-0/0/5: the partner information is DEFAULTED - no LACP PDU is arriving `Defaulted` is Juniper's own term for no LACP PDU having arrived before the Current-state timer expired a second time. In plain terms the other end is not speaking LACP, or is not configured for it. A member whose Mux state is anything other than Collecting distributing is graded the same way, because only that state forwards Version and platform notes worth knowing before you read a verdict `show chassis cluster information` has existed since Junos 12.1X47-D10, so it is available across the whole range this script covers. Its `Redundancy mode` field - active-active or active-backup - was only added in 24.4R1, so nothing here depends on it: it is read when present and prints as a dash when it is not, and a dash never means the cluster has no mode. `show chassis environment` prints a different shape on every platform. A branch SRX names each fan ("SRX345 Chassis fan 1 OK Spinning at normal speed"), a high-end chassis reports trays ("Upper Fan Tray"), and on vSRX the command returns next to nothing. No fan block means the check is NOT APPLICABLE - it is never reported as a failed fan. `show system license` has one column nobody has published: what goes in Expiry for a feature with `installed 0`. Juniper's examples all print `permanent` or a countdown, and none of them shows an unlicensed feature, so the parser accepts the column EMPTY as well as a dash and grades the row either way. Requiring it would have dropped the whole row on one of the two shapes, and a feature running unlicensed would then have been reported as OK - a clean pass over something nobody looked at, which is worse than undetermined. LACP and LLDP may not apply either. Without an aggregated Ethernet interface there is no LACP to grade, and with LLDP not configured the command errors out. Both come back as not applicable Why one fault shows up twice A failed fan is two findings, not one: `show chassis environment` reports the fan, and `show chassis alarms` reports the Major alarm the chassis raised about it. Both are graded, and that is deliberate - they are different evidence for the same fault, and a box that showed one without the other would be telling you that something is wrong with the reporting rather than with the fan. The same pairing exists for licences. A feature in use without a licence is a WARNING from `show system license`, and Junos also raises a Minor SYSTEM alarm for it - a different table from the chassis one, which is why the alarms check reads both and labels every row with the table it came from The two checks that are context and never a verdict `show chassis hardware` gives the model and the chassis SERIAL NUMBER, which goes into the device summary and the report header. It is INFO, always: a serial number cannot be a finding. It is collected because every RMA, licence and support case starts with it, and because on a cluster it is the one field that tells the two nodes apart at a glance - a hostname can be anything, the serial is the box. `show lldp neighbors` is the same: a neighbour table, INFO, never graded. It is the fastest way to confirm what is actually on the other end of a port, and the fastest way to notice that what you expected is not there The FPC and PIC checks, which are not high-end only Any SRX with FPCs answers `show chassis fpc detail`, and the state of each slot is a finding in its own right. The one that matters is Present with a hardware note: the chassis can SEE the card and is not using it, because it is in the wrong slot or this Junos does not support it. An operator can spend months believing that capacity is in service. That is CRITICAL, and so is Dead. Dormant, Diag, Probed and Probe-wait are WARNING - held in reset or mid-probe, and none of them should persist. Empty and Offline are not findings at all: a slot can legitimately be empty, and offline is a decision somebody took. `detail` is the same command with more in it. The slot states are graded exactly as before; the temperature, CPU, memory and DRAM it adds are shown in the table and never graded, and a field the platform did not print is a dash and not a zero. `show chassis fpc pic-status` goes one level down, to the PICs. A PIC that is not Online is a WARNING rather than a CRITICAL, for the same reason Offline is not a finding at FPC level: a PIC can have been taken offline on purpose, and calling that "verified as bad" is the conflation this whole script refuses to make. It is still worth saying, because an offline PIC is capacity nobody is using. The two checks that only appear on a high-end chassis On an SRX5400, 5600 or 5800 the Routing Engine can sit idle while the SPUs are saturated, so the Routing Engine and the session summary are not the whole story. Two extra checks read the services cards: SPU memory and flow-table occupancy per FPC and PIC, and SPU utilisation over the last minute. The slots they read are the ones `show chassis fpc detail` reported as Online. On branch and mid-range they are reported as not applicable, because the commands do not exist there - the same treatment the cluster checks get on a standalone SRX. Three honest notes about them. The 80% SPU memory figure is Juniper's own ("If this number exceeds 80 percent, there might be a software problem (memory leak)"), and it is quoted as theirs. SPU CPU is shown and NOT graded, because Juniper publishes no threshold for it and a number invented here would read as a vendor limit - same reasoning as the temperature. And these two checks have never been run against the output of a real high-end chassis: they are written to the field names Juniper publishes, and they report undetermined rather than a number when the layout does not fit. Why a dash is not a zero A counter the script could not find prints as a dash. "0 CRC errors" and "I could not locate the CRC counter" are opposite findings, and reporting the second as the first says a broken link is healthy.
The HTML Report
Every run writes one HTML file, and that file is the deliverable. It is a single self-contained document: no CDN, no web font, no JavaScript. It opens from a laptop with no network, and it prints. Where it goes Into the current directory, named srx-healthcheck-<UTC timestamp>.html unless you pass --report somewhere/else.html. The full path is printed at the end of the run, so you can copy it straight out of the terminal. Add --open to have it opened in your browser when the run finishes, and --no-report if you only want the terminal output. How to open it Double click it, or drag it onto a browser window. On a server with no desktop, copy it off first: scp you@server:/opt/netops/srx-healthcheck-*.html . It is one file. There is nothing else to copy with it. What is in it, in order The report is organised by CLUSTER and not by box, because half of these checks describe the cluster and not a node. The nodes are COLUMNS in every table on the page, so you read both at once and nothing is printed twice. A standalone SRX is a group of one and renders with a single column. It opens with the EQUIPMENT, because the first thing you need is which firewall you are looking at. The heading says what the equipment IS - Cluster SRX, or Standalone SRX for a box that is not clustered - and under it one row per field and one column per node: hostname, model, serial number, Junos, uptime, boot time and last reboot reason. Then the severity table: how many things need action now, how many can be planned, how many could not be checked at all, each with a plain-language label beside the count. The worst severity that needs action - critical, warning or undetermined - is marked in its own colour. A row counting zero is never marked, and neither is OK, INFO or N/A whatever they count: the mark is there so your eye falls where somebody has to act, and a run with nothing to claim marks no row at all. When one run polls more than one cluster the severity table sits above them all instead, because it counts all of them. There is no introductory prose, because nobody reads it. Then Redundancy groups, one row per group with the failover count and what each node reports for it; Findings, ONE ROW PER CHECK with a cell per node - including the cluster checks, whose two cells say the same thing and say it twice - and the worst first; Could not verify, kept deliberately apart from the findings; Every check, the severity of all 22 against each node and nothing else; and Detail. The footer is one line: when the report was generated, the name of this script, PermitANY, and the Junos release these commands were written against. The date is there because an archived report with no date on it is worthless, and the release is there because these commands do change between releases. Detail, and why it is short A detail block exists for a check that came back CRITICAL, WARNING or UNDETERMINED, and for nothing else. The run above produces five of them. There used to be one per check, which on a healthy cluster meant 18 blocks of which 16 were explaining that something was fine - two thirds of the document, describing the part nobody needs to read. What is healthy is answered by its row in Every check. That is the whole answer, and the report is a five-minute read because of it. Each block is the verdict per node, one line of what to do, and the command table behind it folded so that the nodes are columns: one row per fan, per filesystem, per PIC, per interface, with a short state in each node's column. The node that passed keeps its column, because "node0 Failed / node1 OK" is the comparison that makes the row worth reading. What the report deliberately does not repeat The model, serial number, Junos release, uptime, boot time and last reboot reason are printed in Equipment and nowhere else, so the two checks that read them have no block of their own. If either comes back undetermined it still appears, as a finding. A value is printed where it is decided and nowhere else. The equipment table identifies, the findings table judges, the detail explains. A number that helps understand a finding is quoted inside that finding rather than tabulated again. Sending it to somebody It is designed to be forwarded or printed as it stands. There is no page on the internet it depends on, so it will still render correctly in a year, which is the point of attaching it to a change record.
Running It from SecureCRT
The second version on this page is a different script, not a flag on the first one. It runs on the Python interpreter built into SecureCRT, which has no pip, so it uses nothing but the standard library: no netmiko, no PyYAML, nothing to install. The first line of the file, `# $language = "python"`, selects the interpreter that ships INSIDE SecureCRT. That is what makes this version install-free, and it is the one line of the file not to edit. Changing it to "python" does not ask for a newer Python: it asks for an EXTERNAL, third-party interpreter that you would have to install yourself and keep on the PATH, and without it SecureCRT refuses to start the script at all. Written against the SecureCRT 9.x scripting API. Nobody has run it inside SecureCRT for you, so if it does not start, the first entry under When Something Fails is the reason in almost every case. On older releases, use the command-line version instead, which is the same checks and the same report. It does not read an inventory and it never asks for a password - it works on the SRX session you already have open. Use it when you are already logged in, when the firewall is reachable from your jump host but not from your laptop, or when you are not allowed to install Python packages on the machine you are sitting at. Before you run it Have the tab open, connected, logged in, and sitting at the operational mode prompt - the one that ends in >. That is all. The three steps 1. Save the file as srx_healthcheck.py anywhere you like. The usual home is the folder SecureCRT already uses for scripts: Options > Global Options > General > Default Script Directory. 2. Click on the tab of the SRX you want to check, so it is the active session. 3. Scripts > Run Script..., pick the file, and wait. The status bar shows which command it is on. When it finishes, a dialog gives you the verdict, every finding, and the full path of the HTML report. The report goes to your Documents folder; change OUTPUT_DIR at the top of the file to move it. If you run it with no session connected Nothing is sent anywhere. It tells you to connect first and stops. The same happens if the session is in configuration mode - the prompt ends in # - because this script only sends operational commands and will not try to work around a prompt it does not recognise. What this version cannot do It checks one device, the one in front of you - and that is enough for every cluster check, because one node reports both nodes. A lost peer, a down control link, a down fabric link, the redundancy group history and the cluster LED are all graded here exactly as the inventory version grades them. The one thing a single session cannot give you is the CONFIRMATION that both nodes are primary for the same redundancy group at the same moment, which needs the two views side by side. Run it on each node and you get two reports that each flag the signature; for the cross-node comparison, use the inventory version. It also returns no exit code, because a SecureCRT script has nowhere to return one to. Automation is the other version's job.
Exit Codes for Cron and CI
The inventory version returns a code, so it can gate a pipeline or fire an alert without anyone reading the output: 0 every check passed on every device 1 at least one WARNING, or an undetermined check, or a device that could not be reached 2 at least one CRITICAL, or a split-brain 130 you pressed Ctrl-C An undetermined check returns 1 on purpose. It is not a failure of the firewall, but it is a failure of the evidence, and a health check that exits 0 when it could not actually look is worse than no health check. A nightly run that keeps its reports: 0 5 * * * cd /opt/netops && python junos_srx_healthcheck.py \ --inventory devices.yaml \ --report /var/www/srx/$(date +\%F).html
The Inventory File
devices.yaml tells the script WHICH firewalls to poll. It is plain text in YAML format and it lives wherever you point --inventory at it. Credentials do NOT go in it. See Credentials below. devices: - host: 10.10.0.11 # REQUIRED: IP address or DNS name os: junos # REQUIRED: always junos for an SRX - host: srx-branch-07.example.com os: junos Optional fields on any device: device_name: HQ node0 A label for the report, instead of the host cluster: hq Which chassis cluster this node belongs to node: node0 Overrides the node the prompt reports port: 2222 SSH port, if not the default 22 username: admin Overrides NET_USER for this one device password: secret Overrides NET_PASS - avoid, use the environment secret: enablepw Unused on Junos, kept for inventory compatibility The cluster field is the one worth understanding. It groups the two nodes of the same chassis cluster so their views can be compared, which is what finds a split-brain. Without it the script falls back to grouping by the Cluster ID reported by the device - and a Cluster ID is only locally significant, so two unrelated clusters are very often both cluster 1. When the grouping had to be inferred that way, the finding says so, and you should label them. The node field you almost certainly do not need. It overrides the node ID, and the node ID is normally read from the firewall itself. Where the node0 and node1 column headings come from A chassis cluster writes the node that is answering into its own CLI prompt, on the line above it: {primary:node0} admin@srx-dc-node0> That is where the report gets the heading of each column, and it needs nothing from you. The word in braces is the ROLE and it changes at a failover; the nodeN beside it is the fixed node ID, which is the part being read. All six redundancy states are accepted, so a node sitting in hold or ineligible still lands in its own column. If the prompt cannot be read - a customised prompt, a session that does not show the braces - the order is: the `node:` field from the inventory, then the single node banner the device prints over its own output when exactly one appears, and then nothing, in which case the column is headed with the device label. Nothing is guessed at any step: a wrong node label would put the right finding in the wrong column. Set `node:` only to override. If it disagrees with the prompt, the prompt wins - a firewall cannot be wrong about its own identity - and the report says so under the equipment table so the inventory can be corrected. A defaults block applies to every device below it: defaults: os: junos port: 22 devices: - host: 10.10.0.11 # inherits os: junos - host: 10.10.0.12 # inherits os: junos YAML rules that trip people up: * Indentation must be SPACES, never tabs. * The "- " before host marks a new list item. Keep it aligned. * A space after the colon is required. host: 10.10.0.11 correct host:10.10.0.11 WRONG - YAML parse error * Values with special characters need quotes. password: "p@ss: word"
All Valid OS Values
junos Juniper Junos the only value this script accepts This is an SRX tool. The inventory format is shared with every other device-facing script in this catalogue, so the os field is still required and still validated - but any value other than junos is rejected with a message rather than tried and misparsed. ios Cisco IOS rejected by this script iosxe Cisco IOS-XE rejected by this script iosxr Cisco IOS-XR rejected by this script nxos Cisco NX-OS rejected by this script eos Arista EOS rejected by this script panos Palo Alto PAN-OS rejected by this script fortios Fortinet FortiOS rejected by this script A Juniper MX or EX also runs Junos, and the system-level checks would work on one - but the flow, cluster and VPN checks are SRX features, so pointing this at an MX gives you a report that is half undetermined. Use it on SRX.
Credentials
Read from the environment. Never from the inventory file, and never from a command-line argument. Both of those are deliberate. Inventory files get committed to git, and a password in git is a security incident. A command-line argument is visible to every user on the machine in "ps auxww" while the script runs, and it lands in your shell history afterwards. NET_USER username required NET_PASS password required NET_ENABLE enable secret optional, unused on Junos Linux or macOS: export NET_USER='admin' export NET_PASS='your-password' Windows PowerShell: $env:NET_USER = 'admin' $env:NET_PASS = 'your-password' Windows Command Prompt: set NET_USER=admin set NET_PASS=your-password These last only for the current terminal session. Open a new terminal and you must set them again. That is the point - the password never lands on disk. The SecureCRT version asks for nothing, because it uses the session you are already authenticated on.
Every Flag
--inventory PATH REQUIRED. Your devices.yaml. --limit TEXT Only poll hosts whose name contains TEXT. The safe way to widen from one device to many. --report PATH Where to write the HTML report. Default: srx-healthcheck-<UTC timestamp>.html in the current directory. --no-report Terminal output only, no HTML file. --open Open the finished report in your browser. --check-errors Also collect `show interfaces extensive` and grade input, framing, CRC and carrier-transition counters. Off by default because it is slow on a chassis with hundreds of ports. --spu-fpc LIST Which FPC/SPC slots to read on a high-end chassis. Default auto, which takes the slots reporting Online from `show chassis fpc detail`. Pass a comma separated list such as 0,1 to override. Ignored on branch and mid-range, where the command does not apply. --replay DIR Read captured command output from DIR/<host>/<key>.txt instead of connecting to anything. Used to test the parsers offline, and to re-analyse a capture taken during an outage after the device has recovered. --read-timeout SECONDS Per command. Default 90. Raise it on a high-end chassis with --check-errors. --workers N How many devices to poll at once. Default 8. --json-out PATH Also write the same results as JSON, for feeding into something else. The SecureCRT version has no flags. The three things worth changing are constants at the top of that file: OUTPUT_DIR, where the report is written, CHECK_INTERFACE_ERRORS, the equivalent of --check-errors, and SPU_FPC_SLOTS, the equivalent of --spu-fpc, which is "auto" and takes a list such as [0, 1] instead.
When Something Fails
SecureCRT: "Unable to load the Python scripting engine" The first line of the file is asking for an interpreter this SecureCRT does not have. It has to read exactly `# $language = "python"`, which is the one built into SecureCRT. If it reads "python3" it is asking instead for a separate Python installation that has to be on the PATH, and the script never starts at all. Change that line back and run it again. "Set NET_USER and NET_PASS" The credentials are not in this terminal's environment. Usually you opened a new terminal. Set them again - see Credentials. ModuleNotFoundError: No module named 'netmiko' The pip install step was skipped. See Before You Start. If you cannot install packages on this machine, use the SecureCRT version, which needs none. "os is 'ios' - this script is SRX only" Every device in the inventory needs os: junos. This tool only speaks SRX. "authentication failed" Wrong username or password, or the account lacks SSH access. Verify by connecting manually first: ssh admin@10.10.0.11 "unreachable or SSH timed out" The firewall is not answering on the SSH port. Check with: ping 10.10.0.11 ssh -p 22 admin@10.10.0.11 If only some devices time out, raise --read-timeout. If all of them do, it is the path and not the timeout. Half the report says "could not verify" The account can log in but cannot run some commands, or the release does not implement them. Look at the Could not verify section: each row gives the exact command and the exact message the device returned. A `show security ...` command rejected on a device that is not in flow mode is normal; a `show chassis routing-engine` rejected is usually a permissions class that is too narrow. The SPU checks say the slots could not be enumerated `show chassis fpc` did not answer, or answered in a shape this script does not recognise, so there is no slot number to read. There is deliberately no fallback to slot 0: reading a slot with no evidence it exists and calling the result the state of the chassis is worse than saying it is unknown. Either fix the first problem or name the slots yourself: python junos_srx_healthcheck.py --inventory devices.yaml --spu-fpc 0,1 The two SPU checks say "could not parse" on a high-end chassis Juniper documents the field names of `show security monitoring fpc` but not an example of its output, so the column layout this parser expects is unverified. When the shape does not fit, the check reports undetermined rather than a number it is not sure of. Run the command by hand, read it yourself, and send the output along if you want the parser fixed: show security monitoring fpc 0 Worth saying plainly: these two checks have never been seen against real SRX5400/5600/5800 output. They are built from the field names Juniper publishes, which is why they fail to undetermined and not to a number. The cluster checks all say N/A The device answered that chassis cluster is not enabled, so it is a standalone SRX and those three checks do not apply. That is a correct result, not a failure. A split-brain is reported and you know these are two different clusters They were grouped by Cluster ID because the inventory gave no cluster label, and two separate clusters very often share ID 1. Add `cluster: <name>` to each node and run it again. The finding says when the grouping was inferred, for exactly this reason. The SecureCRT version does nothing, or hangs on the first command Almost always the prompt. It reads the prompt off the screen, so a session sitting at a `---(more)---` pager, at a login banner, or in configuration mode gives it nothing to synchronise on. Press Enter in the tab until you see a clean operational prompt ending in >, then run it again. The SecureCRT report is not where you expected It goes to your Documents folder, and the dialog at the end prints the full path. If that folder does not exist it falls back to your home directory. Change OUTPUT_DIR at the top of the file to pick somewhere else.