# Lens 09 — Documentation truthfulness

**Target:** main @ `8598d74` (v0.5.3). **Verdict:** 7 findings survive verification — 1 HIGH, 1 MAJOR, 3 MEDIUM, 2 MINOR. All 7 are **confirmed** (reproduced empirically via `py -c` round-trips or by reading the exact contradicting code/CI file). The v0.5.3 truth-maintenance pass (#300) fixed its three named claims cleanly — none of the below overlap with it. The headline: the project's flagship "Tier 1 auto-translatable" promise (README front page, CAPABILITIES.md, canonical README, 4 vendor pages) includes two surfaces — `timezone` and `syslog_servers` — that **no codec (timezone: 0/12) or only one codec (syslog: junos-only) actually wires**, and the drop on the source side is completely silent (no banner, no Tier-3 detection, verified). Second headline: CAPABILITIES.md §A claims to "enumerate every UnsupportedPath and LossyPath declared today" and is badly stale versus the live matrices, including one row that says "Supported (Wave C)" for a path the matrix declares Lossy.

Checks that came back CLEAN (no finding): README front-page example output (byte-identical, reproduced through `run_plan_with_overrides` incl. `ge-1/0/1` port xlate); README trust-signal numbers (5 CODEC_BUG residual, 8-of-12 expectation coverage, 56 YAMLs, 1224 cells — all match PHASE4_RECONCILIATION.md); the 8 variance classes; demo scenarios (4, keys match); CLI flags (`sanitize -i/-o/-s/--dry-run`, `demo --pair/--list`, `serve`); BUG_REPORTING.md curl examples (multipart form matches route), issue templates + WANTED.md exist; SECURITY.md `|safe`-filter claim (zero hits), `yaml.safe_load` exclusivity, `_FILENAME_RE`, open-in-editor extension whitelist, openapi.json auth exemption, Dockerfile `ENTRYPOINT netcanon serve` + `NETCANON_HOST=0.0.0.0`; `migration.py` docstring endpoint inventory (12/12 match); ARCHITECTURE per-pane/`unsupported_rename_categories` state (matches code exactly); target profiles all declare `max_vlans`; `run_full_mesh --matrix` flag exists; junos vendor page (accurate — claims syslog, which junos really does round-trip, and does NOT claim timezone).

---

### FINDING 1 — HIGH — Tier-1 promise includes `timezone` + `syslog_servers`, which essentially no codec implements; drop is silent on the source side

- **Claim sites:**
  - `README.md:283-289` — "**Tier 1 — auto-translatable.** hostname, interfaces (…), VLANs, static routes, DNS / NTP / syslog servers, timezone. **Every shipped codec parses + renders these fully** (the experimental `cisco_iosxe` NETCONF stub excepted…)"
  - `docs/CAPABILITIES.md:92` — Tier-1 list: "`dns_servers`, `ntp_servers`, `syslog_servers`, `timezone`" (hedged by "per-codec tables under §A are the source of truth" — but §A omits every one of these exceptions, see Finding 2)
  - `netcanon/migration/canonical/README.md:189-193` — "Fields: … `timezone`, `syslog_servers` … **Codecs that don't yet wire a Tier 1 field are bugs, not gaps.**"
  - `docs/vendors/cisco_iosxe.md:27`, `docs/vendors/aruba_aoss.md:23` (even adds "+ DST rule"), `docs/vendors/arista_eos.md:22`, `docs/vendors/fortigate.md:27` — "What translates well … DNS / NTP / syslog servers[, timezone]"
  - `docs/glossary.md:39-40` — "Tier 1 = … DNS/NTP/syslog"
- **Reality (all verified):**
  - `timezone`: the ONLY references to timezone in the entire codec tree are capability-matrix `unsupported` declarations (`grep timezone netcanon/migration/codecs/*/parse.py` → zero hits; every one of the 12 matrices declares `/system/timezone` unsupported, reason "Render emits no clock/timezone stanza; intent.timezone is dropped on migration"). Empirically: `cisco_iosxe_cli`, `arista_eos`, `aruba_aoss`, `fortigate_cli`, `juniper_junos` all parse a config containing their native timezone stanza to `intent.timezone == ''`. **0 of 12 codecs wire it — the canonical field is dead.**
  - `syslog_servers`: 11 of 12 matrices declare `/system/syslog-server` unsupported; empirically cisco/arista/aoss/fortigate parse `logging host …` to `[]`. Only `juniper_junos` round-trips syslog (verified parse+render).
  - `dns_servers` / `ntp_servers`: declared unsupported (and empirically drop) on `cisco_nxos` + `cisco_iosxr` (both verified: `ip name-server`/`ntp server` parse to `[]`), plus `aruba_aoscx` (both), `vyos` (dns), `opnsense` (ntp) — so "every shipped codec" is false for these too since the Phase-1.5 codecs landed.
  - **The drop is silent when the affected codec is the SOURCE:** parsing a cisco config with `clock timezone` + `logging host` yields `dropped_tier3_sections == []` and `raw_sections == []` — no migrate-page banner fires (the walker can't see fields parse never populated; matrix declarations only fire in the target direction). Even the honest newer vendor pages' phrasing "declared `unsupported` so the loss is visible, not silent" (`docs/vendors/cisco_nxos.md:190`) is only true in the target direction.
- **Failure scenario:** operator translates a Cisco access switch to Arista/Junos, reads README "Tier 1 … syslog servers, timezone — every shipped codec parses + renders these fully", pastes a config with `logging host 10.1.1.9` + `clock timezone`; output contains neither, **no banner or review comment appears anywhere**, device ships with no audit logging and wrong clock offset.
- **Fix (docs-first, matching the #292 no-phantom-affordance doctrine):** (a) remove `timezone` + `syslog_servers` from every Tier-1 claim list (README, CAPABILITIES Tier-1 bullet, canonical README, glossary, the 4 older vendor pages' "What translates well" sections) or reword to name only what is actually universal (hostname / interfaces / VLANs / static routes) with DNS/NTP marked "most codecs — see per-codec tables"; (b) either demote the two fields' Tier-1 labels in `intent.py:831-834` / move them below the Tier-1 section comment, or wire them (a much bigger job); (c) optionally add `logging|clock timezone` to the IOS-style Tier-3 detection patterns so the source-side drop at least banners.
- **Confidence:** confirmed (5 codecs reproduced; matrix declarations read for all 12).

### FINDING 2 — MAJOR — CAPABILITIES.md §A claims complete enumeration of lossy/unsupported paths; tables are badly stale, including a "Supported" row the matrix declares Lossy, and three of four hard-coded counts are wrong

- **Claim sites:** `docs/CAPABILITIES.md:181-183` — "The table below enumerates **every** `UnsupportedPath` and `LossyPath` declared today."; per-codec headline counts at `:336` ("46 supported surfaces", cisco_nxos), `:388` ("36", aruba_aoscx), `:415` ("30", vyos); nxos summary row `:30` "**only** IPv6 anycast + Tier-3 remain unsupported"; `:731-732` "This document deliberately omits hard-coded counts (per AGENTS.md hard rule on prose-rot)".
- **Reality (live registry dump, `py` import of all 12 matrices):**
  - `cisco_iosxe_cli`: doc lists 4 lossy + 10 unsupported paths; live matrix has **12 lossy + 12 unsupported**. Missing from the doc: lossy `/interfaces/interface/vrrp-groups/group/mode`, `/routing-instances/instance/instance-type`, `/routing/static-route/description`, `/snmp/v3-user/engine-id`, `/vlans/vlan/description`, `/vlans/vlan/ipv4/address/virtual-gateway-address`, `/vlans/vlan/ipv4/address/virtual-gateway-mac`; unsupported `/system/timezone`, `/system/syslog-server` (Finding 1).
  - **Direct class contradiction:** `docs/CAPABILITIES.md:189` documents `/interfaces/interface/ipv4/address/virtual-gateway-address` on cisco_iosxe_cli as "**Supported** (Wave C)"; the live matrix declares it **Lossy** ("SD-Access anycast … round-trips, but a SEPARATE cross-vendor VARP virtual IP …" — the MTX-3/#280 demotion never reached the doc).
  - `arista_eos`: doc ≈4 lossy/8 unsupported vs live **14 lossy/10 unsupported** (missing e.g. `/vlans/vlan/ipv4/address/secondary-ip`, `/vxlan-vnis/{flood-list,mcast-group}`, `/routing/static-route/{description,metric,interface}`, `/interfaces/interface/voice-vlan`, timezone/syslog).
  - Counts: cisco_nxos live supported = **44** (doc: 46); aruba_aoscx = **34** (doc: 36); vyos = **29** (doc: 30); cisco_iosxr = 22 (doc: 22 ✓).
  - nxos `:30` "only IPv6 anycast + Tier-3 remain unsupported" — false: nxos declares 20 unsupported incl. `/system/dns-server`, `/system/ntp-server`, `/system/timezone`, `/system/syslog-server` (none of which is anycast or Tier-3).
  - `:731-732` self-contradiction: the doc says it deliberately omits hard-coded counts while carrying four of them.
- **Failure scenario:** operator planning an EVPN migration reads the cisco_iosxe_cli table, sees anycast virtual-gateway-address "Supported", relies on it for a Junos→IOS-XE VARP migration; the live matrix (and migrate-page banner) says lossy — doc and product disagree in a way that directly changes a go/no-go decision.
- **Fix:** regenerate §A from the registry (the `/definitions` page already renders live matrix counts — either script the doc tables from the same source or replace the per-path tables with a pointer to `GET /api/v1/migration/adapters/{name}/capabilities` + `/definitions`, keeping only prose reasons). Delete or auto-derive the four "N supported surfaces" headline counts; fix the nxos `:30` "only … remain unsupported" summary.
- **Confidence:** confirmed (registry dumped and diffed).

### FINDING 3 — MEDIUM — README + HOW_WE_TEST claim e2e/desktop tiers "run locally, not in CI" and that CI pytest uses `-x`; ci.yml has run both tiers on every PR (and never uses `-x`)

- **Claim sites:** `README.md:378-382` — "CI runs the **unit + integration** tiers on Python 3.11/3.12/3.13/3.14 against Ubuntu; **the e2e + desktop tiers run locally, not in CI. The CI pytest invocation uses `-x`** (stop at first failure)…"; `docs/HOW_WE_TEST.md:16-18` — "the **E2E** (Playwright) and **desktop** (PySide6) tiers run locally and on demand, not on every PR"; `docs/HOW_WE_TEST.md:42-44` + `:141-142` — "(only the unit + integration tiers gate)".
- **Reality:** `.github/workflows/ci.yml:100-139` defines an unconditional `e2e` job ("E2E (Playwright)", `pytest tests/e2e -m e2e -q --no-header`, with the in-file comment "the **previous** policy ran e2e locally only") and `:141-199` a `desktop` job ("Desktop (PySide6)", `pytest tests/desktop -m desktop`). No pytest invocation in ci.yml passes `-x` (all use `-q --no-header`; the test matrix is `fail-fast: false`). `SECURITY.md`'s release-gate section (`:558-560`) correctly lists `E2E (Playwright)` and `Desktop (PySide6)` among the **required** merge checks — so README/HOW_WE_TEST contradict both the workflow file and SECURITY.md.
- **Failure scenario:** a contributor (or an operator auditing the release gate) reads README/HOW_WE_TEST, concludes UI/desktop regressions can reach main untested and that a red CI run stops at the first failure — both wrong; conversely someone reconciling SECURITY.md's required-checks list against README sees the two "sources of truth" disagree and can't tell which one to trust.
- **Fix:** update `README.md:378-382` to "CI runs unit + integration on 3.11–3.14, plus e2e (Playwright) and desktop (PySide6) jobs on 3.13, all gating every PR" and drop the `-x` sentence; same correction at `docs/HOW_WE_TEST.md:16-18`, `:42-44`, `:141-142` (the cross-mesh-audit-not-in-CI half of `:141` can stay, though the committed-baseline guard `tests/integration/test_cross_mesh_ci_guard.py` deserves a mention).
- **Confidence:** confirmed (read ci.yml).

### FINDING 4 — MEDIUM — VyOS backup definition still documented as provisional / "Not yet validated on live hardware" despite the #113 live-validation graduation

- **Claim sites:** `netcanon/definitions/library/README.md:254-257` — "### VyOS 1.3 / 1.4 — ⚠ **Not yet validated on live hardware** — collection wiring verified in code + against sample output only (Notes column flags it). Confirm on a real router."; `ARCHITECTURE.md:44-47` — "…plus **provisional** definitions for Cisco NX-OS, Cisco IOS-XR, Aruba AOS-CX, **and VyOS**".
- **Reality:** commit `fe4085b` (PR #113, 2026-06-17) graduated the def after a real end-to-end backup of a live VyOS rolling instance: `netcanon/definitions/library/vyos/vyos/1.x.yaml:57` now reads "Backup-collection **LIVE-VALIDATED 2026-06-17**…", `tests/unit/definitions/test_vyos_definition.py:65-69` asserts `"LIVE-VALIDATED" in notes` (and the absence of the old marker), and `docs/CAPABILITIES.md:45-50` was updated in the same commit ("**`CiscoNXOS`, `CiscoIOSXR`, and `ArubaCX`** are provisional … `VyOS` was provisional too until it was live-validated … and graduated"). #113 missed these two files. Bonus inconsistency: the library README's "(Notes column flags it)" is now affirmatively false — the live `/definitions` Notes column shows LIVE-VALIDATED for VyOS.
- **Failure scenario:** an operator deciding whether to trust the VyOS backup def follows ARCHITECTURE's pointer to the library README ("per-vendor authoring notes **and validation status**"), reads "Not yet validated — confirm on a real router", and either skips a perfectly validated def or wastes a lab cycle re-validating; meanwhile CAPABILITIES.md and the live app tell the opposite story.
- **Fix:** in `netcanon/definitions/library/README.md:254-257` replace the warning bullet with the LIVE-VALIDATED 2026-06-17 note (mirroring the YAML); in `ARCHITECTURE.md:45-46` drop VyOS from the provisional list ("…plus provisional definitions for Cisco NX-OS, Cisco IOS-XR, and Aruba AOS-CX").
- **Confidence:** confirmed (git show fe4085b; current YAML + test + both stale files read).

### FINDING 5 — MEDIUM — migrate.html "Validation OK" banner cites SNMPv3 auth/priv algorithms and VRRP priority/preempt as "not yet checked" — exactly the sub-fields #205/#206 wired into the walker

- **Claim site:** `netcanon/templates/migrate.html:1526-1532` (in `renderBannerContents`) — "✓ Validation OK for the checked fields. Every translatable field Netcanon inspects maps to a supported path on the target, but some advanced sub-fields (e.g. **SNMPv3 auth/privacy algorithms, VRRP priority/preempt**) are **not yet checked** — review the output…".
- **Reality:** `netcanon/migration/canonical/xpath_walker.py:283-290` yields `/snmp/v3-user/auth-protocol` / `priv-protocol` (+ passphrases/group/engine-id — PR #205, "walk SNMPv3 USM sub-fields so crypto downgrades stop reporting ok") and `:176-177` yields `/interfaces/interface/vrrp-groups/group/priority` + `/preempt` (PR #206). These xpaths flow into `classify()` and the validation report, so both named examples ARE checked; the copy predates the walk-expansion PRs and was never updated.
- **Failure scenario:** operator gets a green "Validation OK" banner on a migration carrying SNMPv3 users, reads that crypto-algorithm downgrades wouldn't have been flagged, and either performs an unnecessary manual audit or — worse — concludes the validation layer is weaker than it is and stops trusting genuine warnings. The copy also un-advertises the exact hardening the 2026-06 review shipped.
- **Fix:** update the banner string to either drop the parenthetical or cite genuinely-unwalked examples (e.g. VRRP sub-second timers / track-object decrements, per-codec `raw_sections` internals); keep the "review the output; Netcanon has no deploy path" tail.
- **Confidence:** confirmed (walker code read; PRs #205/#206 in history).

### FINDING 6 — MINOR — SECURITY.md not updated for the v0.5.1 security controls (SEC-9 CSP header, SEC-5 `_drain` caps) despite its own "must be updated whenever a security-relevant change is made" contract

- **Claim site / contract:** `SECURITY.md:3-5` ("It must be updated whenever a security-relevant change is made to the codebase") and the "Updating This Document" checklist (`:645-657`).
- **Reality:** `netcanon/main.py:73-106,350-355` sets a `Content-Security-Policy` header on every response (strict same-origin default + scoped `/docs` jsDelivr variant) since v0.5.1 (#290), and the paramiko `_drain` wall-clock/byte caps (#288) hardened the collector — `grep -i "content-security-policy" SECURITY.md` → zero hits; the Known-Limitations resource-exhaustion row describes the semaphore but not the `_drain` caps. Both controls are documented only in CHANGELOG. (Not a false statement — a contract-violating omission; readers of the security model get an incomplete controls inventory, and a future auditor may re-recommend a CSP that already exists.)
- **Fix:** add a short "Browser hardening — Content-Security-Policy" subsection (default policy, the `/docs`-scoped CDN variant + why, `'unsafe-inline'` rationale) and one sentence on the `_drain` caps under the SSH/backup hardening rows.
- **Confidence:** confirmed (header middleware read; SECURITY.md grepped).

### FINDING 7 — MINOR — CHANGELOG v0.5.1 entry asserts "`strip_unsupported` is the mechanism that actually drops the offending paths" — a mechanism v0.5.2 (#292) established never existed

- **Claim site:** `CHANGELOG.md:147-151` (v0.5.1, "API-7 (docs)") — "…a `block` renders anyway and flags the job `partial` — **`strip_unsupported` is the mechanism that actually drops the offending paths**. (#288)".
- **Reality:** the very next release's entry (`CHANGELOG.md:94-96`, v0.5.2 #292) removed the documented `strip_unsupported` affordance as "never implemented", and `tests/unit/migration/test_pipeline.py:129` locks in "there is no `strip_unsupported` mechanism". A reader landing on the 0.5.1 entry (e.g. via `git blame` or a release page) is told an API affordance exists that never did. Keep-a-Changelog treats entries as historical, so a full rewrite is wrong — but an inline erratum is standard practice.
- **Fix:** append a bracketed erratum to the 0.5.1 API-7 bullet: "*[Correction in 0.5.2/#292: `strip_unsupported` was itself never implemented — `force`/`block` semantics as described are the whole behaviour.]*".
- **Confidence:** confirmed (both entries + guard test read).

---

## Notes / non-findings worth recording

- `build/lib/**` still contains the pre-#292 `strip_unsupported` docstrings — `build/` is gitignored (verified `git check-ignore build`), so not a doc defect; ignore.
- `netcanon/api/routes/README.md:38` enumerates ui.py pages as "(`/`, `/configs`, `/migrate`, `/definitions`, `/jobs`, `/sanitize`)" — omits `/devices`, `/schedules`, `/configs/{a}/vs/{b}`, `/docs`. Not framed as exhaustive; NIT only.
- CAPABILITIES.md prose calls the NETCONF codec an "experimental stub" (`:205`) while its table row (`:29`) and the codec (`certainty=best_effort`) say best_effort — "experimental" is a reserved certainty term (red banner) so the wording is sloppy, but the table/UI are correct; NIT.
- ARCHITECTURE.md:145-151 codec-declaration sketch uses pseudo-field names (`vendor`, `format`, `capability_matrix`) that differ from the real ClassVars (`input_format`, `capabilities`/`_CAPS`, vendor bound via matrix `vendor_id`) — clearly illustrative; NIT.
- HOW_WE_TEST.md:141 "the cross-mesh audit is … NOT a CI gate" — technically the audit *run* isn't, but `tests/integration/test_cross_mesh_ci_guard.py` gates the committed baseline in CI; worth a clarifying clause when fixing Finding 3, not a separate finding.
- SECURITY.md sanitiser table omits local-user-name and SNMPv3-user-name redactions that BUG_REPORTING.md documents (both real) — an under-claim; fold into any SECURITY.md refresh.
