SN Provenance Ledger | imas-standard-names

✓ SHIPPED 2026-07-09 The graph is now a robust provenance ledger.

All five workstreams landed in imas-codex (main). The one-time rebuild ran on production: orphans 1818→0, edge/scalar desyncs 254→0, content preserved (live 2753 / accepted 2446 unchanged). Every non-superseded StandardName now traces to ≥1 StandardNameSource (dd 8740 · derived 541 · signals 12 · manual 22). The provenance-dropping bulk import is removed; sn merge (id-matched, review-only PR acceptance) + a diff-by-id catalog_reconcile replace it; a per-run reattach + orphan-count invariant keeps the ledger from drifting again.

Full landed record + outcomes table + figure: § landed. Follow-ons: the 22 manual names are NOT anchorless — they were hand-added to the catalog and brought in by the (removed) bulk import with no source; their DD leaves exist, so recovery is deterministic — tracked in sn-pedestal-source-recovery (f-spl-004 resolved). Website source de-noising remains (f-spl-003).

§1 — Problem & method

Invariant we want: the graph is the ledger — every non-superseded StandardName traces to one, or a group of, StandardNameSource(s). Sources need not be DD leaves: a source may be a DD path, a derived-from-parent group, a signal, or a hand-authored/manual origin.

Observed reality: of 2753 live names, only 595 carry a direct DD source; 572 are fully orphaned (no source, no parent) and 1404 are only structurally linked (HAS_PARENT) with no leaf-source. Every SN was ultimately DD-driven at first generation, so this is provenance loss, not names that were never sourced. Root-caused by systematic debugging (evidence → mechanism → code confirmation).

Goal (locked): a rebuild that recovers 100% provenance — the final graph's link topology (StandardNameSource nodes + PRODUCED_NAME + FROM_DD_PATH + HAS_PARENT) must be identical to a fresh, from-scratch rebuild — without regenerating names or docs (that content is preserved and its cost saved). This is possible because the link topology is deterministic: dd-sources come from the DD graph and derived/parent structure from the ISN grammar decomposition — both re-derivable with no LLM. The recovery must not lose the latest edits (this session's renames, docs, stages): it rebuilds links, preserves content.

The catalog YAML is a clean public artifact (no pipeline internals). Bootstrapping / restoring the graph from it (step 3) is what makes the ledger lossy — names re-enter as origin=catalog_edit with no StandardNameSource. Provenance survives only for names generated fresh in the current graph's lifetime.

§2 — Evidence

Linkage of 2753 live names (a name may hold more than one relation; buckets are exclusive by precedence dd>derived-source>parent>none):

bucket	count	meaning
directly DD-linked	595	`StandardNameSource(dd)` → `IMASNode`
derived (HAS_PARENT only)	1404	only 118 trace to a DD-linked ancestor; 1286 root at conceptual parents
other-sourced	182	derived-source / non-dd, no parent
orphaned (no edge at all)	572	no source, no parent — the acute failure

Origin × linkage — the loss concentrates in catalog_edit (imported):

origin	n	dd-src	other-src	HAS_PARENT	NO edge	source_paths scalar
catalog_edit	2106	232	78	1343	559	55
(null)	198	192	0	14	6	190
pipeline	172	168	0	35	3	14
derived	277	3	270	137	4	2

Freshly-generated names (null/pipeline) are ~98% DD-sourced. Imported names (catalog_edit) are the loss: 559 orphaned, 1343 parent-only.

Source-side health: 9102 StandardNameSource nodes, 6549 dangling (no PRODUCED_NAME) — mostly un-composed DD leaves, but the ledger cannot distinguish "never composed" from "produced a name whose edge was lost." 41 orphans carry a source whose produced_sn_id scalar names them but the PRODUCED_NAME edge is missing (edge/scalar desync). Only 15 orphan ids match a DD leaf by name (most orphans are grammar-composed/aggregate, not 1:1 DD names).

§3 — Root causes (code-confirmed)

RC-1 — Export→import round-trip severs provenance (primary). export.py:48,551 defines _PROVENANCE_FIELDS = {source_paths, dd_paths} and deliberately does not emit them ("we don't emit pipeline provenance"). catalog_import.py:31 forbids those keys and the importer (:413) MERGEs StandardName + HAS_UNIT + structural edges (HAS_PARENT/HAS_ERROR/HAS_PREDECESSOR) but never a StandardNameSource/PRODUCED_NAME. So the public catalog YAML is provenance-free by design, and rebuilding the graph from it strands every imported name (origin=catalog_edit) without a source. Rename is not a cause — its cascade carries pn_sources (graph_ops.py:9238); reset only flips source status='extracted', never deletes edges.

RC-2 — Edge/scalar provenance desync. Some name-creating/mutating ops set (or leave) a source's produced_sn_id scalar while the PRODUCED_NAME edge is absent — 41 currently-reattachable orphans. The graph stores provenance in two places (edge + scalar) that can diverge; reconcile_provenance (loop B2c) clears stale scalars but does not reattach a live scalar's missing edge.

RC-3 — No ledger invariant / integrity gate. Nothing asserts "every non-superseded SN has ≥1 PRODUCED_NAME source." Provenance loss is silent and accumulates across import/reset/merge cycles; the audit and export gates do not flag orphaned names.

§4 — The ledger model we will enforce

Every non-superseded StandardName MUST resolve to ≥1 StandardNameSource. Source kinds (none privileged; a name may cite a group):

dd — a DD leaf (the extract anchor). Directly-extracted names.
derived — a parent/group of SNs (aggregation, charge-state roll-up, transform). The "source" is the composed-from set, not a DD leaf.
signal — a facility/diagnostic signal.
manual — hand-authored with no upstream anchor (explicit, auditable, not an accident).

The graph is authoritative; the published catalog is a complete, restorable projection of it (full sources), consumed by a reconciler — never a recreating import.

The graph is the single authoritative ledger. Deterministic extract/derive (top-left) is what a fresh build runs; the one-time WS1 rebuild replays it against existing names, binding provenance from a2f8831 + grammar (no LLM). The catalog is a complete projection: a PR is the review surface, sn merge folds reviewed edits back (review-only, no refine), and a reconciler — never a recreating import — can restore from it.

§5 — Recovery base

The exported artifact did carry near-complete provenance until three days ago. ISNC commit a2f8831 (2026-07-03) holds 1941 sources blocks + 5030 dd_path refs for 2117 names (~92%); by rc61 (8fba45f, 07-06) it had collapsed to 292, and rc62 to 338 — the erosion tracks the import/export round-trips run since. a2f8831 is therefore the authoritative recovery map for the bulk; the DD graph and the ISN grammar close the remaining gap deterministically.

✓ landed 2026-07-09

§6 — Workstreams (all landed)

All five shipped in imas-codex on main. Full detail + commit SHAs: § landed record.

WS1 — provenance rebuild to fresh-parity. provenance_rebuild.py replays the deterministic fresh-build (reattach → rederive structure → seed_parent_sources → relink FROM_DD_PATH) then binds residue by anchor authority: a2f8831 map (dd) → surviving source_paths scalar (dd) → derived parent (composed-from-children) → manual. Ran on production: orphans 1818→0, content preserved. Fresh-parity is by construction (same fixpoint routines a fresh build runs) + idempotent re-run + empty orphan set.
WS5 — ledger invariant + reattach + gate. ledger.py invariant queries; reconcile_provenance reattach pass (desyncs 254→0); persist_refined_name scalar-sync; sn run reports orphan count each cycle; graph-marked read-only invariant tests (0 orphans / 0 desyncs) enforce via release Gate A.
WS3/WS4 — restorable catalog. export emits the full sources projection (lossless); new catalog_reconcile restores by id (no node recreation). The provenance-dropping bulk sn import was removed completely (no backwards compat).
WS2 — sn merge. id-matched PR acceptance: attach like sn edit, full review with NO refine, ≥θ accept else validation_status='quarantined'; rename via the cascade (carries PRODUCED_NAME). CLI verb wired.

Open residue: 22 anchorless manual names (f-spl-004, optional RC-snapshot recovery) and the website source de-noising (f-spl-003).

§ Decisions

How do we persist ledger provenance across publish/restore (WS1/WS4)?

LOCKED (Simon McIntosh, 2026-07-09): Option C. The graph is the authoritative ledger for the live workflow; every release still emits the full sources field so the catalog is a complete, restorable snapshot — but restore uses a reconciler (diff-by-id, no node recreation), never the recreating import.

What is the recovery target?

LOCKED (Simon McIntosh, 2026-07-09): a rebuild recovering 100% provenance whose final link set (StandardNameSource + PRODUCED_NAME + FROM_DD_PATH + HAS_PARENT) is identical to a costly fresh rebuild — but WITHOUT regenerating names/docs (content preserved, cost saved) and WITHOUT losing the latest edits. Achievable because the link topology is deterministic (DD graph + grammar), proven by diffing against a reference deterministic build.

What is the PR-acceptance command?

LOCKED (Simon McIntosh, 2026-07-09): sn merge . It merges a reviewed catalog PR back into the graph-ledger, id-matched, provenance-preserving.

How are human PR edits processed on merge?

LOCKED (Simon McIntosh, 2026-07-09): human edits attach exactly like the sn edit tool and pass through the FULL review pipeline, but with NO refine step — a human-reviewed name must not be silently mutated afterwards (we don't want externally-reviewed names looking reviewed and then drifting). Below-threshold edits are quarantined + flagged for human attention, never auto-refined.

How is a source inferred when reconstructing (WS1 residue)?

LOCKED (Simon McIntosh, 2026-07-09): conservative — the rebuild replays the deterministic extract/derive (DD graph + grammar), so dd/derived sources are recovered wherever a fresh build would create them; anything with no deterministic anchor becomes an explicit source_type='manual' source. Never invent a DD path.

§ Followups

Lock WS1/WS5 decisions, then repair the ledger (reattach → invariant → backfill → round-trip fix)

RCA landed (§3): the export→import round-trip severs provenance (RC-1), 41 names have edge/scalar desync (RC-2), and no invariant guards it (RC-3). Two decisions are open and gate the durable fix: provenance-persistence (sidecar vs graph-as-ledger) and orphan-reconstruction (conservative vs grammar-infer). Immediate, decision-free steps: WS3 (reattach the 41 produced_sn_id→missing-edge desyncs via reconcile_provenance) and WS4 (add the ledger invariant + audit/export gate: every live SN has ≥1 source). WS1/WS2/WS5 (round-trip persistence, import reconstruction, the 572+1404 backfill) follow the locked decisions.

Project: imas-codex (graph_ops / catalog_import / export / loop) + imas-standard-names-catalog
Plan: sn-provenance-ledger (http://localhost:8765/imas-standard-names/sn-provenance-ledger.html)
Section: WS3/WS4 first (decision-free), then WS1/WS2/WS5
Tier: opus

Context
  The graph must be a robust ledger: every non-superseded StandardName resolves to >=1 StandardNameSource (dd / derived / signal / manual — sources need not be DD leaves). RCA §3: export strips source_paths/dd_paths (export.py:48,551); import forbids them + creates no source (catalog_import.py:31,413); 41 names have a produced_sn_id scalar but no PRODUCED_NAME edge; no invariant guards any of this.

State to read (CODE / FILES / DATA)
  imas_codex/standard_names/export.py (_PROVENANCE_FIELDS, ~549); catalog_import.py (MERGE at 413, forbidden keys at 31, structural edges at 473); graph_ops.py reconcile_provenance + PRODUCED_NAME MERGEs (2188/3940/5751/9238); loop.py B2c reconcile block.
  Graph: the 41 reattachable desyncs (source.produced_sn_id names a live SN, no PRODUCED_NAME edge); 572 orphans + 1404 parent-only.

Scope locks / constraints
  - Honour the locked provenance-persistence + orphan-reconstruction decisions before WS1/WS5.
  - Reattach (WS3) and invariant (WS4) need no decision — do them first.
  - Never silently invent a dd-source: attach dd/derived only where derivable; residue is explicit source_type='manual'.
  - The published catalog stays provenance-free (public artifact); provenance lives in the graph (+ the WS1 channel).

Done-when
  1. WS3: 0 edge/scalar desyncs (reconcile reattaches them).
  2. WS4: invariant added; orphan count reported by sn run reconcile + a doc/audit gate; regression-guarded.
  3. WS1/WS2/WS5 implemented per the locked decisions; every live SN traces to >=1 source; round-trip is lossless.
  4. This followup resolved.

Superseded — all decisions now locked (provenance-persistence=Option C, recovery-scope=rebuild-fresh-parity, acceptance-command=sn merge, human-edit-handling=review-only+quarantine, orphan-reconstruction=conservative) and the workstreams detailed (§6). See f-spl-002.

Build: provenance rebuild to fresh-parity (WS1) → invariant (WS5) → retire import + reconciler backstop (WS3/WS4) → sn merge (WS2)

All decisions locked. Recommended order: (1) WS1 — the one-time provenance rebuild that recovers 100% and reaches link-parity with a fresh build, content-preserving (biggest single win); (2) WS5 — add the ledger invariant + reattach the 41 edge/scalar desyncs so loss can't recur; (3) WS3+WS4 — retire the recreating bulk import and make release emit full sources consumed by a diff-by-id reconciler; (4) WS2 — the sn merge acceptance workflow (attach human PR edits like sn edit → full review, NO refine → accept ≥θ else quarantine+flag). WS1 must prove fresh-parity by diffing its link topology against a reference deterministic build in a scratch DB before it is accepted.

Project: imas-codex (graph_ops / build_dd / catalog_import / export / loop / a new sn merge) + imas-standard-names-catalog
Plan: sn-provenance-ledger (http://localhost:8765/imas-standard-names/sn-provenance-ledger.html)
Section: WS1 → WS5 → WS3/WS4 → WS2
Tier: opus

Context
  Make the graph a robust ledger: every non-superseded SN traces to >=1 StandardNameSource (dd / derived / signal / manual). Honour the live locked decisions above. The recovery target is a rebuild whose link topology equals a fresh from-scratch build, WITHOUT regenerating names/docs and WITHOUT losing the latest edits.

State to read (CODE / FILES / DATA)
  ISNC commit a2f8831 (2026-07-03): standard_names/*.yml `sources` blocks (1941) + dd_path (5030) — the recovery map. Compare to rc62 (cfa32ad) for what eroded.
  imas_codex/graph/build_dd.py (deterministic DD extract), standard_names/graph_ops.py (PRODUCED_NAME/HAS_PARENT MERGEs 2188/3940/5751/9238; reconcile_provenance), workers.py (derive), export.py (_fetch_sources_for_entry ~682; include_sources ~1097/1321; _PROVENANCE_FIELDS 48), catalog_import.py (413/31/473), edit.py (apply_edit; the review-only path), loop.py (B2c reconcile).

Scope locks / constraints
  - WS1 is provenance-ONLY: rebuild StandardNameSource + PRODUCED_NAME + FROM_DD_PATH + HAS_PARENT by replaying the deterministic extract/derive against EXISTING names (bind by id from a2f8831 + grammar + DD); never touch name/description/docs/stage. Residue with no deterministic anchor -> explicit source_type='manual'. Prove fresh-parity by diffing the link set against a reference deterministic build in a scratch DB (empty diff to accept).
  - WS2 sn merge: human PR edits attach exactly like sn edit (change as candidate + reason), FULL review, NO refine; >=threshold accept (-> human_approved), else quarantine+flag. Name edits via the rename cascade (carries pn_sources), never delete/recreate.
  - WS3: retire the recreating bulk import as the restore path. WS4: release always emits full sources; restore via a diff-by-id reconciler (no node recreation).
  - WS5: assert every live SN has >=1 source; reattach the 41 live-scalar/missing-edge desyncs; wire into sn run reconcile + release/audit gates.
  - Do NOT tackle the website true-vs-derived source de-noising (future, f-spl-003).

Done-when
  1. WS1: 100% of live names carry >=1 source; link topology diff vs a fresh deterministic build is empty; content unchanged; latest edits intact.
  2. WS5: 0 orphans, 0 edge/scalar desyncs; invariant gate live.
  3. WS3/WS4: bulk import retired; release carries full sources; reconciler restore is lossless.
  4. WS2: sn merge implemented + tested (accept path + quarantine path; no-refine verified).
  5. Followups resolved.

SHIPPED (2026-07-09). All workstreams landed in imas-codex on main. WS1 (013849db, 91e83a1a): provenance_rebuild.py replays the deterministic fresh-build (reattach -> rederive structure -> seed_parent_sources -> relink FROM_DD_PATH) then binds residue by anchor authority (a2f8831 map -> source_paths scalar -> derived parent -> manual). Ran on production: orphans 1818->0, edge/scalar desyncs 254->0, content preserved (live 2753 / accepted 2446 unchanged). 95 initial 'manual' corrected to 67 derived + 6 dd + 22 genuine residue. WS5 (f3edb899): ledger.py invariants + reattach pass in reconcile_provenance + persist_refined_name scalar-sync + per-run orphan report + graph-marked invariant tests (Gate A). WS3/WS4 (2255b7ff, 67a0a93a): export emits full sources projection; catalog_reconcile diff-by-id restore. Bulk import removed completely (4b949931). WS2 (c344e4c2): sn merge — id-matched PR acceptance, attach like sn edit, full review NO refine, accept>=theta else quarantine; CLI verb wired. 'manual' StandardNameSource type declared (b3d541dc). Full record: archive/sn-provenance-ledger-landed.html. Residual: f-spl-004 (22 anchorless names), f-spl-003 (website de-noise).

Future: de-noise the website per-entry source display (true sources vs derived)

The ISNC website shows a per-entry source list that conflates the ISN grammar decomposition (always derivable, noisy) with true pipeline sources. Group TRUE sources (dd / signal) apart from DERIVED (composed-from / grammatical), so the display is meaningful. Flagged by the lead for a future session; not in scope of the rebuild/merge work.

Project: imas-standard-names (SPA / entry rendering) + imas-codex (export sources field)
Plan: sn-provenance-ledger
Section: website source display
Tier: sonnet

Context
  Per-entry 'sources' on the ISNC website mix grammatical decomposition with true pipeline provenance, creating noise. Separate true sources (dd / signal) from derived (composed-from).

State to read (CODE / FILES / DATA)
  The SPA entry renderer; export.py _fetch_sources_for_entry (source_type on each source); the grammar decomposition path.

Scope locks / constraints
  - Presentation/grouping only; do not change the ledger model.

Done-when
  1. Website groups true vs derived sources clearly.
  2. This followup resolved.

Optional: recover the 22 anchorless 'manual' names from a prior RC graph snapshot

After the WS1 rebuild, 22 live names (0.8%) carry source_type='manual' — deeply-composed leaves (mostly toroidal__velocity_at_pedestal_top, plus a few flux/energy leaves) whose grammatical parent is not a live SN and whose dd anchor is lost from every deterministic source (absent from all ISNC commits incl. the a2f8831 high-water mark, no surviving graph anchor, grammar->DD matching unreliable). 'manual' is the honest, auditable classification. Optional authoritative recovery: a prior full-graph RC release snapshot (on the fork) where their PRODUCED_NAME + dd source were intact — load into a second Neo4j, diff by id, rebind. Alternatively regenerate via the pipeline (LLM; changes content). Decide per-name value vs cost.

Project: imas-codex (standard_names) + a prior RC graph snapshot
Plan: sn-provenance-ledger
Section: 22 anchorless manual names
Tier: sonnet

Context
  22 live names carry source_type='manual' (no deterministic dd/derived anchor). List: MATCH (s:StandardNameSource {source_type:'manual'})-[:PRODUCED_NAME]->(n:StandardName) RETURN n.id. They are genuine residue; recovery is optional.

State to read (CODE / FILES / DATA)
  imas_codex/standard_names/ledger.py (find_provenance_orphans), provenance_rebuild.py (bind_recovery_sources). A prior RC full-graph snapshot from the fork (GHCR) where these names had intact PRODUCED_NAME + dd sources.

Scope locks / constraints
  - Never fabricate a DD path. Only bind a dd source recovered from an authoritative snapshot (or regenerate via the pipeline if the lead accepts content change).
  - Load the snapshot into a SECOND Neo4j (GraphClient with explicit uri), diff by id, rebind via bind_recovery_sources; never touch the active graph's node content.

Done-when
  1. The 22 names either carry a recovered dd source (from snapshot) or are confirmed genuinely anchorless and left as manual with the lead's sign-off.
  2. This followup resolved.

Superseded by the sn-pedestal-source-recovery plan. Investigation found the 22 are NOT anchorless: they were hand-added to the catalog and brought in by the (now-removed) bulk import with no StandardNameSource; their DD leaves EXIST (the 16 pedestal velocities → summary/local/pedestal/velocity_phi//value). Recovery is deterministic — extract the leaf as a real StandardNameSource(dd) and attach it — no RC-snapshot needed. sn edit/merge/pipeline cannot create sourceless names, so the fault is already closed by the import removal. Tracked in docs/sn-pedestal-source-recovery.html.