Iteration 04 — P0 shipped — the novelty gate is now runnable, tested code DONE

Built the three-tier novelty dedup engine (the fast catalogue-uniqueness pre-filter) as a stdlib-only module and verified it against the real 44-candidate catalog — 8/8 checks pass.

✅ What has been grounded

The novelty gate now EXISTS as code, not just a plan. findings/evolution/shared_data/novelty_check.py answers 'is this candidate already in our catalog?' and returns one of three verdicts: covered / unique (=novel) / needs_review. It reads the live ledger + knob inventory as the catalog.

Three tiers, cheapest-first (fast by design). Tier 1 — exact: a content hash of the candidate's definition (name+family+knob-keys) and a normalized name/alias index → an exact or renamed-case duplicate is caught in an O(1) lookup. Tier 2 — lexical: only the same-family / token-overlapping rows are compared (no all-pairs blow-up); a score ≥95 = covered, 85–95 = needs_review (gray zone, never silently merged), <85 falls through. Tier 3 — semantic: gated off here (no offline model).

Verified on the real catalog — 8/8 self-tests pass. Exact name → covered · case/space variant → covered · production alias bar_petrosian_fd → covered (cand-0043) · a made-up feature → unique · exact Asymmetric MF-DFA H+ → covered · near-duplicate → covered/needs_review · and H+ never collapses into H- (the distinctness rule holds).

An honest limitation surfaced — and is documented, not hidden. An abbreviation synonym (A-MF-DFA ≡ Asymmetric MF-DFA) scores only 66.7% on the lexical tier, so without the gated semantic Tier-3 it returns unique. This is correct behaviour for a lexical matcher; closing it needs an offline embedding model (a later, optional add). The verdict always reports tier3_semantic: disabled so nobody mistakes it for a full semantic check.

Dependency-safe — runs anywhere. rapidfuzz/jellyfish/jsonschema are absent in the environment, so the engine is stdlib-only (hashlib, difflib, unicodedata) and uses the better libraries automatically only if they are ever installed. No environment setup, no external services.

➡️ What I am doing next

Wire novelty_check into candidate_ledger.py so every --add/--preflight auto-runs the gate and records the dedup.{status,mechanism,score,matched_ids} provenance on the row.

Implement D2 (append-only writes + file lock + compare-and-swap + atomic rename) and the one-time migrate() of the 44 rows.

Then P1 (lifecycle transition table + opaque spec-hash ids) and P2 (input-lock verdict pinning + hash-chained history), and the /registry:doctor + /registry:check-full commands.

Optional: add a small offline embedding model to enable the semantic Tier-3 (catches abbreviation synonyms like the A-MF-DFA case).

🎯 How this moves us toward the end goal

This is the operator's #1 ask made real: a fast, correct novelty pre-filter. It runs in front of the expensive orthogonality test and decides — cheaply, with evidence — which candidates are genuinely new to the catalog and therefore worth the bigblack compute. The three-way verdict plus the never-silently-merge review band means the loop won't double-evaluate (the stated pain) and won't wrongly discard a borderline case. It is tested, deterministic, and dependency-free — so tomorrow's discovery loop can lean on it immediately.

Verified behaviour (8/8 self-tests on the live 44-row catalog)

input	verdict	how
`Permutation Entropy` (exact)	covered	name_exact → cand-0004
`permutation entropy` (case/space)	covered	normalized name → cand-0004
`bar_petrosian_fd` (alias)	covered	alias index → cand-0043
`Asymmetric MF-DFA H+`	covered	name_exact → cand-0029 (NOT H-/cand-0030)
`Higuchi fractal dimension`	covered	exact/near → cand-0045
`Quantum Recurrence Spectral Flux 9000`	unique	all tiers clear → novel
`A-MF-DFA H+` (abbreviation)	unique	lexical 66.7% < band — known semantic-tier gap (honest)

Try it: python3 findings/evolution/shared_data/novelty_check.py --check "Your Feature" --family mfdfa · self-test: --self-test.

← Iter 03 Iter 05 →

Campaign spoke · APPEND-ONLY loop record · regenerate via build.py · part of the Three-Axis Probes dashboard · registry SSoT: candidate_ledger.jsonl