Metadata-Version: 2.4
Name: deadwood-audit
Version: 0.1.0
Summary: Ablation auditor for agent systems: finds components that run but never causally matter (dead / redundant / live), with placebo controls and bite-proofs. From AN2B Labs TR-020.
Author: J. DeVere Cooley, AN2B Labs
License: MIT
Project-URL: Repository, https://github.com/jcools1977/an2b-labs
Project-URL: Report, https://github.com/jcools1977/an2b-labs/blob/main/tr020/report/TR020_report.md
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.26

# AN2B Labs Technical Report #020
## Zero-Callers for Cognition: an Ablation Auditor for Agent Systems Is Certified, the Popular Defaults Come Up Clean, and the Cheap Shortcut Is Refuted

**J. DeVere Cooley, AN2B Labs**
**Status: v1.0, published 2026-08-28; ledger at an2b.com/labs**
**Pre-registration: commit `7b7262d`, 2026-08-24, github.com/jcools1977/an2b-labs**

---

## Abstract

Agent systems can contain components that run, produce output, and burn
tokens without ever causally affecting a final answer. TR-020 built an
automated ablation auditor for this failure mode and asked three frozen
questions: can one-at-a-time masking recover known-dead components, what
is the dead fraction in popular real configurations, and can a cheap
textual-influence tracer substitute for full ablation. The answers:
**the auditor is certified** (29/29 components class-exact against a
sealed ground truth it could not read, precision and recall 1.000,
replicated across disjoint probe halves); **the canonical defaults are
clean** (0 of 8 components dead or redundant across LangChain,
LlamaIndex, CrewAI, and AutoGen tutorial configurations, 95% CI
[0, 0.32]); and **the shortcut is refuted** (surrogate kappa 0.29
against a frozen gate of 0.7, with thresholds committed before any
trace existed). The refutation's mechanism is the report's sharpest
finding: dead components in real systems talk about the same things as
their systems, so four of five planted dead components scored textual
influence of 0.99-1.00, statistically level with the twenty live ones.
Topicality is not causality; only intervention tells them apart. The
protocol verdict is the pre-registered **SPLIT**: ablation certified,
surrogate not validated. Everything ran on one 16 GB Mac mini at $0,
and the audit's final verify carries exactly two red legs, each red for
a reason pre-registered before its data existed.

## 1. Question and hypothesis

**H1** (three conjuncts, frozen): systematic component ablation
identifies dead modules with high agreement against ground truth in
seeded systems; real popular agent configurations contain a nontrivial
dead fraction (point estimate reported, any value a finding); and a
cheap textual-influence surrogate predicts full-ablation results well
enough for continuous monitoring (kappa >= 0.7).

**H0:** causal influence is too entangled for one-at-a-time ablation,
or every component in reasonable systems genuinely matters.

The verdict, ruled in advance for this exact configuration of results
(DECISIONS D23): conjunct one holds, conjunct two is answered with a
zero, conjunct three fails. **Split: the auditor stands, the shortcut
falls.**

## 2. Method

Pre-registered thresholds, kill criteria, and negative controls
(root commit `7b7262d`); every implementation judgment in
`tr020/DECISIONS.md` (D1-D23), each timestamped before the numbers it
could have bent toward.

**Taxonomy (D1).** Three verdicts, forced by the benchmark's own
duplicate-tool plant before any system was built: **dead** (no effect
masked alone or jointly), **redundant** (no effect alone, effect when
masked jointly with a partner), **live** (effect alone). The pairwise
sample separates dead from redundant.

**Criterion (frozen).** A component has no effect if masking it changes
under 5% of canonicalized answers AND the judged-quality delta CI
includes zero, over 150 probes per system.

**Seeded benchmark.** Seven systems, 29 components, covering the
planted archetypes: appended-but-never-read, produced-and-ignored,
irrelevant-only retrieval, duplicate tools implemented as consumer
robustness (a reconciler that takes the first parseable value), and an
all-live control. Ground truth is sealed by hash; the auditor package
provably never reads it (grep-enforced). Every trace is byte-
reproducible in isolation (per-item seeding, sabotage-tested), so any
verdict can be replayed by a skeptic.

**Measurement layer, gated red-then-green.** The canonicalizer had to
match 80 human-ratified change/no-change pairs 100%; the judge (Llama
3.1 8B, greedy, disjoint from the actor) had to see three classes of
planted damage by at least 1.0 point on its 10-point scale (measured
separations 2.7-5.3). Ratification is hash-bound: the gate's first
exercise caught three labels that were code artifacts, corrected under
cross-instance adversarial review with PI override (D13, D17, D18).

**Models.** Actor Qwen3-1.7B-4bit, judge Llama-3.1-8B-Instruct-4bit,
both pinned by HF snapshot commit; greedy everywhere; one 16 GB
Apple M4 Mac mini.

## 3. Seeded results: the auditor is certified

**Recovery: 29/29 class-exact. Precision 1.000, recall 1.000.** All
five dead plants found dead, including the irrelevant-retrieval plant
(masking useless-but-present context left span answers stable, the one
question no stub could pre-answer). Both redundant pairs recovered
through the joint-effect signature. Zero flags on the all-live system.
Three-class verdicts identical on both probe halves for every
component. The interaction KILL (more than 10% of live components
falsely flagged) never fired; joint ablations behaved additively where
sampled.

**The placebo control is red, and the red is a finding** (D19,
ratified). Paraphrasing a component's own output was inert on span and
number systems (0.0-2.3% answer change) and structurally non-inert on
text systems (38-96%), with quality deltas that moved in BOTH
directions when a stronger model paraphrased a weaker system's
intermediates. Two methodology results follow: (1) placebo controls
for agent audits must split wording-inertness from quality-inertness,
because greedy generative cascades make the former unsatisfiable for
live components (any upstream rewording changes downstream wording);
(2) **paraphrase provenance moves quality**: an 8B paraphrasing a 1.7B
system's intermediates improved some pipelines and degraded others, so
a placebo paraphraser must be capability-matched or the control
measures the paraphraser. The failure is directionally asymmetric:
cascades can only push away from invariance, so they can inflate
change-rates on live components but cannot manufacture the false deads
that would taint recovery. That is why recovery was perfect through
six wording-brittle systems.

## 4. Wild phase: the canonical defaults are clean

Four configurations, named against pre-committed criteria (popularity,
local-endpoint runnable, distinct architecture families, defaults as
shipped) before any was audited: LangChain ReAct tool agent (145k
stars), LlamaIndex starter RAG (52k), CrewAI researcher-writer crew
(58k), AutoGen planner-executor (61k). Each adapter passed a
**bite-proof** before its audit counted: masking a component the answer
demonstrably depends on must visibly change output, the check must fail
under deliberate interception sabotage, and traces must be
byte-identical across runs. All four passed on their first attempt.

**Dead fraction: 0/8, 95% CI [0, 0.32].** Every retriever, tool,
researcher, writer, planner, and executor is causally live. No system
tripped the unauditability wire (placebo quality arm with a
capability-matched paraphraser and the certified 1.0-point resolution
floor: zero false fires on the seven known-good seeded systems, zero
fires in the wild).

The nuance worth more than the headline: AutoGen's planner showed
answer-change 1.00 under masking with a quality CI including zero.
Total textual influence, quality contribution below certified
resolution. Both arms are therefore reported per component; and the
generalization boundary is stated plainly: **shipped defaults are
curated; the folklore about bloated agent stacks describes accretion**,
the tools and stages teams bolt on after the tutorial. Auditing
accreted production systems is the tool's actual job; this census says
the starting points are lean.

## 5. The shortcut, refuted: topicality is not causality

The surrogate (span matching plus embedding similarity over persisted
traces) ran with every threshold frozen before any trace existed (D23:
3-gram containment >= 0.2, substring for short outputs, bge-small
cosine >= 0.60 as rescue, dead below 5% influence), behind a
determinism certificate (two trace passes byte-identical; one full
masked condition per system recomputed and matching its persisted
change rate exactly, seven of seven).

**Kappa: 0.29 primary (gate 0.7), 0.15 secondary.**

![Figure 1](fig1_surrogate_overlap.png)

*Figure 1. Textual influence rate for all 29 seeded components by
ablation verdict. Four of five dead components sit at 0.99-1.00, level
with the live cluster; only the vocabulary-disjoint plant is
separable. No threshold on this axis distinguishes dead from live.*

The mechanism: the tracer caught exactly one dead component, the
recipes retriever in a geography system, the only plant whose
vocabulary is disjoint from its system's topic. The unread critic
critiques a draft of the same brief; the ignored plan contains the very
words the executor sorts; the unread sentiment score quotes the ticket
the reply answers. **Dead components in real systems talk about the
same things as their systems**, so content similarity cannot separate
provenance from topicality. Because the thresholds were frozen first,
no tuning could have escaped this, and the overlap is complete (three
dead components at exactly 1.000), so no other threshold exists either.
The miss is the measurement's verdict on the method, not on its
calibration. The wild phase had already flagged the same dissociation
from the other side: a planner with total textual influence and
below-resolution quality contribution.

**The invitation:** the seeded benchmark (seven systems, sealed ground
truth, committed probes, byte-replayable traces) is public. Any future
tracer that believes it can predict causal deadness from content now
has a sealed exam to take. The negative result is released as
infrastructure.

## 6. Verdict

Per the pre-registered reading (D23): **SPLIT.** The ablation auditor
is certified for use, its seeded and wild findings stand, and the
cheap continuous-monitoring surrogate is not validated, published here
as its own negative sub-finding. The final verify runs every
instrument leg green and exactly two legs red, each red for a reason
committed before its data existed: the placebo change-arm (D19) and
the surrogate kappa (D23). Nothing was moved to make anything green.

## 7. Limitations

- One actor model class (1.7B) and one judge (8B, 1.0-point certified
  resolution); effects below that floor are invisible by design.
- The wild census covers canonical tutorial defaults (n=8 components,
  interval stated), not accreted production systems, and its
  generalization boundary is exactly there.
- Any content tracer is structurally blind to redundancy (both
  duplicates' content propagates); the secondary kappa quantifies it.
- Wild adapters recompose two frameworks' sequential flows with one
  added interception seam each, logged as plumbing; bite-proofs bound
  the risk.
- LangChain's shipped tutorial tools are cloud SaaS and were
  substituted with local equivalents preserving the tool-calling
  architecture.

## 8. The tool and reproducibility

The harness ships as **`deadwood-audit`** (CLI `deadwood`): the trace
runtime, structure-preserving masking, the three-verdict criterion,
pairwise redundancy separation, and the placebo and bite-proof
machinery, pointed at any system that can expose components through
the trace interface. Public repository: github.com/jcools1977/an2b-labs,
`tr020/`. Pre-registration is the root commit; the decision log runs
D1-D23; every threshold predates every number; per-component
predictions, traces certificates, the sealed benchmark, and `verify.sh`
are committed. Hardware: one Apple M4 Mac mini, 16 GB. Incremental
cost: $0. Wall-clock: roughly three attended days, including one
overnight seeded run and one morning of wild audits.
