Metadata-Version: 2.4
Name: detectorproof
Version: 0.1.0
Summary: Measure the detectors before trusting them: a declared panel of frozen synthetic-speech detectors, scored on your control set, with a chance-level detector refused rather than averaged in.
Author-email: "Panagiotis (Panos) Gkilis" <bedvibe@bedvibe.studio>
License: MIT
Project-URL: Homepage, https://ai.bedvibe.studio/
Keywords: audio-deepfake-detection,synthetic-speech-detection,detector-robustness,false-positives,neural-audio-codec,analysis-resynthesis,pre-registration,reproducibility,auc,multiple-comparisons
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Multimedia :: Sound/Audio :: Analysis
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Development Status :: 3 - Alpha
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: ruff>=0.16; extra == "dev"
Requires-Dist: mypy>=2.3; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=6; extra == "dev"
Dynamic: license-file

# detectorproof

**Measure the detectors before you trust any of them.**

`detectorproof` is not a synthetic-speech detector and must not become one. It is an
instrument for measuring detectors: what a declared panel of frozen checkpoints can and
cannot do on *your* material, and what their judgments are reacting to.

One command is built: `panel`. It scores a declared set of frozen detectors over a labelled
control set, reports the area under the ROC curve for each, and refuses to let an
incompetent one contribute to anything downstream.

No audio is read and no model is loaded. It works on a score table you already have, has no
dependencies, and runs on a machine that will never hold the corpus.

```
pip install detectorproof
```

Or from a clone, which is what you want if you intend to run the tests:

```
git clone https://github.com/Mormolykos/detectorproof.git
cd detectorproof
pip install .          # or: pip install -e ".[dev]" && pytest
```

Python 3.10 or newer, no runtime dependencies. Both forms install the `detectorproof`
command and the importable package. See `CHANGELOG.md` for release status.

## Why this exists

On one corpus, three widely benchmarked anti-spoofing detectors sat at chance while their
model cards advertised sub-1% equal error rates. A panel that quietly averaged them in would
have reported a confident number built partly on models that could not do the task at all.

Separately, and this is the finding the package is built around: **genuine human speech can
move strongly toward "synthetic" after ordinary benign processing that generates nothing.**
A neural-codec round trip of real studio recordings moved 9 of 13 frozen detectors past
multiple-comparison correction, three of them on 47 of 47 utterances at the maximum a rank
statistic can reach. A phase-retrieval algorithm from 1984 moved 12 of 13.

Those numbers are not asserted here. They come from a published, independently audited
study, and every one of them regenerates from its deposit:

> Gkilis, P. (2026). *Reconstruction history, not synthesis: benign processing moves frozen
> detectors on genuine human speech.* Zenodo.
> [10.5281/zenodo.22819223](https://doi.org/10.5281/zenodo.22819223) · CC BY 4.0

⚠️ Read its scope limits before quoting it: three speakers, 47 utterances, neutral speech,
one language; seven of the thirteen detectors come from a single research group; **no
mechanism is identified and no listening test was run.** Thirteen of that paper's own claims
were withdrawn during three rounds of independent adversarial audit, and the withdrawals are
published in it.

So a detector score is not interpretable without knowing the provenance of the audio it was
computed on — and before any of that, you need to know which detectors on your panel can do
the task on your material at all. That is what `panel` answers.

## Usage

```
detectorproof panel \
  --panel panel.json \
  --scores scores.csv \
  --labels labels.csv \
  --condition CTRL
```

**`panel.json`** — the declaration, fixed before the run.

```json
[
  {"name": "det-a", "orientation": "higher_bonafide", "identity": "sha256:1f3a...", "source": "..."},
  {"name": "det-b", "orientation": "higher_synthetic", "identity": "rev:c1dbe2ae"},
  {"name": "det-c", "orientation": "unresolved", "identity": "commit:72c32e28"}
]
```

`orientation` is resolved from the model's own documentation or training-label path, never
from what makes your benchmark look right. `unresolved` is a first-class value: the detector
is scored and reported, and its *sign* is not interpreted.

`identity` pins the weights — a revision, a commit, a file hash. It is required. A panel that
cannot say which checkpoint produced a number is not frozen.

**`scores.csv`** — `detector,item,condition,score`. One row per detector per item per
condition. Scores are in each detector's native units. A missing measurement is written
`NOT_MEASURED` (or `n/a`), never `0`.

**`labels.csv`** — `item,label`, where label is `bonafide` or `synthetic`.

In Python:

```python
from detectorproof import Detector, run_panel, render

report = run_panel(detectors, scores, labels, condition="CTRL")
print(render(report))

for result in report.competent_detectors():   # the only set you may aggregate over
    ...
```

## The refusals

These are enforced in code, not described here. Each was bought by a specific measured
failure.

| refusal | the failure that bought it |
|---|---|
| A detector at or near chance is **NOT COMPETENT ON THIS MATERIAL** — reported in full, excluded from every aggregate | three benchmarked detectors at chance while advertising sub-1% EER |
| The panel is **declared before the run**. An undeclared detector in the scores is a fault; a declared detector with no scores stays in the report as NOT MEASURED | a panel assembled after seeing results is a selection, not a measurement |
| Two scores for one detector and item are a **fault**, not something to average | overlapping analysis windows treated as independent samples inflated one published proportion from 75% to 86% |
| **NOT MEASURED is a third state** — never zero, never an absence of effect | a missing measurement reported as zero is worse than reporting nothing |
| A near miss at the band edge is **recorded**, not resolved by widening the band | one detector landed at 0.447 against a declared band of [0.45, 0.55] and the band stayed where it was |

There is deliberately **no function that recommends a transformation** to move a score, and
there will not be one. In the work this came from, an intervention tuned against a chosen
detector statistic made the audio *more* separable, not less — one detector's AUC moved from
0.576 to 0.763. The instrument reports; it does not advise. That refusal is the point, not an
omission.

## What a verdict means

`NOT COMPETENT ON THIS MATERIAL` is a statement about **that checkpoint on your control
set**. It is never a statement about a detector's general quality. Every detector was trained
by someone else, on other data, for other purposes.

An empty competent set is a reportable result about your panel and your material. It is
**not** evidence that your audio is genuine, and it is not evidence that detection is
impossible. Absence of evidence is not evidence of absence, and nothing downstream may run on
an empty competent set.

## Design

Zero dependencies in the core. Ties are handled by midranks everywhere, because a detector
whose scores compress near zero produces long runs of equal values and a rank statistic that
breaks ties arbitrarily reports a different answer depending on the order its input arrived
in. AUC is computed from the rank sum rather than by counting pairs.

Every field this package reads has a **declared domain** — its type, its admissible values,
and what absence means. Validation asks whether a value is inside that declaration; it does
not carry a list of bad values anyone has seen. That list is unbounded and the domain is not.

## The family

`trainproof` (training runs) · `ttsproof` (TTS failure modes) · `spkproof` (speaker-verification
false rejection) · **`detectorproof`** (detector competence and robustness).

"Did the processing preserve speaker identity?" is a different question with different
statistics, and `spkproof` answers it. **Version 0.1.0 has no dependency on `spkproof` and does
not call it** — the two are used side by side, not chained.

## Status

`panel` is built and tested. `probe`, `identity` and `falsepositive` are specified and not
built. They are deliberately not built at the same time — `panel` is the gate the other three
depend on, and it has to refuse correctly on a detector already known to be incompetent
before anything is built on top of it.

## Licence

MIT.
