Metadata-Version: 2.5
Name: osaudit
Version: 0.2.0
Summary: Automated open-science audit: trace every claim of a paper to its evidence
Project-URL: Homepage, https://github.com/ASSERT-KTH/osaudit
Project-URL: Issues, https://github.com/ASSERT-KTH/osaudit/issues
Author: Martin Monperrus
License-Expression: MIT
License-File: LICENSE
Keywords: audit,citations,latex,open science,provenance,reproducibility
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing :: Markup :: LaTeX
Requires-Python: >=3.13
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# osaudit

Automated open-science audit: report every claim of a scientific paper that is not traced, mechanically, to the evidence that supports it. A number to a results file, a quote to a page of a verified source, a result to the instrument and the input that produced it. Silent when everything is traced.

What a repository must contain to be auditable is specified in the [conventions](https://github.com/ASSERT-KTH/assert-research-agent/blob/main/CONVENTIONS.md), shared with the [research agent](https://github.com/ASSERT-KTH/assert-research-agent), which enforces them while the work is done; `osaudit` checks them afterwards, from the repository alone.

## Install

    uv venv && uv pip install -e '.[dev]'

PDF input needs `pdftotext` (poppler-utils).

## Usage

In the paper repository, write `audit.toml`:

```toml
[paper]
tex = ["paper/sections/*.tex"]      # or: pdf = "paper/main.pdf"

[results]                           # optional: a paper without a study audits its quotes only
files = ["study/results/*.json"]
instruments = ["study/*.py"]        # the code that produces the results

[[results.depends]]                 # optional: what one results file depends on (default: all)
file = "study/results/survey.json"
instruments = ["study/survey.py", "study/_common.py"]

[[results.manual]]                  # a results file made by a person
file = "study/results/labels_check.json"
by = "A. Author"
date = "2026-10-17"
reason = "the author checked 20 labels of the classifier"

[[numbers.derive]]                  # a number computed from results values
value = "392"
expr = "panel.overall.total - panel.overall.answers.yes"

[[numbers.allow]]                   # a number the study does not produce
value = "2,000,000"
reason = "population size published by the cited census"
cite = "census2020"

[[quotes]]                          # a quote as read in its source
key = "doe2001archive"              # BibTeX key
text = "We call the archive the second memory of a laboratory, after the minds of its members."
version = "publisher"               # publisher|author-pdf|preprint|reprint|abstract|secondary
url = "https://doi.org/10.1000/doe2001"
page = "12"                         # omitted for an abstract
date = "2026-10-01"                 # read on
status = "verified"                 # verified|unverified

[study]                             # optional: where the study keeps its files (defaults shown)
prereg = "prereg"
explore = "explore"
experiments = "experiments"
results = "results"
```

Then:

    osaudit                     # every check (static parts)
    osaudit numbers             # every printed number traced to results, derivation or allow
    osaudit quotes              # every cited quote verbatim in a verified [[quotes]] entry
    osaudit results             # every results file has a provenance that still holds
    osaudit results --inputs    # ... and its inputs still have their recorded sha256
    osaudit results --rerun     # ... and its command, rerun at its time anchor, prints the same
    osaudit study               # every confirmatory result pre-registered, unamended, on unseen data
    osaudit CHECK --json        # a JSON array of findings, [] when clean
    osaudit CHECK --root PATH --config FILE

One line per finding, `file:line: rule: message` (for a PDF, the line is the page):

    paper/results.tex:6: numbers/untraced: 392 matches no results value, derivation or allow entry; candidate expr: panel.overall.total - panel.overall.answers.yes | survey.complete - panel.overall.answers.yes
    audit.toml:8: numbers/derive-mismatch: 70.0% declared, but ages.by_age["<30"].holds / ages.by_age["<30"].informative = 0.625, printed 62.5%
    paper.tex:2: quotes/mismatch: ``the only memory of a laboratory'' is not verbatim in the entry of doe2001archive (closest: audit.toml:4)
    study/results/survey.json:1: results/no-provenance: no 'provenance' block
    study/ages.py:12: results/wall-clock: datetime.now( reads the clock or a file time; read the time anchor from $OSAUDIT_NOW instead
    results/passrate.json:9: study/data-reused: input data/test/part.csv has the sha256 of an input of results/look.json, which is not confirmatory

Exit status: 0 clean, 1 findings, 2 the audit could not run (configuration, input).

### Producing results

Produce each results file through `osaudit run`, which records its provenance (command, inputs and their sha256, git commit, time anchor) in the file:

    osaudit run --input data/answers.csv -o study/results/survey.json -- python study/survey.py data/answers.csv

The instrument prints its results as a JSON object on stdout, and reads the time anchor from `$OSAUDIT_NOW` instead of the clock. Its instruments must be committed; `run` refuses otherwise. A clock read that is justified carries `# osaudit: allow results/wall-clock -- <reason>` on its line.

### Confirmatory and exploratory results

A run says which kind of result it produces, and the provenance records it (`mode`, `study`):

    osaudit run --prereg prereg/2026-10-21-passrate.md --input data/test -o results/passrate.json -- python experiments/passrate.py data/test
    osaudit run --explore explore/2026-10-20-look.md --input data/train.csv -o results/look.json -- python explore/look.py data/train.csv

A pre-registration and an exploration log are Markdown files with TOML front matter, in the formats of the conventions. `run --prereg` refuses, before the command sees any data, a run that `osaudit study` would reject. `osaudit study` then proves, for every confirmatory results file, from git alone:

- `study/not-preregistered`: its pre-registration is committed at the run's commit, with valid front matter (hypothesis, data, alpha, m, correction, ...);
- `study/prereg-mismatch`: the command runs a registered experiment, and every input is inside the registered data;
- `study/amended`: the registration did not change between its first commit and the run without an `[[amendments]]` entry, and no amendment was made after data was seen;
- `study/data-reused`: no input was seen before, as an input of an exploratory or unlabelled result (same sha256, under any name) or in an exploration log's `data_seen`.

A results file without a mode is unlabelled: never confirmatory, silent for these rules. When some results are exploratory, a printed number traced only to them (or a derivation with an exploratory operand) must say "exploratory" in its paragraph (in a PDF, its page): `numbers/exploratory-unlabelled`.

### Results values

The `provenance` block is not a result and is never matched. A value is named `<file stem>.<key>...`, with `["key"]` for keys that are not identifiers and `[i]` for list indices: `ages.by_age["<30"].share`. A printed number matches a value within half a unit of its last digit; a percent matches 100·v. Expressions allow `+ - * /`, parentheses and numeric constants, and are never evaluated as Python.

Candidates proposed for an untraced number are single operations over counts (sums, differences, shares), ranked by closeness in the results tree. They are hints: nothing is accepted until written into `audit.toml`.

### What is not audited as a number

LaTeX comments, `tikzpicture` and verbatim environments, verbatim quotes (` ``...'' `, audited as quotes), arguments of `\input \label \cref \cite \url \begin` and similar, parts of a document (`Section 4.3`, `Table~1`, `p.~107`), and numbers glued to a word (`123d0d04`, `3.0cm`, `RQ1`). In a PDF: numbered citations, list labels, lines holding only a number, and everything after the last "References" heading.

### Quotes

A quote is ` ``...'' ` with a citation in the same sentence (the nearest after it, else before it); its page is the `p.~X` of that citation. It must be verbatim, after normalization of LaTeX, typography and case, in the full sentence of an entry for that key; `[...]` marks a cut. In a PDF, citations are numbered, so a quote `“...”` matches any entry by its text.

## Status

`numbers`, `quotes`, `results`, `study` and `run` are implemented. References are checked by a separate tool, [check-hallucinations](https://github.com/ASSERT-KTH/check-hallucinations); cross-references and reviews are specified in the [conventions](https://github.com/ASSERT-KTH/assert-research-agent/blob/main/CONVENTIONS.md) and not built.

## License

MIT
