Metadata-Version: 2.5
Name: reference-verifier
Version: 0.2.2
Summary: Check that every reference in your paper exists, against Crossref, OpenAlex, DBLP, arXiv and PubMed.
Project-URL: Homepage, https://the-aurora-lab.github.io/tools/reference-verifier/
Project-URL: Source, https://github.com/the-aurora-lab/reference-verifier
Project-URL: Issues, https://github.com/the-aurora-lab/reference-verifier/issues
Project-URL: Changelog, https://github.com/the-aurora-lab/reference-verifier/blob/main/CHANGELOG.md
Author-email: Aurora Lab <auroralab.skills@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: bibtex,citations,crossref,hallucinated citations,latex,references,research integrity
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing :: Markup :: LaTeX
Requires-Python: >=3.10
Requires-Dist: bibtexparser<2,>=1.4
Requires-Dist: httpx>=0.27
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == 'dev'
Description-Content-Type: text/markdown

# Reference Verifier

Check that every reference in your paper exists before a reviewer, a venue or arXiv checks for you.

`refverify` reads your `.bib` (or your LaTeX paper and the bibliography it uses) and looks each entry up
in Crossref, OpenAlex, DBLP, arXiv and PubMed (and Semantic Scholar, if you have a key). It reports
references that no registry lists, DOIs that are not registered, and DOIs that belong to a different paper.

```
$ refverify main.tex

Reference Verifier 0.2.2: 41 references from refs.bib

  37 found · 2 not found · 1 wrong DOI · 1 uncertain

  WRONG DOI  chen2023    Scalable graph transformers for molecules (2023)
                         DOI 10.1016/j.patcog.2021.108045 belongs to a different work: "..."
  NOT FOUND  smith2024   Quantum attention for genome alignment (2024)
                         no matching work in Crossref, OpenAlex, DBLP or arXiv
  ...
```

## Install

```
pip install reference-verifier
```

Python 3.10 or newer. To keep the command in an environment of its own: `pipx install reference-verifier` or
`uv tool install reference-verifier`.

## Use

```
refverify refs.bib                 # every entry in a .bib
refverify main.tex                 # only the keys main.tex cites (follows \input and \bibliography)
refverify main.tex --json out.json # machine-readable results too
```

Exit status is 1 when a journal or conference reference is not found or has a bad DOI, so `refverify`
can run in a script or CI job. `--strict` also fails on uncertain matches.

Set your email so Crossref and PubMed can contact you if something goes wrong (their "polite pool"):

```
export REFVERIFY_MAILTO=you@example.org
```

OpenAlex searches draw on a daily budget ($0.001 per search; looking a DOI up is free). Without a key
everyone on your network shares $0.10 a day; a free key from openalex.org gives you $1 a day of your own,
enough for about a thousand searches:

```
export OPENALEX_API_KEY=...
```

Semantic Scholar finds reports and preprints the other registries miss, but its anonymous pool is
usually rate-limited, so it is used only when you set a key (free on application at semanticscholar.org):

```
export S2_API_KEY=...
```

If a registry stops answering during a run (rate limits, a bot check), `refverify` skips it for the rest of
the run and says so; rerun later to include it.

## What the results mean

| Result | Meaning |
|---|---|
| found | A registry lists a work with this title and matching authors (or year, when the entry has no authors). Also: an arXiv identifier that resolves to the same authors and year under a newer title, and a software or dataset DOI (Zenodo releases change title per version) with the same first author; both carry a note. |
| NOT FOUND | No registry lists a matching work. Check it by hand: books, theses, reports and very new papers are often missing from registries. Misses on white papers, blog posts and technical reports are marked low severity and do not fail the run. |
| WRONG DOI | The DOI resolves, but to a paper with a different title. |
| BAD DOI | The DOI is not registered with any DOI agency. |
| UNCERTAIN | Something close exists (a similar title, or the same title with different authors). |
| skipped | Web pages, software entries and code or model releases (GitHub, Hugging Face, PyPI...) without a DOI, and entries with nothing to look up. |

A reference that is not found is not proof of fabrication, and a found one can still have a wrong year,
venue or author list. `refverify` checks existence.

## How well it works

Measured with 0.2.0 (0.2.1 changes only how file names are shown, 0.2.2 only how it is packaged) on a held-out benchmark of 246 labelled references that was built separately from the
development set and from the earlier held-out set, and scored once, with no changes to the matching rules afterwards
([`benchmark/heldout-0.2/`](https://github.com/the-aurora-lab/reference-verifier/tree/main/benchmark/heldout-0.2); sources, licences and labelling rules in
[`HELDOUT.md`](https://github.com/the-aurora-lab/reference-verifier/blob/main/benchmark/heldout-0.2/HELDOUT.md)):

| Measure | Result | 95% CI |
|---|---|---|
| Real references found | 94 of 95 | 94.3–99.8% |
| Fabricated references flagged | 54 of 54 | 93.4–100% |
| Wrong DOIs flagged as a DOI problem | 30 of 30 | 88.6–100% |
| Retracted papers found (they exist; flagging retractions is not part of this check) | 35 of 35 | 90.1–100% |
| Precision of flags (flags on fabricated or wrong-DOI entries, out of all flags) | 84 of 85 (98.8%) | 93.6–99.8% |

- The real reference it missed is a DOI-less ICLR 2022 paper that only DBLP indexes; DBLP's search turned the tool
  away during the run (it shows a bot check to scripts, which the tool does not try to get past).
- Real references were drawn at random across computer science, biomedicine, physics, mathematics, social science,
  the humanities and other fields, written in the messy forms real `.bib` files use. The fabricated ones come from
  public compilations of hallucinated citations found in ACL-family papers and ICLR 2026 submissions, so all 54 are
  computer science.
- "Fabricated" means listed by one of those compilations and found neither in any registry nor by a web search for
  the exact title. That is strong evidence, not proof.
- 0.1.0 was scored on an earlier held-out set (`benchmark/heldout/`; its numbers are in the
  [CHANGELOG](https://github.com/the-aurora-lab/reference-verifier/blob/main/CHANGELOG.md)). The development set the rules were tuned on (342 references) is in
  [`benchmark/`](https://github.com/the-aurora-lab/reference-verifier/tree/main/benchmark); its scores are in `benchmark/results/summary.md` and are not a fair
  measure. The benchmark is in the repository, not in the package.

Rerun it from a clone of the repository: `python benchmark/evaluate.py --data benchmark/heldout-0.2/heldout.jsonl --out benchmark/heldout-0.2/results --heldout`
(results vary slightly as registries change and become reachable or not).

## Privacy and etiquette

Titles, first authors, years and identifiers from your bibliography are sent to the registries listed above.
Nothing else leaves your machine. Answers are cached for 30 days in `~/.cache/reference-verifier/`, so
repeat runs are fast and light on the registries. Requests are rate-limited per registry, including the one
request every three seconds that arXiv asks for.

## Reference Verifier Pro

This free core checks that each reference exists. Reference Verifier Pro, a paid add-on, also checks each one against
the publisher's record and the retraction databases (retractions and corrections, wrong DOIs with the likely correct
one, authors, years, venues and pages, duplicates and cite keys, published versions of preprints), writes corrected
copies to review and dated audit reports, and checks a whole class's papers at once. A real run, the held-out
benchmark and the licences: https://the-aurora-lab.github.io/tools/reference-verifier/

## Licence

MIT
