Metadata-Version: 2.4
Name: refsource
Version: 0.1.1
Summary: Look up reference data that carries its own source: every value comes with the URL it was read from and a verbatim quote from that page.
Author: referencesource.org
License: MIT
Project-URL: Homepage, https://referencesource.org
Project-URL: Catalogue, https://referencesource.org/catalog.json
Project-URL: Method, https://referencesource.org/method/
Keywords: reference-data,citations,provenance,verification,fact-check,regulations,compliance,lookup,datasets,source-quote
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# refsource

**Reference data lookups where every value comes back with the URL it was read
from and a verbatim quote from that page.**

```bash
pip install refsource
```

```python
import refsource

rows = refsource.lookup("conforming-loan-limits", state="AL",
                        county_name="AUTAUGA COUNTY")

rows[0]["limit_1_unit"]
# '$832,750'

rows[0]["limit_1_unit"].source
# 'https://www.fhfa.gov/document/d/cll/fullcountyloanlimitlist2026_hera-based_final_flat.csv'

rows[0]["limit_1_unit"].quote
# '01,001,AUTAUGA COUNTY,AL,33860,"$832,750 ","$1,066,250 ",...'
```

A value is a plain string everywhere a string is expected, and it carries its
own citation. That is the whole idea: an answer you can check beats an answer
you have to trust.

272 datasets, 90,792 records, from
**[referencesource.org](https://referencesource.org)** — regulatory thresholds
and deadlines, licensing rules by US state, version and end-of-support
calendars, certification registers, exposure limits, insurance minimums,
standards supersessions. Each one states its coverage, its sources and the
date it was last checked.

## Why it exists

If you ask a language model for a county loan limit, a state's minimum liability
cover, or when an API model shuts off, you usually get a confident answer with
no way to check it. In our own measurement of 271 such questions, 19% came back
confident and **wrong**.

This package answers the same questions with the source attached, so the
checking step is available rather than skipped — in your code, in a notebook,
or in whatever an agent is doing on your behalf.

## Usage

### Find a dataset (no network — the catalogue ships with the package)

```python
refsource.datasets("loan limit")
# [<Dataset conforming-loan-limits (3235 records, verified 2026-08-10)>]

refsource.fields("auto-insurance-minimums")
# ['state', 'bodily_injury_per_person', 'bodily_injury_per_accident', ...]
```

### Look records up

```python
refsource.lookup("auto-insurance-minimums", state="TX")
refsource.lookup("auto-insurance-minimums", state=["TX", "NM", "AZ"])
refsource.search("ai-model-deprecation-and-retirement", "gpt-4", limit=5)
refsource.get("conforming-loan-limits", "01001")
```

Matching is case-insensitive and forgiving about spacing and punctuation, so
`"Autauga County"` finds `"AUTAUGA COUNTY"`. A filter naming a field that does
not exist raises `NoSuchField` instead of returning an empty list — an empty
result from a typo looks exactly like an empty result from a real absence, and
one of those two is a wrong answer.

Datasets do not agree on how to spell things — one writes a state as `TX`, the
next as `Texas`, because each says what its source says. When a filter matches
nothing, ask what is actually there:

```python
refsource.dataset("auto-insurance-minimums").values_of("state", limit=5)
# ['Alabama', 'Alaska', 'Arizona', 'Arkansas', 'California']
```

### Read the provenance

```python
rec = refsource.lookup("conforming-loan-limits", fips_full="01001")[0]

rec.source_url      # the page or file this record was read from
rec.source_quote    # the passage that states it, verbatim
rec.url             # the published record's own page
rec.verified        # '2026-08-10'
rec.stale_after     # '2027-08-10'
print(rec.cite())   # a citation you can paste somewhere

v = rec["fha_limit_1_unit"]
v.source            # apps.hud.gov — a SECOND publisher's value,
                    # kept with its own source rather than folded in
v.confirmed         # the value was found word-for-word in the quote
v.derived           # our own reading rather than the page's words
v.disagreement      # other sources' versions of this same field, if any
```

Three things are deliberately visible rather than smoothed over:

- **Fields from a second publisher keep that publisher's URL and quote.** The
  FHFA conforming limit and the HUD/FHA limit sit in the same record; each cites
  the file it came from. Attributing one to the other would be a false citation.
- **`derived` marks our reading**, not the page's words — a state name we
  normalised, an identifier we assembled.
- **`disagreement` is not hidden.** Where two sources state a field differently,
  you get both, each with its own quote, and you decide.

### Staleness

Every dataset carries the date by which it should be re-checked. Read the
records of one that is past it and you get a `StaleDataWarning` naming the page
with the current copy. Nothing is silently served as fresh.

### Offline, caching, mirrors

Records are fetched on first use and cached (24h by default; the catalogue
itself needs no network at all).

```python
refsource.configure(
    cache_ttl=86400,        # 0 = always re-fetch, negative = never expire
    offline=True,           # cache only, never open a connection
    strict=True,            # refuse a bundle that no longer matches its pinned hash
    base_url="file:///path/to/site",   # a local copy or a mirror
)
```

Every one of those has an environment variable too: `REFSOURCE_CACHE`,
`REFSOURCE_CACHE_TTL`, `REFSOURCE_OFFLINE`, `REFSOURCE_STRICT`,
`REFSOURCE_BASE_URL`, `REFSOURCE_TIMEOUT`.

**Hash pinning.** The package holds the SHA-256 of every bundle as of the
release. If a fetched bundle differs, the dataset was re-verified upstream since
this version was cut — you get the live copy plus a `ChangedUpstreamWarning`, or
an `IntegrityError` under `strict=True`. The point is that a change is visible
rather than silent.

## Command line

```console
$ refsource datasets loan limit
$ refsource fields conforming-loan-limits
$ refsource lookup conforming-loan-limits fips_full=01001
$ refsource search ai-model-deprecation-and-retirement gpt-4 --limit 5
$ refsource show conforming-loan-limits 01001 --json
```

```
01001
  county_name: AUTAUGA COUNTY
  state: AL
  limit_1_unit: $832,750
  fha_limit_1_unit: 541,287
      from https://apps.hud.gov/pub/chums/cy2026-forward-limits.txt
      quoted: "3386000000MONTGOMERY, AL 203B S02200000541287069305008377001041125AL001…"
  source: https://www.fhfa.gov/document/d/cll/fullcountyloanlimitlist2026_hera-based_final_flat.csv
  quoted: "01,001,AUTAUGA COUNTY,AL,33860,"$832,750 ","$1,066,250 ","$1,288,800 ","$1,601,750 ""
  page:   https://referencesource.org/conforming-loan-limits/01001/
  verified 2026-08-10
```

## What is in the catalogue

A sample of the 272 datasets:

| dataset | what it answers |
|---|---|
| `conforming-loan-limits` | the FHFA and FHA loan limits for every US county |
| `auto-insurance-minimums` | minimum liability cover by US state |
| `ai-model-deprecation-and-retirement` | when an API model was deprecated and what replaces it |
| `software-end-of-support` | end-of-support dates from each vendor's own page |
| `iso-standard-supersessions` | what withdrew or replaced an ISO standard |
| `workplace-exposure-limits` | OSHA and Cal/OSHA permissible exposure limits |
| `fips-140-module-validation-status` | whether a cryptographic module's validation is still active |
| `drinking-water-contaminant-limits` | EPA maximum contaminant levels |

`refsource.datasets()` lists them all, offline.

## Data, licensing and accuracy

The records are facts with attribution, not reproductions. Each dataset states
its own licence position and links the source it was read from; the package code
is MIT. Where a source's terms forbid reuse, the dataset is not published at
all.

No value is ever supplied by this package or by a model — if a source does not
state something, the row is omitted rather than guessed. Where you need to be
sure, the quote and the URL are right there: check it.

Found something wrong? That is the one thing worth reporting —
https://referencesource.org/ has the contact and the method behind every
dataset.

## Related

- **[`refsource` for JavaScript](https://www.npmjs.com/package/refsource)** —
  the identical package on npm: same catalogue, same citations,
  `npm install refsource`.
- **[`referencesource-mcp`](https://www.npmjs.com/package/referencesource-mcp)**
  — the same catalogue as tools for AI assistants: `npx referencesource-mcp`,
  or add `https://referencesource.org/mcp` to any MCP client.
- **Bulk data** — every dataset publishes `data.json` at its own URL, and the
  catalogue is at https://referencesource.org/catalog.json.
- **This package on PyPI** —
  [pypi.org/project/refsource](https://pypi.org/project/refsource/).
