Metadata-Version: 2.4
Name: gov-budget-data
Version: 0.4.1
Summary: Load and analyze reconciled public-budget data with Polars and DuckDB
Project-URL: Changelog, https://nickhand.github.io/gov-budget-data/project/release-notes/
Project-URL: Documentation, https://nickhand.github.io/gov-budget-data/
Project-URL: Homepage, https://nickhand.github.io/gov-budget-data/
License-Expression: MIT
License-File: LICENSE
Keywords: budget,civic-data,open-data,public-finance
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: duckdb>=1.5.1
Requires-Dist: packaging>=24.2
Requires-Dist: polars>=1
Requires-Dist: pyarrow>=17
Requires-Dist: pydantic>=2
Provides-Extra: notebook
Requires-Dist: altair<7,>=6.2; extra == 'notebook'
Requires-Dist: great-tables<1,>=0.22; extra == 'notebook'
Requires-Dist: matplotlib<4,>=3.10; extra == 'notebook'
Requires-Dist: vl-convert-python<2,>=1.9; extra == 'notebook'
Description-Content-Type: text/markdown

# gov-budget-data

`gov-budget-data` is the user-facing Python package for discovering, loading,
and analyzing reconciled public-budget data. It uses
[Polars](https://pola.rs/) for DataFrames and
[DuckDB](https://duckdb.org/) for projection, filtering, and SQL over the
release's Parquet files.

The package is currently an alpha. Its default API fails closed around
unverified data and ambiguous time-series observations rather than returning a
plausible but underspecified number.

The package code is licensed under the MIT License. Data is distributed
separately: releases that name `CC0-1.0` dedicate only this project's
transformed structured data under CC0 and do not relicense source PDFs,
source-page images, or third-party records.

## Install

`gov-budget-data` requires Python 3.12 or newer.

```bash
python -m pip install gov-budget-data
```

For the shared notebook presentation layer—Matplotlib and Altair themes plus
Great Tables—install the optional extra:

```bash
python -m pip install "gov-budget-data[notebook]"
```

The `civic` default uses darker U.S. Web Design System color tokens; the
quieter `metropolitan` palette is also included. Both work in Matplotlib and
Altair and have `light` and `soft` canvas variants:

```python
from gov_budget_data import presentation as pub

palette = pub.apply_style("civic")
pub.enable_altair_theme("civic")
print(palette.series)
```

The helpers also provide explicit dollar, percentage, and count axes; a
horizontal unit label above the y scale; consistent panel proportions; thin
horizontal-bar geometry; direct labels; source lines; and publication tables.
See the repository's [notebook style guide](../../notebooks/STYLE_GUIDE.md)
for complete examples.

The wheel contains the reader and analysis code, not the data corpus. Code and
data are versioned separately so users can pin a data snapshot without
redownloading it for every package update.

## Get a data release

Downloading and opening data are separate operations:

```python
import gov_budget_data as gbd
import polars as pl

cached = gbd.fetch("latest")
data = gbd.open_release(cached)
print(data.info().data_identity)
```

`fetch()` is the package's only network API. It downloads an immutable release
into the user cache and validates the release-index contract, archive digest,
safe archive layout, size limits, and the release's inner file manifest. Pass
`index_sha256=...` when an independently published index digest is available
and the exact index bytes should also be pinned.
`open_release()` performs no network access; it opens and revalidates an
already cached or unpacked release:

```python
data = gbd.open_release("/path/to/unpacked/release")
print(data.info())
```

By default, `fetch()` uses the project's production release index at
`https://dpfqddwqzniu6.cloudfront.net/releases/index.json`. Pass
`index_url=...` or set `GBD_RELEASE_INDEX_URL` to use a mirror or another
publisher. An explicit argument takes precedence over the environment
variable. Set `GBD_CACHE_DIR` to override the platform-standard cache
directory:

```bash
export GBD_RELEASE_INDEX_URL="https://mirror.example/releases/index.json"
export GBD_CACHE_DIR="/data/gov-budget-data-cache"
```

Then:

```python
cached = gbd.fetch("latest")
data = gbd.open_release(cached)
```

`latest` is convenient for exploration. Pin an explicit data-release version
in reproducible analyses.

Installing the Python package never downloads data, and opening a release
never updates it silently.

## Discover what is available

Start with the catalog instead of guessing file names:

```python
for dataset in data.datasets():
    print(dataset.name, dataset.fiscal_year_min, dataset.fiscal_year_max)

profile = data.describe("collections_monthly_city")
print(profile.title)
print(profile.jurisdictions)
print(profile.time_grains)
print(profile.measures)
print(profile.columns)

coverage = data.coverage("collections_monthly_city")
print(coverage.documents)
print(
    coverage.editions.filter(
        pl.col("status").is_in(["internal_absent_candidate", "applicability_unknown"])
    )
)
print(coverage.unresolved)
```

`describe()` reports coverage, trusted-row counts, dimensions, observed
measures and units, temporal grains, and the physical schema without
materializing the full dataset as a DataFrame. Annual observations use
`fiscal_year` with `period=None`; `period` remains an optional within-year
locator rather than carrying a synthetic annual label.

`coverage()` performs the deeper release audit. Its `documents`, `entities`,
and `series` frames report observation counts at progressively finer grains.
Raw rows define the entity and exact-series inventory, so an all-unverified
series remains visible with zero trusted observations instead of disappearing.
The `n_raw_*` and `n_trusted_*` fields preserve that distinction;
`include_unverified=True` changes which rows count as observed but does not
erase it. `editions` is a review ledger of inferred exact-series/report
candidates, and `unresolved` shows raw labels that still lack stable IDs while
separating trusted from unverified rows. `observed_underived` keeps raw source
editions visible when their temporal contract or catalog period cannot be
derived. `internal_absent_candidate` means an edition falls between raw
observations; `applicability_unknown` means it falls after the last raw
observation. Neither is an expected or missing report, and neither is zero.
Until producer-side applicability is declared,
`coverage_status` remains `coverage_unknown`.
For annual panels, each exact-series row reports the full year range between
its first and last raw observation. `absent_value_years` have no raw row,
while `unverified_value_years` have raw evidence but no trusted row;
`missing_value_years` reflects the selected trust view.

Page-typed publications expose both an umbrella dataset and virtual
`family__table_type` datasets. Call `datasets()` for the exact catalog shipped
by a release.

## Load facts

`load()` returns a Polars DataFrame. Common filters and column projection are
pushed into DuckDB before data reaches Python:

```python
frame = data.load(
    "qcmr_positions",
    fiscal_year=(2022, 2026),
    columns=[
        "document_id",
        "fiscal_year",
        "period",
        "entity_id",
        "label",
        "value",
        "value_unit",
        "source_page",
    ],
)
```

Use `filters={...}` for another physical column listed by `describe()`:

```python
frame = data.load(
    "phila_budget_detail_adopted",
    filters={"ctx_fund": "General Fund"},
    columns=["fiscal_year", "org_entity_id", "label", "value", "value_unit"],
)
```

For advanced work, query registered views directly:

```python
result = data.sql(
    """
    SELECT fiscal_year, count(*) AS rows
    FROM qcmr_positions
    GROUP BY fiscal_year
    ORDER BY fiscal_year
    """
)

connection = data.connect()
try:
    print(connection.sql("SHOW TABLES").fetchall())
finally:
    connection.close()
```

Each dataset has a trusted view named for the dataset and a `<dataset>_raw`
diagnostic view. The raw view is not a second source of truth; it exists to
inspect rows withheld by the default trust boundary.

## Trace facts to their source

Every current v5 data release includes a checksummed source-document catalog.
Inspect coverage and acquisition provenance without opening fact tables:

```python
sources = data.documents(
    jurisdiction="city_of_philadelphia",
    fiscal_year=(2025, 2026),
    eligible=True,
)

source = data.document("phila_budget_detail_adopted_fy26_book2")
print(source.source_url)
print(source.source_sha256)
print(source.publisher_reported_date)
print(source.retrieved_at)
```

Publisher-reported dates, checksummed catalog-observation times, archive
captures, and project retrieval times remain distinct. Missing publication
claims stay null; repository history is never substituted. Split artifacts
also expose their parent publisher-object hash and one-based source page.

Map a loaded row back to its one-based PDF page:

```python
row = data.load(
    "phila_budget_detail_adopted",
    columns=["document_id", "source_page", "label", "value"],
).row(0, named=True)

receipt = data.citation(row["document_id"], page=row["source_page"])
print(receipt.label)
print(receipt.page_url)
print(receipt.source_sha256)
```

Some historical documents do not yet have a recoverable publisher or archive
URL. Their stable document id, exact source SHA-256, and page locator remain
available; `source_url` and `page_url` are then `None`.

## Find stable entities

Labels in government publications change and can be reused in different
contexts. Search the shipped entity catalog and query by stable IDs:

```python
matches = data.search_entities(
    "police",
    dimension="department",
    jurisdiction="city_of_philadelphia",
)

for match in matches[:5]:
    print(match.entity.id, match.entity.name, match.score)
```

Other entity helpers include:

```python
data.entities(dimension="fund")
data.entity("phl_city_wage_tax")
data.lineage(entity_id="phl_dept_parks_and_recreation")
```

Use the stable `entity_id` and `org_entity_id` fields where a dataset exposes
them, and declared lineage instead of joining years on presentation labels.
Fund and program context is currently exposed through its verified
`ctx_fund*` and `ctx_program*` fields; not every contextual label has a stable
entity mapping yet.

## Build an ambiguity-safe series

`series()` selects one semantic observation per `(value_year, period)` and
retains units, grain, and source coordinates:

```python
wage = data.series(
    "collections_monthly_city",
    entity_id="phl_city_wage_tax",
    column_role="fytd_actual",
    row_kind="total",
)
```

Current/prior tax-year cohorts use the same tax entity and an explicit
qualifier:

```python
current_wage = data.series(
    "collections_monthly_city",
    entity_id="phl_city_wage_tax",
    entity_qualifier="current",
    column_role="fytd_actual",
    row_kind="line_item",
)
```

The function does not aggregate by default. It raises
`AmbiguousObservationError` when a query spans incompatible tags, units,
table types, row kinds, organizational contexts, or repeated observations
across any publication edition. Narrow the query with semantic selectors; use
`aggregate=...` only when the remaining observations truly share one meaning.
Default sums and means also refuse when an intended contributor is unverified,
so a filtered-out row cannot silently understate a derived total. Passing
`include_unverified=True` makes the result diagnostic; a sum with any null
operand remains null rather than treating missing as zero.
Rolling publication families must declare their canonical edition policy in
the normalized dataset; `series()` never guesses which revision should win.

The default `value_type="canonical"` preserves the opened release's numeric
dtype. New fact-schema v2 money series can be exact Decimal, while immutable
v1 releases retain their original Float64 values. Statistical and charting
libraries can request an explicit conversion of only the normalized value:

```python
wage_for_model = data.series(
    "collections_monthly_city",
    entity_id="phl_city_wage_tax",
    column_role="fytd_actual",
    series_key="current_tax_year",
    value_type="float64",
)
```

For source-faithful chart labels, `gbd.format_values(frame, style="source")`
adds `display_value` from the preserved source lexeme and declared unit basis
without changing the canonical numeric value. It rejects mixed or unknown
units rather than guessing.

V2 series also retain selected and raw source text, source-scale values,
multipliers, displayed increments, and review evidence in `sources`. Common
value metadata is available as top-level columns. Reviewed values are included
by default only when their value, table, and header evidence all pass. For
diagnostic aggregation, each source struct retains those operand trust gates,
and the derived value is verified only when every operand passes them.

Every non-direct v2 value carries a content-bound evidence ID. Resolve it from
the same verified release to inspect the immutable typed decision record:

```python
receipt = data.explain_value(value_evidence_id)
print(receipt.value_disposition, receipt.evidence_detail)
print(receipt.source_value, receipt.value_multiplier, receipt.canonical_value)
```

`retained_numeric` describes the final disposition, not its trust status; use
`value_verified` to distinguish trusted from diagnostic retained claims.
Ordinary direct parses intentionally have a null evidence ID.

Some sources publish useful numeric breakdowns beside an already-counted
parent row. Reviewed instances are retained as `row_kind="memo"`: they are
queryable series observations but are excluded from ETL table arithmetic.
Select row kinds explicitly before aggregating raw `load()` output so a memo
is not summed with its parent.

The default `entity_id` selects the row's subject. For schedules where a
department is contextual metadata around program, class, or position rows,
set `entity_field="org_entity_id"` and use `filters={...}` to select the
intended fund, program, or other grain.

Declared organizational succession can be followed explicitly when the
selected dataset contains the predecessor observations:

```python
human_services = data.series(
    "pa_general_fund_enacted",
    entity_id="pa_human_services",
    entity_field="org_entity_id",
    column_role="budget_estimate",
    row_kind="total",
    follow_lineage="backward",
)
```

Lineage seams remain visible in the result. Undeclared splits or
apportionments fail rather than being guessed.

## Schema and provenance

Dataset families vary, and Parquet files are unioned by name. Common columns
fall into four groups:

| Group | Representative columns |
| --- | --- |
| Release and document identity | `document_id`, `document_type`, `jurisdiction`, `fiscal_year`, `period` |
| Observation | `entity_id`, `column_role`, `column_tag`, `value_year`, `value`, `value_unit` |
| Organizational context | `org_entity_id`, `ctx_department`, `ctx_fund`, `ctx_program` |
| Evidence and trust | `source_page`, `table_index`, `row_index`, `parser_name`, `table_reconciled`, `table_accounted`, `headers_verified` |

Call `describe(name).columns` for the authoritative physical schema in the
release you opened. During the alpha period, prefer named projections over
persisting assumptions about `SELECT *`.

Every released fact retains the source document id, one-based PDF/source page,
table, and row coordinates needed to identify its evidence. Source PDFs are
not bundled in the Python wheel or data archive. The release ships a
checksummed source catalog with acquisition URLs and exact source hashes, plus its entity
catalog, organizational lineage, authoritative extraction selection,
source-integrity decision, and optional annotation sidecars under the same
validated manifest.

## Correctness boundary

By default:

- facts from source-quarantined documents are absent, while their provenance
  and quarantine decision remain visible in the source catalog;
- exactly one authoritative extraction backend is selected per document;
- rows without accounted totals checks or verified headers are excluded;
- evidence-pinned source discrepancies can be accounted without being
  misrepresented as arithmetically reconciled;
- units and semantic context are retained in series output; and
- release files are checked against their recorded size and SHA-256 digest.

`include_unverified=True` and other diagnostic opt-ins are for investigation.
They do not promote those rows to trusted facts. Never aggregate mixed or
unknown units, and do not treat a raw display label as a stable identity.

## Local pipeline data and compatibility API

Repository contributors can still pass a pipeline artifacts root or packaged
release directly to the module-level functions:

```python
frame = gbd.load("qcmr_positions", source="/path/to/release")
datasets = gbd.datasets(source="/path/to/release")
```

Alternatively, set `GBD_DATA_ROOT`. An explicit `DataStore` from
`open_release()` is recommended for applications because every call remains
bound to one visible data snapshot.

The installed package version is exposed directly:

```python
import gov_budget_data as gbd

print(gbd.__version__)
```

## Documentation

The [public documentation](https://nickhand.github.io/gov-budget-data/)
covers installation, available datasets, release management, the complete API,
worked examples, and the package's trust model.
