Metadata-Version: 2.4
Name: gov-budget-data
Version: 0.2.0
Summary: Load and analyze reconciled public-budget data with Polars and DuckDB
Project-URL: Homepage, https://github.com/nickhand/gov-budget-data
Project-URL: Issues, https://github.com/nickhand/gov-budget-data/issues
Project-URL: Repository, https://github.com/nickhand/gov-budget-data
License-Expression: MIT
License-File: LICENSE
Keywords: budget,civic-data,open-data,public-finance
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: duckdb>=1.5.3
Requires-Dist: packaging>=24.2
Requires-Dist: polars>=1
Requires-Dist: pyarrow>=17
Requires-Dist: pydantic>=2
Description-Content-Type: text/markdown

# gov-budget-data

`gov-budget-data` is the user-facing Python package for discovering, loading,
and analyzing reconciled public-budget data. It uses
[Polars](https://pola.rs/) for DataFrames and
[DuckDB](https://duckdb.org/) for projection, filtering, and SQL over the
release's Parquet files.

The package is currently an alpha. Its default API fails closed around
unverified data and ambiguous time-series observations rather than returning a
plausible but underspecified number.

The package code is licensed under the MIT License. Data is distributed
separately: releases that name `CC0-1.0` dedicate only this project's
transformed structured data under CC0 and do not relicense source PDFs,
source-page images, or third-party records.

## Install

`gov-budget-data` requires Python 3.12 or newer.

```bash
python -m pip install gov-budget-data
```

The wheel contains the reader and analysis code, not the data corpus. Code and
data are versioned separately so users can pin a data snapshot without
redownloading it for every package update.

## Get a data release

Downloading and opening data are separate operations:

```python
import gov_budget_data as gbd

cached = gbd.fetch("latest")
data = gbd.open_release(cached)
print(data.info().data_identity)
```

`fetch()` is the package's only network API. It downloads an immutable release
into the user cache and validates the release-index contract, archive digest,
safe archive layout, size limits, and the release's inner file manifest. Pass
`index_sha256=...` when an independently published index digest is available
and the exact index bytes should also be pinned.
`open_release()` performs no network access; it opens and revalidates an
already cached or unpacked release:

```python
data = gbd.open_release("/path/to/unpacked/release")
print(data.info())
```

By default, `fetch()` uses the project's production release index at
`https://dpfqddwqzniu6.cloudfront.net/releases/index.json`. Pass
`index_url=...` or set `GBD_RELEASE_INDEX_URL` to use a mirror or another
publisher. An explicit argument takes precedence over the environment
variable. Set `GBD_CACHE_DIR` to override the platform-standard cache
directory:

```bash
export GBD_RELEASE_INDEX_URL="https://mirror.example/releases/index.json"
export GBD_CACHE_DIR="/data/gov-budget-data-cache"
```

Then:

```python
cached = gbd.fetch("latest")
data = gbd.open_release(cached)
```

`latest` is convenient for exploration. Pin an explicit data-release version
in reproducible analyses.

Installing the Python package never downloads data, and opening a release
never updates it silently.

## Discover what is available

Start with the catalog instead of guessing file names:

```python
for dataset in data.datasets():
    print(dataset.name, dataset.fiscal_year_min, dataset.fiscal_year_max)

profile = data.describe("collections_monthly_city")
print(profile.title)
print(profile.jurisdictions)
print(profile.time_grains)
print(profile.measures)
print(profile.columns)
```

`describe()` reports coverage, trusted-row counts, dimensions, observed
measures and units, temporal grains, and the physical schema without
materializing the full dataset as a DataFrame. Annual observations use
`fiscal_year` with `period=None`; `period` remains an optional within-year
locator rather than carrying a synthetic annual label.

Page-typed publications expose both an umbrella dataset and virtual
`family__table_type` datasets. Call `datasets()` for the exact catalog shipped
by a release.

## Load facts

`load()` returns a Polars DataFrame. Common filters and column projection are
pushed into DuckDB before data reaches Python:

```python
frame = data.load(
    "qcmr_positions",
    fiscal_year=(2022, 2026),
    columns=[
        "document_id",
        "fiscal_year",
        "period",
        "entity_id",
        "label",
        "value",
        "value_unit",
        "source_page",
    ],
)
```

Use `filters={...}` for another physical column listed by `describe()`:

```python
frame = data.load(
    "phila_budget_detail_adopted",
    filters={"ctx_fund": "General Fund"},
    columns=["fiscal_year", "org_entity_id", "label", "value", "value_unit"],
)
```

For advanced work, query registered views directly:

```python
result = data.sql(
    """
    SELECT fiscal_year, count(*) AS rows
    FROM qcmr_positions
    GROUP BY fiscal_year
    ORDER BY fiscal_year
    """
)

connection = data.connect()
try:
    print(connection.sql("SHOW TABLES").fetchall())
finally:
    connection.close()
```

Each dataset has a trusted view named for the dataset and a `<dataset>_raw`
diagnostic view. The raw view is not a second source of truth; it exists to
inspect rows withheld by the default trust boundary.

## Trace facts to their source

Every current v5 data release includes a checksummed source-document catalog. Inspect
coverage and acquisition provenance without opening fact tables:

```python
sources = data.documents(
    jurisdiction="city_of_philadelphia",
    fiscal_year=(2025, 2026),
    eligible=True,
)

source = data.document("phila_budget_detail_adopted_fy26_book2")
print(source.source_url)
print(source.source_sha256)
```

Map a loaded row back to its one-based PDF page:

```python
row = data.load(
    "phila_budget_detail_adopted",
    columns=["document_id", "source_page", "label", "value"],
).row(0, named=True)

receipt = data.citation(row["document_id"], page=row["source_page"])
print(receipt.label)
print(receipt.page_url)
print(receipt.source_sha256)
```

Some historical documents do not yet have a recoverable publisher or archive
URL. Their stable document id, exact source SHA-256, and page locator remain
available; `source_url` and `page_url` are then `None`.

## Find stable entities

Labels in government publications change and can be reused in different
contexts. Search the shipped entity catalog and query by stable IDs:

```python
matches = data.search_entities(
    "police",
    dimension="department",
    jurisdiction="city_of_philadelphia",
)

for match in matches[:5]:
    print(match.entity.id, match.entity.name, match.score)
```

Other entity helpers include:

```python
data.entities(dimension="fund")
data.entity("phl_city_wage_tax")
data.lineage(entity_id="phl_dept_parks_and_recreation")
```

Use the stable `entity_id` and `org_entity_id` fields where a dataset exposes
them, and declared lineage instead of joining years on presentation labels.
Fund and program context is currently exposed through its verified
`ctx_fund*` and `ctx_program*` fields; not every contextual label has a stable
entity mapping yet.

## Build an ambiguity-safe series

`series()` selects one semantic observation per `(value_year, period)` and
retains units, grain, and source coordinates:

```python
wage = data.series(
    "collections_monthly_city",
    entity_id="phl_city_wage_tax",
    column_role="fytd_actual",
)
```

The function does not aggregate by default. It raises
`AmbiguousObservationError` when a query spans incompatible tags, units,
table types, row kinds, organizational contexts, or repeated observations.
Narrow the query with semantic selectors; use `aggregate=...` only when the
remaining observations truly share one meaning.

The default `entity_id` selects the row's subject. For schedules where a
department is contextual metadata around program, class, or position rows,
set `entity_field="org_entity_id"` and use `filters={...}` to select the
intended fund, program, or other grain.

Declared organizational succession can be followed explicitly when the
selected dataset contains the predecessor observations:

```python
human_services = data.series(
    "pa_general_fund_enacted",
    entity_id="pa_human_services",
    entity_field="org_entity_id",
    column_role="budget_estimate",
    row_kind="total",
    follow_lineage="backward",
)
```

Lineage seams remain visible in the result. Undeclared splits or
apportionments fail rather than being guessed.

## Schema and provenance

Dataset families vary, and Parquet files are unioned by name. Common columns
fall into four groups:

| Group | Representative columns |
| --- | --- |
| Release and document identity | `document_id`, `document_type`, `jurisdiction`, `fiscal_year`, `period` |
| Observation | `entity_id`, `column_role`, `column_tag`, `value_year`, `value`, `value_unit` |
| Organizational context | `org_entity_id`, `ctx_department`, `ctx_fund`, `ctx_program` |
| Evidence and trust | `source_page`, `table_index`, `row_index`, `parser_name`, `table_reconciled`, `headers_verified` |

Call `describe(name).columns` for the authoritative physical schema in the
release you opened. During the alpha period, prefer named projections over
persisting assumptions about `SELECT *`.

Every released fact retains the source document id, one-based PDF/source page,
table, and row coordinates needed to identify its evidence. Source PDFs are
not bundled in the Python wheel or data archive. The release ships a
checksummed source catalog with acquisition URLs and exact source hashes, plus its entity
catalog, organizational lineage, authoritative extraction selection,
source-integrity decision, and optional annotation sidecars under the same
validated manifest.

## Correctness boundary

By default:

- source-quarantined documents are absent from the public release;
- exactly one authoritative extraction backend is selected per document;
- rows from tables that did not reconcile or whose headers were not verified
  are excluded;
- units and semantic context are retained in series output; and
- release files are checked against their recorded size and SHA-256 digest.

`include_unverified=True` and other diagnostic opt-ins are for investigation.
They do not promote those rows to trusted facts. Never aggregate mixed or
unknown units, and do not treat a raw display label as a stable identity.

## Local pipeline data and compatibility API

Repository contributors can still pass a pipeline artifacts root or packaged
release directly to the module-level functions:

```python
frame = gbd.load("qcmr_positions", source="/path/to/release")
datasets = gbd.datasets(source="/path/to/release")
```

Alternatively, set `GBD_DATA_ROOT`. An explicit `DataStore` from
`open_release()` is recommended for applications because every call remains
bound to one visible data snapshot.

The installed package version is exposed directly:

```python
import gov_budget_data as gbd

print(gbd.__version__)
```

## Development

The parser/normalization pipeline lives in the separate `gov-budget-etl`
workspace package. See the
[repository](https://github.com/nickhand/gov-budget-data) for development,
data-production, reconciliation, and release documentation.
