Metadata-Version: 2.4
Name: civic-data
Version: 0.1.1
Summary: Cleaned, documented civic open data — a DuckDB/GeoParquet consumer CLI plus the extraction/normalization spine that builds it, with a generated data dictionary and transformation-lineage layer.
Author: NewGuyAI
License: MIT
Project-URL: Homepage, https://newguyai.com
Keywords: civic-data,open-data,duckdb,geoparquet,louisville,data-dictionary
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Scientific/Engineering :: GIS
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE-data
Requires-Dist: duckdb>=1.1
Requires-Dist: pytz>=2024.1
Provides-Extra: producer
Requires-Dist: psycopg2-binary>=2.9; extra == "producer"
Requires-Dist: httpx>=0.27; extra == "producer"
Requires-Dist: python-dotenv>=1.0; extra == "producer"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# civic-data

**Cleaned, normalized, and documented public open data — with a generated, code-coupled data
dictionary and transformation-lineage layer, consumable as a GeoParquet lake with a self-describing
CLI.**

The city gives away raw open data. civic-data's value is not the data — it's the honest,
reproducible record of **what we did to it, why, and what's lost**: schema, grain, NULL rates,
per-year coverage, quarantine loss, bias caveats, and a step-by-step transformation lineage that
doubles as the reproduction recipe. Because that record is *generated from the live database and the
ingest code*, it cannot drift from reality.

v1 covers **Louisville, KY**: ten domain datasets (crime, 311 service requests, landbank sales,
lien orders, foreclosures, food-service establishments, ABC licenses, restaurant inspections,
construction permits, short-term-rental licenses) plus the geography spine (ZIP boundaries, council
districts, neighborhoods, HUD zones, crosswalks, Census population) — ~3.9M rows, all ingested from
source, all lineage-documented (the catalog carries the exact counts).

## Quick start (60 seconds, no database required)

```bash
pip install civic-data             # consumer install — DuckDB only, no Postgres
civic-data cities                  # list published city lakes
civic-data get louisville          # download the lake (sha256-verified) into ./export
civic-data query -d crime --zip 40202 --limit 5
```

## Consuming the data

The consumer product is a **GeoParquet lake** described by a machine-readable `catalog.json`, read
with DuckDB — no server, no Docker, no Postgres. Every verb takes `--json` (for agents) and
`--lake <dir>` (default: `./export`):

```bash
civic-data catalog --json          # the discoverability contract — start here
civic-data datasets                # list datasets by catalog key
civic-data schema crime            # one dataset's columns + shape
civic-data query -d crime --zip 40202 --year 2024 --limit 5
civic-data query -d crime --district 6          # point-in-polygon; geocoded rows only (it says so)
civic-data place 40202             # one place across every dataset: identity + counts
civic-data sql "SELECT zip_code, count(*) FROM crime GROUP BY 1"   # raw DuckDB; keys are views
civic-data query -d crime --sources   # provenance: source pulls, license, how-to-cite
```

**Hand someone a place as a folder:** `civic-data place 6 --export` writes a self-contained
mini-lake (filtered Parquet + the district's boundary/crosswalks/population + a scoped catalog +
generated `llms.txt`/`AGENTS.md`). Every verb above works against that folder via `--lake`.

The lake root also carries generated **`llms.txt`** and **`AGENTS.md`** so an AI agent can bootstrap
itself from the artifact alone. Both are regenerated from the catalog on every export — never
hand-edited (a test enforces it).

## Producing the data (the reproduction path)

```bash
docker compose up -d              # PostgreSQL 16 + PostGIS on host port 5434
python -m venv .venv && . .venv/Scripts/activate   # (Windows: .venv\Scripts\activate)
pip install -e .[producer]        # the pipeline extras (Postgres driver, HTTP client)

civic-data ingest-zips                          # canonical ZIP boundary set (MDM) — run first
civic-data ingest-refs                          # LOJIC geocoding references (address points + intersections)
civic-data ingest-geo && civic-data ingest-population   # boundaries, crosswalks, Census ACS
civic-data ingest crime --all --promote         # source -> bronze -> silver (any of the 10 domains)
civic-data geocode-crime --write --apply        # place ~1.2M crime records (geom + derived ZIP)
civic-data datadict --with-source-audit         # generate the data dictionary + lineage
civic-data export                               # silver -> GeoParquet lake + catalog + agent docs
```

The generated dictionary lands in [`docs/data-dictionary/`](docs/data-dictionary/) — a human-readable
`README.md` and a machine-readable `data-reality.json` (diffing successive runs is drift detection).

Ingest order matters: `ingest-zips` first. The ZIP set is the canonical geography (MDM) every record
validates against; crime/311 ingest refuse to run without it.

## Honesty, by construction

- **Trust tiers** on every dataset (`trendable` / `viewable_caveated` / `snapshot_only`) — a current
  roster is not a time series, and the catalog says which is which.
- **Documented loss, not silent loss:** quarantine counts, geocode coverage (spatial filters match
  geocoded rows only, and the CLI says so), source-shape divergences recorded as findings.
- **Generated, never hand-maintained:** the dictionary, the catalog, and the agent docs are all
  derived from the live database + code; drift guards raise when authored prose falls out of sync.

## Licensing

- **Code:** MIT — see [`LICENSE`](LICENSE).
- **Data & documentation:** CC BY 4.0 — see [`LICENSE-data`](LICENSE-data).

The underlying raw data is **public record, provided as-is**, and its terms were verified per origin
(2026-07-11): Louisville Metro publishes under the **PDDL** (public domain; when republishing
modified data, identify the source, version, and modifications — which the generated lineage layer
does per dataset); LOJIC layers are public domain (*Mapping Data Source: LOJIC*); Census ACS is a
U.S. government work — *this product uses the Census Bureau Data API but is not endorsed or
certified by the Census Bureau.* Every dataset's exact license text ships in its catalog provenance:
`civic-data query -d <dataset> --sources`.

## Provenance

Born out of civic-graph (the private flagship built on top of this layer, and the origin of this
spine). See `wiki.md` for the source of truth and `ai-working/plans/` for the phase plans.
