Metadata-Version: 2.4
Name: recon-gen
Version: 16.0.1
Summary: Outside validation that your financial system is sound — tells midsize financial institutions whether their books balance day to day, and when they don't, where to look first. Ships self-hosted HTMX dashboards and a regulator-ready PDF audit from one L2 institution model.
License-Expression: MIT
Project-URL: Homepage, https://chotchki.github.io/recon-gen/
Project-URL: Documentation, https://chotchki.github.io/recon-gen/
Project-URL: API Reference, https://recon-gen.readthedocs.io/en/latest/
Project-URL: Source, https://github.com/chotchki/recon-gen
Project-URL: Issues, https://github.com/chotchki/recon-gen/issues
Project-URL: Changelog, https://github.com/chotchki/recon-gen/blob/main/RELEASE_NOTES.md
Keywords: self-hosted,audit,reconciliation,analytics,finance
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Office/Business :: Financial :: Accounting
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: >=3.14
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.1
Requires-Dist: pyyaml>=6.0
Requires-Dist: graphviz>=0.20
Requires-Dist: duckdb>=1.1
Provides-Extra: prod
Requires-Dist: starlette>=1.3.1; extra == "prod"
Requires-Dist: uvicorn[standard]>=0.30; extra == "prod"
Requires-Dist: python-multipart>=0.0.31; extra == "prod"
Requires-Dist: httpx>=0.27; extra == "prod"
Requires-Dist: authlib>=1.3; extra == "prod"
Requires-Dist: pyjwt>=2.8; extra == "prod"
Requires-Dist: itsdangerous>=2.2; extra == "prod"
Requires-Dist: psycopg[binary,pool]>=3.1; extra == "prod"
Requires-Dist: oracledb>=3.4.0; extra == "prod"
Requires-Dist: pyarrow>=14; extra == "prod"
Requires-Dist: markdown>=3.6; extra == "prod"
Requires-Dist: openpyxl>=3.1; extra == "prod"
Requires-Dist: reportlab>=4.0; extra == "prod"
Requires-Dist: pypdf>=6.13.3; extra == "prod"
Requires-Dist: pyhanko>=0.30; extra == "prod"
Requires-Dist: mkdocs>=1.6; extra == "prod"
Requires-Dist: mkdocs-material>=9.5; extra == "prod"
Requires-Dist: mkdocs-click>=0.8; extra == "prod"
Requires-Dist: mkdocs-macros-plugin>=1.3; extra == "prod"
Requires-Dist: graphviz>=0.20; extra == "prod"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.5; extra == "dev"
Requires-Dist: filelock>=3.13; extra == "dev"
Requires-Dist: pytest-rerunfailures>=14.0; extra == "dev"
Requires-Dist: testcontainers[oracle,postgres]>=3.7; extra == "dev"
Requires-Dist: psycopg[binary,pool]>=3.1; extra == "dev"
Requires-Dist: oracledb>=2.0; extra == "dev"
Requires-Dist: reportlab>=4.0; extra == "dev"
Requires-Dist: pypdf>=6.13.3; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Requires-Dist: pyright>=1.1.380; extra == "dev"
Requires-Dist: playwright>=1.60; extra == "dev"
Requires-Dist: starlette>=1.3.1; extra == "dev"
Requires-Dist: uvicorn[standard]>=0.30; extra == "dev"
Requires-Dist: python-multipart>=0.0.31; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: httpx2>=2.5.0; extra == "dev"
Requires-Dist: markdown>=3.6; extra == "dev"
Requires-Dist: acme>=2.10.0; extra == "dev"
Requires-Dist: cryptography>=48.0.1; extra == "dev"
Requires-Dist: dnspython>=2.4; extra == "dev"
Requires-Dist: requests>=2.31; extra == "dev"
Requires-Dist: bcrypt>=4.0; extra == "dev"
Provides-Extra: e2e
Requires-Dist: pytest>=7.0; extra == "e2e"
Requires-Dist: pytest-xdist>=3.5; extra == "e2e"
Requires-Dist: playwright>=1.60; extra == "e2e"
Requires-Dist: starlette>=1.3.1; extra == "e2e"
Requires-Dist: uvicorn[standard]>=0.30; extra == "e2e"
Requires-Dist: python-multipart>=0.0.31; extra == "e2e"
Requires-Dist: httpx>=0.27; extra == "e2e"
Requires-Dist: httpx2>=2.5.0; extra == "e2e"
Requires-Dist: markdown>=3.6; extra == "e2e"
Dynamic: license-file

# Recon Generator

[![CI](https://github.com/chotchki/recon-gen/actions/workflows/ci.yml/badge.svg)](https://github.com/chotchki/recon-gen/actions/workflows/ci.yml)
[![Coverage](https://raw.githubusercontent.com/chotchki/recon-gen/badges/coverage-badge.svg)](https://github.com/chotchki/recon-gen/blob/badges/coverage-report.md)
[![PyPI](https://img.shields.io/pypi/v/recon-gen.svg)](https://pypi.org/project/recon-gen/)

## What it does

Recon Generator is an independent validation tool for midsize financial institutions: it tells you whether your books balance day to day, and when they don't, where to look first.

- Accounting is standard. We call it L1, meaning layer 1 in this tool.
- Your institution is not. We call it L2, meaning layer 2 in this tool.

Recon Generator layers the two: standard double-entry invariants on top of the unique shape you declare (your accounts, your rails, your multi-leg transfer templates, your bundling rules, your aging caps), so every way you actually move money is checked against the rules that govern it.

## Who it's for

No single role sees the whole reconciliation, so the tool carries a surface per role:

- **Integrators** — wiring the institution's shape into the tool (Studio editor, L2 Flow Tracing, Hygiene Exceptions).
- **Trainers** — shaping the demo and seeded scenarios so the dashboards exercise every path before go-live (data-shaping panel, scope knobs, plant overlays).
- **Operators** — driving the L1 invariants daily, walking exceptions back to their cause (L1 Dashboard, Daily Statement, L1 Exceptions).
- **Investigators / Executives** — compliance-AML triage and board-cadence rollups off the same base ledger (Investigation + Executives apps).

Every surface speaks YOUR institution's vocabulary — account names, role labels and persona prose all come from the L2 institution YAML, substituted into the rendered output. Swap the L2, the language follows.

**See it live — two public demos, no install:**

- **[Spec Example](https://recon-gen-spec.hotchkiss.io/)** — the smallest viable bank, dashboards only. The four bundled apps served by the self-hosted HTMX runtime.
- **[Sasquatch Bank Example](https://recon-gen-sasquatch.hotchkiss.io/)** — a fuller community-bank flavor, served through the **Studio** surface in read-only demo mode (L2 editor, unified diagram and data-shaping panel — every mutation locked down).

Both render straight from the bundled L2 YAMLs (`tests/l2/{spec_example,sasquatch_pr}.yaml`): read them to evaluate the tool, fork one to start your own. The full persona-driven handbooks, walkthroughs and per-sheet explainers your operators would see live at **[GitHub Pages](https://chotchki.github.io/recon-gen/)**; the Python API reference (tree primitives, dataset contract, db helpers, runner internals — everything ETL authors and integrators crib from) lives at **[ReadTheDocs](https://recon-gen.readthedocs.io/en/latest/)**.

## Architecture

Everything generates from one L2 YAML — your institution's shape — plus your ETL feed, through one shared core, into two renderers. The core is layered **L1 → L2 → L3**: persona-blind primitives, per-app assembly in domain vocabulary, then your persona / customer flavor.

```mermaid
flowchart LR
    YAML["L2 institution YAML<br/>accounts · rails · templates · chains · limits · theme"]
    FEED[("ETL feed<br/>transactions + daily_balances")]

    subgraph core["one shared core"]
      direction TB
      L1["L1 · persona-blind primitives<br/>common/tree · common/models · common/ids"]
      L2["L2 · per-app assembly, domain vocab<br/>apps: l1_dashboard · l2_flow_tracing · investigation · executives"]
      L3["L3 · persona / customer flavor<br/>apps/*/datasets.py SQL · L2 yaml persona block"]
      L1 --> L2 --> L3
    end

    YAML --> core
    FEED --> core
    core --> HTMX["Self-hosted HTMX<br/>Dashboards + Studio"]
    core --> PDF["Auditor-ready PDF<br/>audit report"]
```

Browse the full module tree on [GitHub](https://github.com/chotchki/recon-gen/tree/main/src/recon_gen); every module's API reference lives on [ReadTheDocs](https://recon-gen.readthedocs.io/en/latest/).

## Not an ETL tool

Recon Generator validates data; it doesn't move it. Your transactions and daily-balance feeds land in `<prefix>_transactions` and `<prefix>_daily_balances` (the Data Integration handbook documents the column contract), and Recon Generator reads from there.

We help you implement in two ways:

- **Wiring it in.** Mapping an upstream system into the L1 schema is real work (column mapping, type narrowing, metadata extraction, the supersession contract). The Data Integration handbook documents it column-by-column, and the Studio Deploy-changes pipeline carries an ETL hook so your existing extract plugs in without bolting code onto Recon Generator itself.
- **Synthetic scenarios on your real data.** Once your data is flowing, the test-data generator plants extra scenarios on top (drift events, overdraft breaches, stuck-pending aging, supersession trails, fanout patterns, anomaly spikes) so you can validate every L1 invariant without delaying go-live. Trainer knobs (`scope: full / uncovered_rails / exceptions_only / only_template`, `derive_balances`) shape what gets generated.

## Where it runs

Database backends — **PostgreSQL 17+** and **Oracle 19c+** for the on-prem / cloud-managed production targets, plus **DuckDB** as the zero-install integrator-laptop backend — a pure-Python wheel with an in-process vectorized executor and no server to stand up. The prior SQLite backend was dropped in v13.0.0 since it isn't optimized for analytics.

Two runtime surfaces — pick what your auditors and analysts already trust:

- **Self-hosted HTMX web app** — the four apps as dashboards you run yourself, offline, no external service in the loop. For sensitive deployments that can't reach external SaaS.
- **Auditor-ready PDF audit report** — printable and cryptographically fingerprinted (optionally pyHanko-signed). Same source data as the dashboards, and an end-of-pipeline 3-way agreement test (against the underlying sql) gates that they stay in agreement.

There was a third surface — an AWS QuickSight renderer — through v15.x. It's gone now, removed for cost-risk reasons, NOT because QuickSight is wrong (for an org already living in an AWS portal it's a fine call). If you need it, pin `recon-gen==15.x` with the `[quicksight]` extra; the v15 release notes and the [QuickSight quirks reference](https://chotchki.github.io/recon-gen/reference/quicksight-quirks/) carry the migration details.

## How it's tested

You can't ship a reconciliation tool on "trust me." This tool ships with:

- **Layered test gates** that run in order — unit → db → app2 — so a regression at layer N short-circuits before burning minutes on layer N+1.
- **Strong typing throughout** (Pyright strict on the core, NewType-wrapped identifiers and dataclass invariants), so an entire class of bug becomes a type error at the wiring site instead of a silent zero-row dashboard.
- **Fuzz testing as a property axis** — every test variant runs against random L2 institution shapes (`fuzz:N` for N seeds, pinned via `f<seed>_..` for repro), so the same invariants check against shapes nobody hand-wrote.
- **Deterministic, exhaustive test-data generation** — your L2 institution shape drives positive and negative scenarios that the harness plants automatically: drift, overdraft, limit breach, stuck-pending, stuck-unbundled, supersession audit, fanout, anomaly spikes, money-trail chains. Each scenario is shape-locked per `(L2 instance, dialect)`.
- **Cross-runtime parity** — the same scenario fans out into the self-hosted cell, the audit PDF and the underlying SQL — a 3-way agreement test gates that all three agree on every L1 invariant violation set (the drift the dashboard shows is the drift the PDF prints).

---

The CLI is four artifact groups — `recon-gen schema | data | docs | audit` — plus two server commands, `studio` and `dashboards` (below). Each artifact group runs `apply` / `clean` / `test` (audit adds `verify`, which recomputes a generated PDF's provenance fingerprint); anything destructive defaults to emit and needs `--execute` before it touches the DB or disk. Change the Python (or ask Claude) and refresh the page — you get a new dashboard.

## Demo Docs

- **[L1 Dashboard handbook](https://chotchki.github.io/recon-gen/handbook/l1/)** — 11 sheets covering 5 baseline L1 invariants + 2 aging-watch invariants + supersession audit + per-account-day walk + raw posting ledger. Switch the L2 instance to switch the persona prose without touching dashboard code.
- **[L2 Flow Tracing handbook](https://chotchki.github.io/recon-gen/handbook/l2_flow_tracing/)** — Rails / Chains / Transfer Templates / L2 Hygiene Exceptions for L2 spec verification.
- **[Investigation handbook](https://chotchki.github.io/recon-gen/handbook/investigation/)** — Compliance / Investigation team flow. 4 walkthroughs, one per sheet's question.
- **[Executives handbook](https://chotchki.github.io/recon-gen/handbook/executives/)** — board scorecard: account coverage, transaction volume, money moved.
- **[Data Integration handbook](https://chotchki.github.io/recon-gen/handbook/etl/)** — how the Data Integration Team maps an upstream system into `<prefix>_transactions` + `<prefix>_daily_balances`, validates the load and extends the metadata contract.
- **[Audit Reconciliation Report handbook](https://chotchki.github.io/recon-gen/handbook/audit/)** — auditor-ready PDF generated by `recon-gen audit apply`; covers the L1 invariants, embeds a provenance fingerprint, optionally auto-signs via pyHanko.

Source lives in `src/recon_gen/docs/` (shipped with the wheel — extract with `recon-gen docs export -o ./somewhere/`); rebuild locally with `recon-gen docs serve`.

## Quick start

### Prerequisites

- Python 3.14+
- A PostgreSQL 17+ / Oracle 19c+ / DuckDB database URL for demo mode (PG and Oracle use SQL/JSON path syntax — `JSON_VALUE` / `JSON_QUERY` / `JSON_EXISTS`; DuckDB uses `json_extract_string`)

### Install from PyPI

The base package — the self-hosted dashboards, DuckDB demo mode and the docs:

```bash
pip install recon-gen
```

For demo mode against PostgreSQL 17+ or Oracle 19c+ (the `prod` extra bundles both drivers — `psycopg[binary,pool]` and `oracledb` thin mode, no Oracle Instant Client install):

```bash
pip install "recon-gen[prod]"
```

For demo mode against DuckDB (no extra install — DuckDB ships as a pure-Python wheel in the base install):

```bash
pip install recon-gen
```

> The package was renamed from `quicksight-gen` to `recon-gen` in v11.0.0. The `quicksight-gen` shim stopped publishing at v15 — `pip install quicksight-gen` now pulls only the last pre-v15 shim (which pins an older `recon-gen`), so switch to `recon-gen` directly.

### Setup from source

The repo uses [uv](https://docs.astral.sh/uv/) for env / lock management
(deterministic resolution from `uv.lock`). One command sets up `.venv/`
with every extra:

```bash
uv sync --all-extras
```

Then invoke tools directly via the venv (no `source activate` needed):

```bash
.venv/bin/pytest
.venv/bin/recon-gen --help
```

For a leaner install, swap `--all-extras` for the three real extras (collapsed from eight in BS.6 — one knob per persona):

- `--extra dev` (unit tests + pyright)
- `--extra prod` (everything a production run needs — DB drivers, the self-hosted server, PDF + docs)
- `--extra e2e` (Playwright for the browser layer)

If you'd rather stick with pip, the standard PEP-621 path still works:

```bash
python3 -m venv .venv
.venv/bin/pip install -e ".[dev]"
```

### Configure

> **v14.0.0 cfg shape (shipped 2026-06-14).** Replaced the previous flat-field shape with concern-grouped nested blocks (`db:` / `app2:` / `audit:` / `test:`) and `extends:` inheritance for base + per-env overlays. Field accessors are `cfg.db.url` / `cfg.db.dialect` / `cfg.db.table_prefix` etc. Full migration map (every v13 key → its v14 path): [`docs/audits/de_0_cfg_redesign.md#migration-v13x--v1400-hard-break`](docs/audits/de_0_cfg_redesign.md#migration-v13x--v1400-hard-break).

See [`config.example.yaml`](https://github.com/chotchki/recon-gen/blob/main/config.example.yaml)

```bash
cp config.example.yaml config.yaml
```

`extends:` lets you compose a base cfg with per-env overlays — child wins, dicts deep-merge:

```yaml
# config.prod.yaml
extends: ./config.base.yaml
db:
  url: "postgresql://prod-user:pass@prod-host:5432/recon"
  table_prefix: "recon_prod"
```

> Theme is declared inline on the L2 institution YAML's `theme:` block, not the run config. When the L2 instance carries no `theme:` block, the `DEFAULT_PRESET` in-canvas-accent fallback takes over (silent-fallback contract).

All values can also be set via `RECON_GEN_`-prefixed environment variables (e.g. `RECON_GEN_DEMO_DATABASE_URL` / `RECON_GEN_DIALECT`). Env vars override YAML.

## Demo mode

A deterministic demo generator seeds the four apps so you can see them work without wiring up real data. Every app feeds two per-prefix base tables — `<db_table_prefix>_transactions` (every money-movement leg) and `<db_table_prefix>_daily_balances` (per-account end-of-day snapshots), where `<db_table_prefix>` is `cfg.db.table_prefix` (required).

```bash
# Apply schema + seed to your demo database, then serve the dashboards.
# Requires: db.url + db.dialect in config.yaml and the `[prod]` extra
# installed (bundles psycopg + oracledb; DuckDB needs no extra).
# Per-prefix DDL + seed are emitted at apply time using cfg.db.table_prefix.
recon-gen schema apply -c config.yaml --execute   # tables + matviews
recon-gen data apply   -c config.yaml --execute   # 90-day baseline + plants
recon-gen data refresh -c config.yaml --execute   # populate matviews
recon-gen audit apply  -c config.yaml --execute -o report.pdf  # auditor-ready PDF (optional)
recon-gen dashboards   -c config.yaml             # serve the four apps at /dashboards
```

The self-hosted server re-runs every query on each page load (no cache), so seed changes show up immediately after a fresh `data apply --execute` + `data refresh --execute`.

### Demo scenarios

Two L2 institution YAMLs ship in `tests/l2/`:

- **`spec_example.yaml`** — the persona-neutral default fixture. Generic accounts/rails/chains exercising every L2 primitive without naming a specific institution.
- **`sasquatch_pr.yaml`** — a flavored Sasquatch National Bank persona block carrying the curated demo narrative: SNB control accounts, templated merchant DDAs, Investigation anchor (Juniper Ridge LLC) with three converging scenarios (12-sender fanout cluster, a Cascadia Trust Bank Operations → Juniper anomaly spike, 4-hop layering chain through shell entities).

Pass `--l2 tests/l2/sasquatch_pr.yaml` (or your own) to switch the rendered handbook + demo data narrative without touching dashboard code.

## Self-hosted: Dashboards and Studio

The four apps render off the same L2 instance through the self-hosted stack: an HTMX + d3 server that reads the database directly, no external service in the loop. It comes at two depths.

**Dashboards** is the lean read-only mount — one process serves all four apps plus the handbook at `/docs`:

```bash
pip install 'recon-gen[prod]'
recon-gen dashboards -c config.yaml                # one process, all 4 apps + the handbook at /docs
# → http://127.0.0.1:8765/dashboards
```

It speaks all three SQL dialects (PostgreSQL / Oracle / DuckDB); point `db.url` at any of them. The schema + seed have to already be applied (`schema apply --execute`, `data apply --execute`, `data refresh --execute`) — Dashboards only reads. It's stateless: every GET re-runs the query, filter state round-trips as `?param_X=…` query params (so the URL is the cache key), no auth/sessions — put it behind your own auth front on a network. All browser-side assets (htmx, d3, the filter widgets) ship inside the wheel — it runs offline.

**Studio** (`recon-gen studio`) is everything Dashboards mounts plus the implementation surface we hand integrators, trainers and ETL engineers — the L2-YAML editor, the unified diagram (your accounts / rails / chains / templates as a graph you edit in place), the data-shaping panel (trainer knobs + scenario plants) and Deploy-changes orchestration with an ETL hook. The YAML on disk stays the source of truth; every save is an atomic write. This is the offline-iteration loop — edit the shape and refresh the page, no deploy cycle, no external round-trip.

A 3-way agreement test (`scenario plants ⊆ direct matview SELECT == Dashboards`, `== audit PDF` where it applies) gates the release, so the dashboards match the underlying SQL on every L1 invariant violation set — enforced, not just claimed. Full reference — what ships in the wheel, the maintainer recipes for bumping a vendored asset — in the handbook's [Self-hosting the dashboards](https://chotchki.github.io/recon-gen/reference/self-host/) page.

## Theming

Theme is declared inline on the L2 institution YAML's `theme:` block. When the L2 instance carries no `theme:` block, `build_theme` returns `None` and the single `DEFAULT_PRESET` in `common/theme.py` — the in-canvas-accent fallback — takes over (silent-fallback contract). No registry, no CLI flag.

To customize the demo persona's brand, launch studio and use its built-in editor!

## Tests

```bash
./run_tests.sh up_to=unit                                  # ~20s, no DB
./run_tests.sh up_to=db                                    # db layer, xdist-parallel
./run_tests.sh up_to=db --only test_drift                  # narrow within a layer (pytest -k)
./run_tests.sh up_to=app2                                  # full chain through Playwright
```

The runner enforces ordering — invoking layer N runs layers 1..N-1 first. Layers: `unit → db → app2`. See `CLAUDE.md::Test sequencing` for the full guide.

**Triage a specific failing test:** `./run_tests.sh triage <pytest_nodeid>` spawns the appropriate DB container and drops into a screen-attached pdb at the failure line. Multi-client — both operator and assistant can drive pdb via `screen -x recon-gen-triage` / `screen -S recon-gen-triage -X stuff $'<cmd>\n'`. Teardown: `./run_tests.sh triage-down --yes` (kills the screen session and stops the triage container). Full runbook: `CLAUDE.md::Triage workflow` section.

Coverage:

- **Unit / integration**: models, tags, config, CLI, demo determinism + scenario coverage (per-instance SHA256 seed-hash locks), tree primitives + validators, dataset builders, visual builders, filter groups, cross-reference validation (dataset identifiers, filter bindings, visual ID uniqueness, sheet scoping), explanation coverage, schema + seed SQL structure for both Postgres + Oracle.
- **E2E**: the app2 browser layer collects by default (the standalone `RECON_GEN_E2E=1` gate was retired in DJ.1).
  - *Browser layer (Playwright WebKit, headless)* — the self-hosted dashboards load, sheet tabs, per-sheet visual counts + spot-checked titles, drill-downs, mutual-filter reconciliation tables, date-range filter narrowing, Show-Only-X toggles, Investigation slider + dropdown filters.

E2E tunables (env vars): `RECON_E2E_PAGE_TIMEOUT`, `RECON_E2E_VISUAL_TIMEOUT`. Failure screenshots land in `tests/e2e/screenshots/<app>/` (gitignored).
