Metadata-Version: 2.5
Name: civicpay
Version: 0.1.1
Summary: CivicPay Open Framework — clean-room, open-source reference implementation of financial-data-governance methodology (reconciliation, data quality, exception workflow, audit evidence).
Author: CivicPay Open Framework contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: audit,data-quality,duckdb,finance,open-source,reconciliation
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Financial and Insurance Industry
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Topic :: Database
Requires-Python: >=3.11
Requires-Dist: duckdb>=1.1.0
Requires-Dist: faker>=24.0.0
Requires-Dist: pandas>=2.1.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: rapidfuzz>=3.9.0
Requires-Dist: rich>=13.7.0
Requires-Dist: streamlit>=1.36.0
Requires-Dist: typer>=0.12.0
Provides-Extra: dbt
Requires-Dist: dbt-core>=1.12.0; extra == 'dbt'
Requires-Dist: dbt-duckdb>=1.11.0; extra == 'dbt'
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.5.0; extra == 'dev'
Description-Content-Type: text/markdown

# CivicPay Open Framework

A clean-room, open-source reference implementation of financial-data-governance methodology — payment reconciliation, data-quality monitoring, exception workflow, and a tamper-evident audit-evidence layer — for U.S. community banks, credit unions, and small fintechs.

> **Status:** Pre-alpha (v0.1). Research/teaching reference, not production software.

## Why this exists

Financial-sector data-integrity failures — unreconciled payments, data-quality defects, missing audit trails — can cause real consumer harm and have been the subject of significant regulatory enforcement actions across the U.S. financial sector. The institutions most exposed to these failure modes (community banks, credit unions, small fintechs) are often the least able to afford enterprise tooling. CivicPay Open Framework makes a practitioner-informed reconciliation / data-quality / audit-evidence methodology freely adoptable on synthetic data, using only open-source components.

## What it is (and is not)

CivicPay Open Framework is a **governance toolset**, not a payment processor or core banking system. It runs entirely on **synthetic data**. See [DISCLAIMER.md](DISCLAIMER.md) and [SECURITY.md](SECURITY.md).

- Not a payment processor, money transmitter, or core banking system.
- Not Reg E / BSA / AML compliance advice.
- Not production-ready; not for real PII without an institution-specific security and compliance review.

## Architecture

```text
┌─────────────────────────────────────────────────────────────────────┐
│       0. Enrollment & Validation  (point of capture, v0.2)           │
│       form / CLI → validate → dual-source gate → accept or mismatch   │
└──────────────────────────────────┬────────────────────────────────────┘
                                    │ mismatch routes below
                                    ▼
┌─────────────────────────────────────────────────────────────────────┐
│                         Streamlit Dashboard                        │
│  (recon summary · DQ scores · exceptions · audit log · enrollment)   │
├──────────────┬────────────────┬──────────────────┬───────────────────┤
│  1. Recon    │   2. Data-      │  3. Exception    │  4. Audit-        │
│  matching    │   quality       │  workflow        │  evidence         │
│  + outcome   │   checks +      │  priority + SLA  │  hash-chained     │
│  counts      │   quality score │  aging + resolve │  log + export      │
├──────────────┴────────────────┴──────────────────┴───────────────────┤
│                    DuckDB (embedded, synthetic data)                │
└─────────────────────────────────────────────────────────────────────┘
        determinism (seed=)              tamper-evidence (SHA-256 chain)
```

Layer 0 is upstream of the original four: it prevents dirty data from ever
reaching the ledger, complementing layers 1–4's post-hoc detection.

1. **Payment Reconciliation** — match inbound payment files against ledger entries; classify matched / unmatched / exception.
2. **Data-Quality Monitoring** — completeness, accuracy, consistency, timeliness, anomaly checks; per-dataset quality score.
3. **Exception Workflow** — priority-ranked queue, SLA aging, resolution with root-cause capture.
4. **Audit-Evidence Layer** — append-only, hash-chained (tamper-evident) event log; exportable evidence packages.
5. **Enrollment & Validation** (v0.2) — a point-of-capture complement to the four layers above: a constrained-input Streamlit form and CLI batch path validate candidate records, then a dual-source agreement gate (one pure-Python path, one independent SQL path) accepts a record only when both agree — otherwise it's routed into the same exception workflow for human review.

## Tech stack

Python 3.11+ · DuckDB (embedded, no infrastructure) · dbt (v0.2) · Streamlit · Faker · Typer · RapidFuzz · pytest · ruff.

## Install

```bash
pip install -e ".[dev]"

# Optional: dbt analytical marts (see docs/dbt.md) — not needed for core CLI usage
pip install -e ".[dbt]"
```

## Quickstart

One command runs the full pipeline end-to-end (seed → reconcile → DQ → exceptions → audit verify):

```bash
civicpay run-all
```

Or step by step:

```bash
# 1. Generate deterministic synthetic data and load into DuckDB
civicpay seed --seed 42

# (Or write raw CSV/Parquet only, without loading)
civicpay seed --seed 42 --files-only

# 2. Run payment reconciliation against the ledger
civicpay recon run --batch-id BATCH-001

# 3. Run data-quality checks across all datasets
civicpay dq check --batch-id DQ-001
# (single dataset)  civicpay dq check --dataset transactions

# 4. View the exception queue (sorted by computed priority, with SLA aging)
civicpay exception list --status open
# Resolve one, capturing root cause (emits an audit event)
civicpay exception resolve --id EXC-DQ-001-000001 --root-cause "stale upstream feed"

# 5. Verify the audit hash chain is intact (detects tampering)
civicpay audit verify --batch BATCH-001
# Export a tamper-evident evidence package as JSON
civicpay audit export --batch BATCH-001 --out evidence.json

# Launch the dashboard
civicpay dashboard
```

### dbt analytical marts (v0.2)

```bash
pip install -e ".[dbt]"     # dbt-core + dbt-duckdb, not installed by default
civicpay dbt run             # build mart_recon_summary, mart_dq_summary, mart_exception_aging
civicpay dbt test            # run the dbt schema/data tests
```

Runs directly against the same DuckDB file the pipelines write to — no warehouse. See [docs/dbt.md](docs/dbt.md) for what each mart mirrors and why.

### Enrollment & validation (v0.2)

```bash
# Launch the constrained-input Streamlit enrollment form
civicpay enroll

# Or validate a batch: the seeded pending_enrollments pool by default,
# or an external CSV via --file
civicpay enroll validate
civicpay enroll validate --file records.csv
```

Rules (program caps, term ranges, regions, dual-source tolerance) live in
[`civicpay/config/enrollment_rules.yml`](civicpay/config/enrollment_rules.yml). Accepted records
land in `accepted_enrollments`; dual-source disagreements route to the same
exception queue as recon/DQ (`civicpay exception list/resolve`, or the
dashboard's Enrollment & Validation section, which can also resolve a
mismatch by accepting either computed value or rejecting it for re-entry).

### Data quality configuration

DQ checks are configured in [`civicpay/config/dq_checks.yml`](civicpay/config/dq_checks.yml) — per-dataset check definitions (type, rule, params), type weights, and the exception-routing cap. See [docs/data-quality.md](docs/data-quality.md) for the full check catalogue.

### Determinism & re-runs

Data generation and all pipelines are deterministic given a fixed `--seed`. Each run should use a **fresh `--batch-id`** (the audit log and exception queue are append-only; re-running the same batch id on the same DB collides on primary keys).

See [docs/reconciliation.md](docs/reconciliation.md) for the reconciliation module's full technical documentation (algorithm, data model, configuration, API, test results, and operational behavior). See [docs/data-quality.md](docs/data-quality.md) for the DQ module, [docs/exceptions.md](docs/exceptions.md) for the exception workflow (priority, SLA aging, the demo backlog cohort), [docs/audit.md](docs/audit.md) for the audit-evidence layer (hash chaining, evidence export, and two real bugs found and fixed in it), [docs/architecture.md](docs/architecture.md) for the module status table, and [docs/methodology.md](docs/methodology.md) for the overall design.

## Clean-room provenance

This project is developed clean-room: from public regulatory requirements, published technical standards, and original design — never from any employer's proprietary systems, code, or data. See [PROVENANCE.md](PROVENANCE.md).

## License

Apache-2.0. See [LICENSE](LICENSE).
