Metadata-Version: 2.4
Name: pycharter
Version: 0.0.67
Summary: Runtime data contract enforcement for Python: one Pydantic-native contract validates API objects, Kafka streams, and ETL batches in-process — with quality scoring, a versioned contract registry, and ODCS round-trip.
Author-email: Optophi <contact@optophi.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/optophi/pycharter
Project-URL: Documentation, https://optophi.github.io/pycharter/
Project-URL: Repository, https://github.com/optophi/pycharter
Project-URL: Issues, https://github.com/optophi/pycharter/issues
Project-URL: Contributing, https://github.com/optophi/pycharter/blob/main/CONTRIBUTING.md
Keywords: data-contract,data-contracts,runtime-validation,data-quality,data-governance,json-schema,pydantic,odcs,open-data-contract-standard,kafka,streaming,etl,validation,schema-registry,schema
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Database
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic<3.0,>=2.0.0
Requires-Dist: jsonschema>=4.26.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: sqlalchemy<3.0,>=2.0.0
Requires-Dist: alembic<2.0,>=1.13.0
Requires-Dist: jsonata-python>=0.6.0
Requires-Dist: httpx<1.0,>=0.24.0
Requires-Dist: networkx>=3.0
Provides-Extra: server
Requires-Dist: fastapi<0.142,>=0.136.3; extra == "server"
Requires-Dist: uvicorn[standard]<1.0,>=0.24.0; extra == "server"
Requires-Dist: starlette>=1.2.1; extra == "server"
Requires-Dist: pydantic-settings<3.0,>=2.14.2; extra == "server"
Requires-Dist: python-multipart>=0.0.6; extra == "server"
Requires-Dist: PyJWT<3.0,>=2.8.0; extra == "server"
Requires-Dist: httpx<1.0,>=0.24.0; extra == "server"
Requires-Dist: aiofiles>=25.1.0; extra == "server"
Requires-Dist: prometheus-client>=0.19.0; extra == "server"
Requires-Dist: psycopg[binary]>=3.3.4; extra == "server"
Provides-Extra: pipeline
Requires-Dist: sshtunnel>=0.4.0; extra == "pipeline"
Requires-Dist: boto3>=1.26.0; extra == "pipeline"
Requires-Dist: google-cloud-storage>=3.12.0; extra == "pipeline"
Requires-Dist: azure-storage-blob>=12.0.0; extra == "pipeline"
Requires-Dist: openpyxl>=3.1.5; extra == "pipeline"
Requires-Dist: lxml>=4.9.0; extra == "pipeline"
Requires-Dist: pyarrow>=14.0.0; extra == "pipeline"
Requires-Dist: pymongo>=4.0.0; extra == "pipeline"
Requires-Dist: openlineage-python>=1.0.0; extra == "pipeline"
Provides-Extra: streaming
Requires-Dist: websockets>=12.0; extra == "streaming"
Requires-Dist: watchfiles>=0.21.0; extra == "streaming"
Requires-Dist: prometheus_client>=0.19.0; extra == "streaming"
Requires-Dist: aiohttp>=3.9.0; extra == "streaming"
Requires-Dist: aiokafka>=0.9.0; extra == "streaming"
Requires-Dist: aio-pika>=9.0.0; extra == "streaming"
Requires-Dist: redis>=5.0.0; extra == "streaming"
Requires-Dist: python-socketio>=5.11.0; extra == "streaming"
Provides-Extra: semantic
Requires-Dist: linkml>=1.7.0; extra == "semantic"
Requires-Dist: linkml-runtime>=1.7.0; extra == "semantic"
Requires-Dist: rdflib>=7.0.0; extra == "semantic"
Requires-Dist: owlrl>=6.0.0; extra == "semantic"
Requires-Dist: neo4j<7.0.0,>=6.2.0; extra == "semantic"
Provides-Extra: extraction
Requires-Dist: tree-sitter<1.0.0,>=0.25.2; extra == "extraction"
Requires-Dist: tree-sitter-python<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-javascript<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-typescript<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-go<1.0.0,>=0.23.0; extra == "extraction"
Requires-Dist: tree-sitter-rust<1.0.0,>=0.24.2; extra == "extraction"
Requires-Dist: tree-sitter-java<1.0.0,>=0.23.5; extra == "extraction"
Requires-Dist: markdown-it-py<5.0.0,>=4.2.0; extra == "extraction"
Requires-Dist: pypdf<8.0.0,>=4.0.0; extra == "extraction"
Requires-Dist: sqlglot>=25.0; extra == "extraction"
Provides-Extra: ai
Requires-Dist: anthropic>=0.34.0; extra == "ai"
Requires-Dist: openai>=2.44.0; extra == "ai"
Requires-Dist: mcp<3,>=1.0; extra == "ai"
Provides-Extra: otel
Requires-Dist: opentelemetry-api<2,>=1.20; extra == "otel"
Provides-Extra: cli
Requires-Dist: rich<16.0,>=13.0.0; extra == "cli"
Provides-Extra: docs
Requires-Dist: mkdocs<2,>=1.5.3; extra == "docs"
Requires-Dist: mkdocs-material[imaging]>=9.7.6; extra == "docs"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "docs"
Requires-Dist: pymdown-extensions>=10.0.0; extra == "docs"
Requires-Dist: mike>=2.0.0; extra == "docs"
Provides-Extra: airflow
Requires-Dist: apache-airflow<4,>=3.2.2; extra == "airflow"
Provides-Extra: dagster
Requires-Dist: dagster>=1.7; extra == "dagster"
Provides-Extra: prefect
Requires-Dist: prefect<4,>=3.0; extra == "prefect"
Provides-Extra: postgres
Requires-Dist: asyncpg>=0.29; extra == "postgres"
Requires-Dist: psycopg[binary]>=3.3.4; extra == "postgres"
Provides-Extra: all
Requires-Dist: pycharter[ai,cli,docs,extraction,otel,pipeline,postgres,semantic,server,streaming]; extra == "all"
Provides-Extra: ci
Requires-Dist: pycharter[extraction,pipeline,postgres,semantic,server,streaming]; extra == "ci"
Requires-Dist: testcontainers[postgres]>=4.0.0; extra == "ci"
Requires-Dist: testcontainers[kafka]>=4.0.0; extra == "ci"
Requires-Dist: pytest>=7.0.0; extra == "ci"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "ci"
Requires-Dist: pytest-cov>=4.0.0; extra == "ci"
Requires-Dist: pytest-timeout>=2.1.0; extra == "ci"
Requires-Dist: pytest-xdist>=3.5.0; extra == "ci"
Requires-Dist: pytest-randomly>=3.15.0; extra == "ci"
Requires-Dist: ruff>=0.15.20; extra == "ci"
Requires-Dist: mypy>=1.0.0; extra == "ci"
Requires-Dist: types-PyYAML>=6.0.0; extra == "ci"
Requires-Dist: requests>=2.32.3; extra == "ci"
Requires-Dist: pre-commit>=3.0.0; extra == "ci"
Requires-Dist: mkdocs<2,>=1.5.3; extra == "ci"
Requires-Dist: mkdocs-material[imaging]>=9.7.6; extra == "ci"
Requires-Dist: mkdocstrings[python]>=0.24.0; extra == "ci"
Requires-Dist: pymdown-extensions>=10.0.0; extra == "ci"
Provides-Extra: dev
Requires-Dist: pycharter[all]; extra == "dev"
Requires-Dist: requests>=2.32.3; extra == "dev"
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.1.0; extra == "dev"
Requires-Dist: pytest-xdist>=3.5.0; extra == "dev"
Requires-Dist: pytest-benchmark>=4.0.0; extra == "dev"
Requires-Dist: testcontainers[kafka]>=4.0.0; extra == "dev"
Requires-Dist: testcontainers[postgres]>=4.0.0; extra == "dev"
Requires-Dist: opentelemetry-sdk<2,>=1.20; extra == "dev"
Requires-Dist: ruff>=0.15.20; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Requires-Dist: types-PyYAML>=6.0.0; extra == "dev"
Requires-Dist: pre-commit>=3.0.0; extra == "dev"
Requires-Dist: build>=0.10.0; extra == "dev"
Requires-Dist: twine>=4.0.0; extra == "dev"
Requires-Dist: marimo>=0.9.0; extra == "dev"
Requires-Dist: vulture>=2.11; extra == "dev"
Requires-Dist: deptry>=0.20; extra == "dev"
Requires-Dist: pip-audit>=2.7; extra == "dev"
Dynamic: license-file

# PyCharter

> **PyCharter enforces data contracts where your code runs.**

Define a contract once — from a **Pydantic model** (`pycharter contract
from-model`) or versioned **YAML** — and the same artifact validates API
payloads in-process, gates Kafka consumers record by record, and checks ETL
batches at stage boundaries. Deterministic, Pydantic-native, in your process:
no warehouse round-trip, no separate CI engine deciding after the fact.
**The contract is what actually runs.**

Most contract tooling stops earlier in the lifecycle: datacontract-cli — the
ODCS reference implementation — verifies contracts in CI and batch; Soda and
Great Expectations scan the warehouse; dbt tests run after materialisation.
PyCharter is the **runtime half** of that ecosystem: it executes contracts
inside the producing service, and round-trips **ODCS** (v3.1.0 export, v3.0.x /
v3.1.x import) so the contracts you enforce at runtime slot into the tools you
already use.

**Full documentation:** [https://optophi.github.io/pycharter/](https://optophi.github.io/pycharter/)

[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

**[Where PyCharter fits vs datacontract-cli, Pydantic, Pandera, GX, Soda, dbt →](docs/getting-started/comparisons.md#where-pycharter-fits)**

---

## Validate a record in 30 seconds

```bash
pip install pycharter
```

```python
from pycharter import Validator

validator = Validator.from_dict(
    {
        "type": "object",
        "version": "1.0.0",
        "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"},
        },
        "required": ["name", "age"],
    }
)

result = validator.validate({"name": "Alice", "age": 30})
print(result.is_valid)  # True

result = validator.validate({"name": "Bob", "age": "N/A"})
print(result.errors[0])  # Column 'age': expected integer, found string 'N/A'
```

That's the library at its smallest — no database, no setup. In production
you'd load the same contract from a file (`Validator.from_file("contract.yaml")`)
or a shared store. Failures come back as structured diagnostics (row, column,
expected type, offending value), and the generated model validates as fast as
a hand-written Pydantic model (~0.7 µs/record —
[measured](docs/operations/benchmarks.md)).

Prefer the shell? The same check works on a whole file:

```bash
pycharter validate contract.yaml data.jsonl   # colorized report, exit 0/1
```

---

## Why PyCharter

Teams often maintain a Pydantic model at the API, a Pandera schema in the
pipeline, a Great Expectations suite for quality, and an ODCS YAML as the
"contract" — and they drift. PyCharter unifies authoring and enforcement so
the contract is what actually runs, **where** it runs being the point:

| Tool | Where enforcement runs | Object | Dataset | Quality | Contract | Versioned |
|------|------------------------|:------:|:-------:|:-------:|:--------:|:---------:|
| Pydantic | In-process (single objects) | ● | ○ | ○ | ◐ | ◐ |
| Pandera | In-process (DataFrames) | ◐ | ● | ◐ | ◐ | ◐ |
| Great Expectations | Warehouse / batch jobs | ○ | ● | ● | ◐ | ◐ |
| Soda | Warehouse-side, scheduled scans | ○ | ● | ● | ◐ | ◐ |
| dbt tests | Warehouse, post-materialisation | ○ | ● | ◐ | ◐ | ● |
| datacontract-cli (ODCS ref. impl.) | CI / batch | ○ | ◐ | ◐ | ● | ● |
| **PyCharter** | **In-process runtime: APIs, streams, ETL** | **●** | **●** | **●** | **●** | **●** |

<sub>● full · ◐ partial · ○ none — markers reflect scope, not quality; see the
[full comparison](docs/getting-started/comparisons.md#where-pycharter-fits)</sub>

- **One definition, every surface** — model or YAML → API gate, stream gate,
  ETL steps, quality jobs, ODCS round-trip.
- **Enforces, not just authors** — rejects or coerces bad records *before*
  they land; warehouse-side engines measure what *did* land.
- **Deterministic and auditable** — versioned contracts, reproducible runs,
  thread-safe shared validators.

Choose Pydantic alone for one-service API shape checks, Pandera for
DataFrame validation in one process, GX/Soda for warehouse-centric batch
quality, datacontract-cli for ODCS authoring and CI gates — and PyCharter
when the contract must be the enforced single source of truth across
services. Full narrative and per-tool sections:
**[PyCharter vs. the alternatives](docs/getting-started/comparisons.md)**.

---

## Gate a FastAPI endpoint

```bash
pip install pycharter[server]
```

```python
from fastapi import Depends, FastAPI
from pycharter.contrib.fastapi import contract_dependency

app = FastAPI()
validate_order = contract_dependency("contracts/orders.yaml")  # built once


@app.post("/orders")
async def create_order(order: dict = Depends(validate_order)) -> dict:
    # `order` is the coerced + validated payload.
    return {"status": "accepted", "order_id": order["order_id"]}
```

Contract violations return a structured 422; the same YAML also drives your
pipelines and workers. Recipe:
[Gate a FastAPI endpoint](docs/cookbook/gate-fastapi-endpoint.md).

## Validate a Kafka topic, per record

```python
import asyncio
from pycharter import kafka


async def main():
    async for record in kafka.consume(
        topic="orders",
        bootstrap_servers="localhost:9092",
        group_id="my-service",
        contract_dir="contracts/orders",  # or contract_store=...
    ):
        process(record.payload)  # only records that passed the contract


asyncio.run(main())
```

Invalid records route to a DLQ with a structured envelope; `detect` /
`shadow` / `enforce` modes support safe rollouts, and `pycharter stream run`
runs it as a worker with Prometheus metrics. Start at
[Streaming — start here](docs/guides/streaming-overview.md).

## Gate contract evolution in CI

```bash
pycharter contract diff old.yaml new.yaml --fail-on-breaking
```

Classifies every field/type/required change as breaking or safe
(backward-compat semantics) — drop into pre-commit or CI so a contract
change can't ship without review.

---

## Concepts

| Concept | What it is | When you use it |
|--------|------------|------------------|
| **Schema** | The shape of the data (JSON Schema): types, required fields, nested objects. | When you only need structure. |
| **Data contract** | Schema + **coercion rules** (`"30"` → `30`) + **validation rules** (min/max, allowed values) + optional **metadata** (ownership, governance). | One artifact for structure, transforms, and business rules. |
| **Contract store** | A database (SQLite, PostgreSQL, …) holding versioned contracts so many apps reuse them. | Multiple services needing one source of truth. |
| **Enforcement point** | Where the contract executes: API dependency, stream consumer, ETL step flags (`coerce` / `validate` / `quality`). | Wherever bad data must be stopped. |

```
  Option A: no database   schema/contract in code or YAML → Validator → validate(data)
  Option B: with store    contract in DB → Validator(store=...)       → validate(data)
  Option C: pipelines     extract → [contract] → transform → [contract] → load
```

Start with Option A; add the store when contracts need versioned sharing; add
pipeline/stream enforcement where data moves. Deep dives:
[Concepts](docs/getting-started/concepts.md) ·
[Contract → ETL → quality journey](docs/getting-started/contract-etl-quality-journey.md) ·
[Cookbook](docs/cookbook/index.md).

---

## Artifacts

PyCharter has **six artifacts**. Every one is a YAML file in the same envelope,
so all six are authored, versioned, diffed, and reviewed the same way:

```yaml
api_version: pycharter.io/v2   # the format marker — always this
kind: DataContract             # which artifact this is (one of the six below)
metadata:
  name: orders                 # identity slug
  namespace: example           # identity scope (optional; absent = global)
  version: "1.0.0"             # content version
spec:                          # the body — shape depends on `kind`
  ...
```

Identity is `(metadata.namespace, metadata.name, metadata.version)`, and
`name` is also the display label — there is no separate title field.

| `kind:` | Defines | Run it with |
|---|---|---|
| **`DataContract`** | The rules a record must satisfy: shape, coercions, validations, and what its fields *mean*. | `Validator`, or a `contract:` block on any pipeline / stream step |
| **`Pipeline`** | A batch extract → transform → load flow. | `pycharter pipeline run` |
| **`StreamingWorker`** | A long-running consumer that validates a stream record by record. | `pycharter stream run` |
| **`Bridge`** | A WebSocket firehose republished to Kafka, so it can be consumed by a scaled worker group. No validation, no DLQ. | `pycharter bridge run` |
| **`ConceptScheme`** | A governed vocabulary: the concepts contract fields bind to. | `pycharter db seed`, semantic API, UI |
| **`ConceptSchemeTemplate`** | A reusable starter vocabulary you copy and adapt into a `ConceptScheme`. | `GET /api/v1/semantic/templates`, UI |

Scaffold one with `pycharter contract init`, `pycharter pipeline init`,
`pycharter stream init`, or `pycharter bridge init`; runnable end-to-end
examples live in [`data/seed/examples/`](src/pycharter/data/seed/examples/).
Files written by an older PyCharter still load, and `pycharter migrate` rewrites
them in place — see
[Artifact format versioning](docs/reference/artifact-format-versioning.md).

### 1. `DataContract`

The artifact every enforcement point loads. One file carries the shape
(`json_schema`), the transforms applied **before** validation
(`coercion_rules`), and the business rules applied **after** the types check out
(`validation_rules`):

```yaml
# contracts/orders.yaml
api_version: pycharter.io/v2
kind: DataContract
metadata:
  name: order
  version: "1.0.0"
  status: active
  type: domain_entity
  description: A customer order accepted by the storefront API.
  ownership:
    roles:
      business_owner: [storefront-team]
      technical_owner: [data-platform]

spec:
  json_schema:
    type: object
    title: order
    version: "1.0.0"
    additionalProperties: false
    required: [order_id, customer_id, total_amount, currency_code, status]
    properties:
      order_id:      {type: string, format: uuid}
      customer_id:   {type: string}
      total_amount:  {type: number, minimum: 0}
      currency_code: {type: string, minLength: 3, maxLength: 3}
      status:        {type: string}
      placed_at:     {type: string, format: date-time}

  # Before validation: make messy input typed. "19.99" -> 19.99, "usd" -> "USD".
  coercion_rules:
    version: "1.0.0"
    rules:
      total_amount: coerce_to_float
      currency_code: coerce_to_uppercase
      placed_at: coerce_to_datetime

  # After validation: business rules beyond types.
  validation_rules:
    version: "1.0.0"
    rules:
      total_amount:
        greater_than_or_equal_to: {threshold: 0}
      currency_code:
        matches_regex: {pattern: "^[A-Z]{3}$"}
      status:
        only_allow:
          allowed_values: [pending, paid, shipped, cancelled]
```

That is the `contracts/orders.yaml` the FastAPI gate above loads — and the same
file backs `pycharter validate contracts/orders.yaml orders.jsonl`, the Kafka
consumer, and the pipeline below. Rules are **named, not code**: 21 built-in
coercions and 16 validations, listed in
[built-in coercions and validations](docs/reference/builtin-coercions-and-validations.md)
(add `severity: warning` inside a rule to report instead of reject). Start one
from a real sample with
`pycharter contract sketch --from-json sample.json -o contracts/orders.yaml`.

A contract's four body sections — `json_schema`, `coercion_rules`,
`validation_rules`, and `field_bindings` (below) — can also be **stored
separately and reused by reference**, so two contracts share one set of
validation rules instead of copying them. Authoring them inline, as above, is
the default and always works.

### 2. `Pipeline`

Batch extract → transform → load. The `contract:` block attached to a step *is*
the enforcement point — put it on whichever steps must be gated:

```yaml
# pipelines/orders_ingest/pipeline.yaml
api_version: pycharter.io/v2
kind: Pipeline
metadata:
  name: orders_ingest
  namespace: examples
  version: "1.0.0"
  description: Load the daily orders CSV, normalize it, export validated JSONL.

spec:
  variables:
    INPUT_PATH: data/input/orders.csv
    OUTPUT_PATH: data/exports/orders.jsonl

  steps:
    - id: extract_orders
      type: extract
      source:
        type: file
        path: ${INPUT_PATH}
        format: csv

    - id: normalize_orders
      type: transform
      input: extract_orders
      operations:
        - rename: {order_total: total_amount}
        - defaults: {status: pending}

    - id: load_orders
      type: load
      input: normalize_orders
      target:
        type: file
        path: ${OUTPUT_PATH}
        format: jsonl
        mode: overwrite
      contract:                      # <- the enforcement point
        ref: {type: store, name: order, version: "1.0.0"}
        coerce: true                 # apply coercion_rules
        validate: true               # reject records that fail the contract
        quality: true                # run the contract's quality checks
```

```bash
pycharter pipeline run ./pipelines/orders_ingest --watch
```

Use `ref: {type: file, path: ../../contracts/orders.yaml}` to point at a
contract on disk instead of the store. Attach the block to any step — gate on
extract, after transform, before load, or all three. Sources include `file`,
`http`, and SQL; `${VAR}` placeholders resolve from the `spec.variables:` block
first, then the environment, with `${VAR:-default}` and `${VAR:?required}`
semantics. More: [Building pipelines](docs/tutorials/pipelines.md).

### 3. `StreamingWorker`

The same contract, enforced record by record on a live stream instead of a
batch. `mode` is what makes a rollout safe: `detect` passes every record through
and only counts violations, `shadow` forwards invalid records *and* copies them
to the DLQ, and `enforce` (the default) sends them to the DLQ only:

```yaml
# streams/orders/stream-worker.yaml
api_version: pycharter.io/v2
kind: StreamingWorker
metadata:
  name: orders
  namespace: example
  version: "1.0.0"
  description: Validate the orders.raw topic against the order contract.

spec:
  variables:
    KAFKA_BROKERS: localhost:9092

  streams:
    - name: orders-validator
      source:
        type: kafka                  # kafka | websocket | sse | rabbitmq | sqs
        topic: orders.raw
        bootstrap_servers: ${KAFKA_BROKERS}
        consumer_group: pycharter-orders
        auto_offset_reset: latest
      contract:                      # same block shape as a pipeline step
        ref: {type: store, name: order, version: "1.0.0"}
        coerce: true
        validate: true
      mode: enforce                  # detect | shadow | enforce
      delivery: at_least_once
      load:
        target:
          type: kafka
          topic: orders.validated
          bootstrap_servers: ${KAFKA_BROKERS}
      dlq:
        target:
          type: kafka
          topic: orders.dlq
          bootstrap_servers: ${KAFKA_BROKERS}

  observability:
    health_port: 8080                # /healthz, /readyz, /metrics
    log_format: json
```

```bash
pycharter stream run ./streams/orders/stream-worker.yaml
```

More: [Streaming — start here](docs/guides/streaming-overview.md).

### 4. `Bridge`

A WebSocket is a single connection, so running more workers against one just
duplicates consumption. A bridge republishes the firehose onto a partitioned
Kafka topic, which a scaled `StreamingWorker` group can then consume. It only
moves bytes — no contract, no validation, no DLQ:

```yaml
# bridges/firehose/bridge.yaml
api_version: pycharter.io/v2
kind: Bridge
metadata:
  name: firehose-bridge
  namespace: example
  version: "1.0.0"
  description: Republish a WebSocket firehose to a Kafka topic.

spec:
  variables:
    KAFKA_BROKERS: localhost:9092

  bridges:
    - name: firehose
      source:
        type: websocket
        url: ${WS_URL:?WS_URL is required}
        data_format: json
        ping_interval: 20.0          # keepalive seconds
        idle_timeout: 60.0           # reconnect after this much silence
      sink:
        type: kafka
        topic: market.raw
        bootstrap_servers: ${KAFKA_BROKERS}
```

```bash
pycharter bridge run ./bridges/firehose/bridge.yaml
```

### 5. `ConceptScheme`

Contracts describe the *shape* of a field. A concept scheme describes what
fields **mean**, once, for every contract that uses them:

```yaml
# vocabulary/ecommerce.yaml
api_version: pycharter.io/v2
kind: ConceptScheme
metadata:
  name: ecommerce
  version: "2.0.0"
  description: Governed vocabulary for the storefront domain.

spec:
  concepts:
    - id: Customer
      label: Customer
      concept_type: entity          # a thing with identity
      definition: A person or organization that places orders.
    - id: Order
      label: Order
      concept_type: event
      definition: A customer's request to purchase one or more products.
    - id: MonetaryAmount
      label: Monetary Amount
      concept_type: attribute       # a property or measurement
      definition: A value of money in a stated currency.
    - id: OrderTotal
      label: Order Total
      concept_type: attribute
      broader: MonetaryAmount       # narrower than MonetaryAmount
      definition: Amount payable for an order, including tax and shipping.
```

Contracts then **bind** their fields to those concepts, in the same contract
file as the schema:

```yaml
spec:
  field_bindings:
    order_id:
      primary:
        concept: {concept: Order}
        concept_scheme: "ecommerce:2.0.0"
        role: identity              # this field identifies the record's entity
    customer_id:
      primary:
        concept: {concept: Customer}
        concept_scheme: "ecommerce:2.0.0"
        role: reference             # points at a different entity
    total_amount:
      primary:
        concept: {concept: OrderTotal}
        concept_scheme: "ecommerce:2.0.0"
        role: attribute             # a property of this record (the default)
      annotations:                  # optional extra meaning, zero or more
        - concept: MonetaryAmount
          concept_scheme: "ecommerce:2.0.0"
          relationship: derived_from
```

A binding may also carry `pii: true` (personal data) or `restricted: true`
(raw values must never be captured) — field-level flags that govern what
downstream profiling, extraction, and AI surfaces are allowed to see.

Bindings are what make two contracts that both call a column `total` comparable,
and they are what the MCP server serves to agents: the context an agent receives
about a field is bound to the same contract your validators enforce. A field may
stay unbound — governance tooling simply flags it for review.

### 6. `ConceptSchemeTemplate`

A starter vocabulary — the same idea as a `ConceptScheme`, but shipped as a
reusable pattern you copy and adapt rather than a governed scheme you enforce
against. PyCharter bundles six (`ecommerce`, `healthcare`, `saas`, `party`,
`product_catalog`, `event_participation`); this is the shape if you write your
own:

```yaml
# templates/subscriptions.yaml
api_version: pycharter.io/v2
kind: ConceptSchemeTemplate
metadata:
  name: subscriptions
  version: "1.0.0"
  description: Tenants, plans, and recurring charges.
  category: domain               # domain (a full domain) | pattern (a reusable cluster)
  tags: [billing, saas]

spec:
  concepts:
    - name: Tenant
      concept_type: entity
      definition: An organization that holds a subscription.
      alt_labels: [Account, Workspace]
    - name: Plan
      concept_type: reference_data
      definition: A named tier a tenant can subscribe to.
    - name: Subscription
      concept_type: event
      definition: A tenant's commitment to a plan over a billing period.

  relationships:
    - {source: Tenant, target: Subscription, type: has, label: holds}
    - {source: Subscription, target: Plan, type: references, label: on plan}
```

Browse and fetch the bundled ones with `GET /api/v1/semantic/templates`, or read
them under
[`data/templates/concept_schemes/`](src/pycharter/data/templates/concept_schemes/).

More on the vocabulary layer: [Vocabulary](docs/guides/concepts.md) ·
[Binding any resource to a concept](docs/guides/resource-bindings.md) · a
complete worked example (22 contracts, a scheme, and a pipeline) ships in
[`data/seed/examples/ecommerce/`](src/pycharter/data/seed/examples/ecommerce/).

---

## Installation

```bash
pip install pycharter
```

That's all you need for in-process validation. Optional extras add
capabilities — install only what you use:

| Extra | Adds | Install |
|-------|------|---------|
| `[fastapi]` | `contract_dependency` request gate | `pip install pycharter[server]` |
| `[streaming]` / `[kafka]` | Streaming validation engine / Kafka source | `pip install pycharter[streaming]` |
| `[api]` | REST API server (FastAPI + Uvicorn) | `pip install pycharter[server]` |
| `[ui]` | Pre-built Web UI (no Node.js required) | `pip install pycharter[server]` |
| `[postgres]` | PostgreSQL drivers for the ETL loader, DLQ, and contract store (asyncpg + psycopg) | `pip install pycharter[postgres]` |
| `[mcp]` | MCP server for AI agents | `pip install pycharter[ai]` |
| `[otel]` | OpenTelemetry tracing instruments | `pip install pycharter[otel]` |
| `[observability]` | Prometheus exporter + webhook sinks | `pip install pycharter[server]` |
| `[airflow]` / `[dagster]` | Orchestrator operators | `pip install pycharter[airflow]` |
| `[all]` | The common stack: api, ui, worker, pipeline, postgres, streaming, messaging, extraction, lineage, semantic-export, collab, cli, docs (not the optional `mcp` / `otel` / `observability` / `airflow` / `dagster` integrations — install those explicitly) | `pip install pycharter[all]` |

### Database setup (only for the contract store, API, or UI)

```bash
pycharter db init     # creates schema (default: sqlite:///pycharter.db)
pycharter db seed     # loads reference data
```

Skip this entirely if you only validate dicts or YAML contracts directly.
PostgreSQL/SQLite have managed Alembic migrations; see
[Store backend tiers](docs/reference/store-backend-tiers.md) and the
[Configuration Guide](docs/guides/configuration.md).

---

## ODCS and the contract ecosystem

PyCharter round-trips the
[Open Data Contract Standard](https://bitol-io.github.io/open-data-contract-standard/):
`to_odcs()` emits v3.1.0 documents that validate against the published
schema; `from_odcs()` imports v3.0.x / v3.1.x documents authored anywhere —
so contracts written with datacontract-cli enforce at runtime here, and
contracts authored here flow back into the ODCS toolchain. PyCharter is not
a rival standard; it is an **execution engine** for the standard.

```python
from pycharter.contract_io import from_odcs, to_odcs
from pycharter import Validator

contract = from_odcs(odcs_document)  # any v3.0.x / v3.1.x doc
validator = Validator(contract)  # enforce it immediately
```

How PyCharter composes with datacontract-cli, Soda, GX, and dbt:
[ODCS integration and interop](docs/guides/odcs-and-interop.md).

## A toolkit for AI agents

PyCharter is also the **toolkit an AI agent reaches for when it must be
correct about your data**: the same versioned contract becomes the schema an
agent is handed, the validator that checks its output, and the vocabulary that
tells it what a field means — over MCP with zero pycharter install on the
agent side. PyCharter never runs the agent; your framework owns the model and
the loop.

```python
from pycharter import Validator
from pycharter.agent import repair_loop

validator = Validator.from_file("contracts/order.yaml")
schema = validator.to_llm_schema()               # pruned, governance-stripped

def emit(feedback):                              # your agent
    return call_your_model(schema=schema, feedback=feedback)

result = repair_loop(emit, validator, on_exhaust="quarantine")
# result.data is contract-clean, or result.dead_letter is an auditable failure
```

- `Validator.to_llm_schema()` / `to_tool_schema("anthropic"|"openai")` — JIT
  schemas and provider tool definitions.
- `ValidationResult.to_agent_feedback()` — a compact correction message.
- `pycharter.contrib.agents` — Claude Agent SDK and Pydantic AI adapters.

PyCharter also ships an MCP server that serves **governed context packs and
contract tools** — `fetch_contract_schema`, `validate_record` (with
`agent_feedback`), `sketch_contract`, `diff_contract_candidate`,
`lint_contract`, `fetch_concept_pack` — to Claude Code, Claude Desktop, or any
MCP client, with role/workspace policy filtering:

```bash
pip install pycharter[ai]
pycharter agent serve            # stdio MCP; or `pycharter agent tools` / `try`
```

Full guide: [PyCharter for AI agents](docs/guides/pycharter-for-ai-agents.md) ·
[Agent integration (MCP)](docs/guides/agent-integration-mcp.md).

---

## Architecture at a glance

| Service | Input | Output | Journey stage |
|---------|-------|--------|---------------|
| **Contract Parser** | Contract files (YAML/JSON) | `ContractMetadata` | Specification → Parsing |
| **Contract Builder** | Separate artifacts or store | Consolidated contract | Storage → Consolidation |
| **Contract Store** | `ContractMetadata` | Stored metadata (DB) | Parsing → Storage |
| **Pydantic Generator** | JSON Schema | Pydantic models | Storage → Model generation |
| **JSON Schema Converter** | Pydantic models | JSON Schema | (Bidirectional) |
| **Runtime Validator** | Pydantic models + data | `ValidationResult` | Model generation → Validation |
| **ETL Pipelines** | Config files or code | `PipelineResult` | Extract → Transform → Load |
| **Quality Assurance** | Contract + data | `QualityReport` | Validation → Quality monitoring |

Each service is independent yet composable. The full tour — including the
REST API and Web UI wrappers — lives in the
[Core services reference](docs/reference/core-services.md); the layered
design philosophy is in [ARCHITECTURE.md](ARCHITECTURE.md).

## Picking the right API

The canonical path is the **`Validator` class** — create one from a contract
file, dict, directory, or store, and call `validate(record)` /
`validate_batch(records)`. It compiles the Pydantic model once, is
[thread-safe to share](docs/operations/benchmarks.md), and is what
production code should use.

| Use case | Approach | Example |
|----------|----------|---------|
| Production service, many validations | `Validator` class | `Validator.from_file("c.yaml").validate(data)` |
| Pandas DataFrame, split good/bad rows | `validate_dataframe()` | `validate_dataframe(df, "c.yaml", on_error="quarantine")` |
| Shell / CI gate on a data file | `pycharter validate` CLI | `pycharter validate c.yaml data.jsonl` |
| Batch processing | `Validator.validate_batch()` | `validator.validate_batch(rows)` |
| You already have a Pydantic model | Low-level function (deprecated at top level) | `from pycharter.runtime_validator import validate` |

ETL flows use **`Pipeline`**; quality checks use **`QualityCheck`**; contract
storage is pluggable via **`ContractStoreClient`** implementations. Built-in
coercions/validations and custom registration:
[reference](docs/reference/builtin-coercions-and-validations.md).

---

## Documentation

- **Docs site:** [https://optophi.github.io/pycharter/](https://optophi.github.io/pycharter/) — or `pycharter docs serve` locally (`pip install pycharter[docs]`)
- [Start here](docs/getting-started/start-here.md) · [Quick Start tour](docs/getting-started/quickstart.md) · [Cookbook](docs/cookbook/index.md)
- [End-to-end user guide](docs/guides/user-guide-end-to-end.md) · [Admin / governance guide](docs/guides/admin-guide-governance.md)
- [Streaming — start here](docs/guides/streaming-overview.md) · [Configuration](docs/guides/configuration.md) · [Validator benchmarks](docs/operations/benchmarks.md)

## Development and testing

```bash
python -m venv venv && source venv/bin/activate
pip install -e ".[dev]" && pre-commit install
pytest tests/unit            # fast suite; `pytest -m integration` needs Docker
./scripts/ci.sh              # full GHA-parity gate before opening a PR
```

See [CONTRIBUTING.md](CONTRIBUTING.md) for the workflow and
[AGENTS.md](AGENTS.md) for AI-assisted contributions.

## Contributing

Contributions are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md). Report
security issues per [SECURITY.md](SECURITY.md); community expectations are in
[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).

## License

MIT — see [LICENSE](LICENSE).

## Links

- **Repository:** [github.com/optophi/pycharter](https://github.com/optophi/pycharter)
- **Issues:** [GitHub Issues](https://github.com/optophi/pycharter/issues)
- **PyPI:** [pypi.org/project/pycharter](https://pypi.org/project/pycharter/)
