Metadata-Version: 2.4
Name: axiomatic-engine
Version: 0.1.0
Summary: A typed orchestration engine for configurable ingestion and transformation pipelines.
Keywords: data-engineering,elt,ingestion,pipelines,dlt,dbt
Author: amisont
Author-email: amisont <134074972+amisont@users.noreply.github.com>
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: dbt-duckdb>=1.10.1
Requires-Dist: dlt[motherduck]>=1.25.0
Requires-Dist: duckdb>=1.5.2
Requires-Dist: fastmcp>=3.2.4
Requires-Dist: psutil>=7.2.2
Requires-Dist: requests>=2.33.1
Requires-Python: >=3.12
Description-Content-Type: text/markdown

# Axiomatic Engine

Axiomatic Engine orchestrates ingestion and transformation pipelines while keeping domain logic in project folders.

## Quick Start

Install the package:

- `uv add axiomatic-engine`

Or with pip:

- `pip install axiomatic-engine`

Run a project entrypoint:

- copy `projects/fake_store/.env.example` to `projects/fake_store/.env`
- set required environment variables for your target warehouse and storage
- run `uv run python projects/fake_store/run_pipeline.py`

This gives a working reference for a REST ingestion source and transformation flow using the engine contracts.

## Contributor Quickstart

The project standardises on `uv` for local quality checks and packaging workflows.

- Install hooks once:
  - `uv run --group dev pre-commit install`
- Run hooks across the repository:
  - `uv run --group dev pre-commit run --all-files`
- Run the full package quality gate:
  - `uv run --group dev --group test python scripts/quality_gate.py`

## Runtime Configuration Contract

The engine reads typed runtime settings from `AXIOMATIC_*` environment variables, with project entrypoints able to override values via CLI flags.

Schema layers are configured independently:

- `AXIOMATIC_SCHEMA_BRONZE`
- `AXIOMATIC_SCHEMA_SILVER`
- `AXIOMATIC_SCHEMA_GOLD`
- `AXIOMATIC_SCHEMA_ANALYTICS`

This keeps medallion naming explicit and consistent across ingestion and dbt model targets.

## Source Routing Contract

Source implementations now expose specific source kinds:

- `rest_api`
- `http_file`

The standard project path is typed source definitions routed through `axiomatic_engine.sources.factory` via `build_source(...)`.

Direct source constructors remain supported for advanced project-specific customisation.

## Ingestion Resource Load Hints

Sources can provide optional per-resource hints through `ResourceLoadHints`:

- `write_disposition`: `append`, `replace`, or `merge`
- `primary_key`: key used by merge semantics
- `schema_evolution_mode`: `auto`, `strict`, or `discard`

These hints are source-agnostic contracts and are mapped by the source bridge when building `dlt.resource(...)`.

## Schema Evolution Policy

The engine supports a hybrid schema evolution policy:

- `auto`: evolve bronze schema when new fields appear
- `strict`: freeze schema and fail on drift
- `discard`: discard unexpected values while keeping rows

Projects can set this per resource so strictness can vary by endpoint.

## Rerun Semantics

Replay behaviour depends on per-resource load hints:

- `merge` with a primary key provides idempotent upsert behaviour
- `replace` gives deterministic snapshot tables on rerun
- `append` preserves full arrival history

Choose by resource based on analytical needs and source stability.

Operationally, ingestion stage gating also considers:

- `force_reload=True` always triggers ingestion
- otherwise storage-cache heuristics decide whether ingestion should run

## Security Defaults

Current security defaults prioritise secret-safe runtime surfaces:

- dbt command environments are filtered to an allowlist instead of forwarding the full process environment
- token-like values in dbt stderr are redacted before being returned in transformation failure details
- ingestion logs avoid dumping raw loader result objects

## Package Quality Gate

The package quality gate is standardised on `uv` and runs the same checks you should later enforce in CI:

- pre-commit checks across the repository
- unit tests
- distribution build (wheel + source distribution)
- distribution metadata checks
- wheel content validation for `src`-layout packaging boundaries

Run it with both dependency groups enabled:

`uv run --group dev --group test python scripts/quality_gate.py`

## Pre-commit Checks

Use pre-commit for fast local feedback before commits.

Install the hooks:

`uv run --group dev pre-commit install`

Run hooks on all files:

`uv run --group dev pre-commit run --all-files`

## Release Publishing

Release publishing is handled by `.github/workflows/release.yml`.

- Trigger: push a tag matching `v*` (for example `v0.1.1`)
- Required PyPI setup: configure a Trusted Publisher for this GitHub repository
- Workflow behaviour:
  - runs the same quality gate command used locally and in CI
  - builds release distributions
  - uploads `dist/*` as workflow artefacts
  - publishes with `pypa/gh-action-pypi-publish` via OIDC
