Metadata-Version: 2.4
Name: etherealogic-aetheriaforge
Version: 0.1.4
Summary: EthereaLogic Databricks Suite — Intelligent Data Transformation Engine
Project-URL: Homepage, https://github.com/Org-EthereaLogic/AetheriaForge
Project-URL: Repository, https://github.com/Org-EthereaLogic/AetheriaForge
Project-URL: PyPI, https://pypi.org/project/etherealogic-aetheriaforge/
Project-URL: Issues, https://github.com/Org-EthereaLogic/AetheriaForge/issues
Project-URL: Documentation, https://github.com/Org-EthereaLogic/AetheriaForge#readme
Author: Anthony Johnson
License: MIT
License-File: LICENSE
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Requires-Dist: pandas>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: scipy>=1.11
Provides-Extra: ingest
Requires-Dist: fastavro>=1.9; extra == 'ingest'
Requires-Dist: lxml>=5.0; extra == 'ingest'
Requires-Dist: openpyxl>=3.1; extra == 'ingest'
Requires-Dist: pyarrow>=14.0; extra == 'ingest'
Description-Content-Type: text/markdown

<p align="center">
  <img src="assets/aetheriaforge-brand-system/icons/aetheriaforge-logo-1200x320.png" alt="AetheriaForge" width="700"/>
</p>

# Intelligent Data Transformation. Coherence-Scored. Evidence-Backed.

**EthereaLogic Databricks Suite — AetheriaForge**

Built by Anthony Johnson | EthereaLogic LLC

---

<p align="left">
  <a href="https://github.com/Org-EthereaLogic/AetheriaForge/actions/workflows/ci.yml"><img src="https://github.com/Org-EthereaLogic/AetheriaForge/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
  <a href="https://pypi.org/project/etherealogic-aetheriaforge/"><img src="https://img.shields.io/pypi/v/etherealogic-aetheriaforge" alt="PyPI version"></a>
  <a href="https://app.codacy.com/gh/Org-EthereaLogic/AetheriaForge/dashboard"><img src="https://img.shields.io/badge/codacy-dashboard-blue" alt="Codacy dashboard"></a>
  <a href="https://codecov.io/gh/Org-EthereaLogic/AetheriaForge"><img src="https://codecov.io/gh/Org-EthereaLogic/AetheriaForge/graph/badge.svg" alt="Codecov coverage"></a>
</p>

---

> If this tool is useful to your team, consider [starring the repo](https://github.com/Org-EthereaLogic/AetheriaForge) — it helps others in the Databricks community find it.

---

Every Medallion transformation introduces information loss. Most pipelines ignore it. AetheriaForge measures it by transforming source records through schema contracts, scoring the result for coherence, applying optional exact-match entity resolution and latest-wins temporal reconciliation, and recording append-only evidence. Nothing is assumed to have passed unless the artifact says so.

## Executive Summary

| Leadership question | Answer |
| ------------------- | ------ |
| What business risk does this address? | Enterprises transforming data through Bronze to Silver to Gold layers have no mathematical model governing how much information loss is acceptable at each stage, no governed entity resolution across source systems, and no auditable evidence trail for transformation decisions. |
| What does this application prove? | A Databricks-deployable transformation engine that scores every operation for coherence, resolves entities across multiple sources, reconciles temporal conflicts, and surfaces queryable evidence artifacts in a read-only operator dashboard. |
| Why does it matter? | Moving data between layers is not the hard problem. Proving that the transformation preserved what it should, resolved what it needed to, and caught what it missed — with evidence — is the problem this solves. |

## Key Exhibits

### Exhibit 1: Forge Contract Registry

The operator dashboard loads all registered forge contracts and shows every dataset with its version, source, target, and coherence engine. Eight datasets registered — all bound to Bronze-to-Silver transformation contracts with Shannon entropy coherence scoring.

<p align="center">
  <img src="assets/aetheriaforge-brand-system/marketplace/screenshots/01-registry.png" alt="Forge Registry tab showing 8 datasets registered with contract versions, source and target locations, and coherence engine" width="900"/>
</p>

### Exhibit 2: Transformation Status with Coherence Scores

The Transformation Status tab surfaces evidence artifacts across all forge operations. Every row is one artifact: filename, dataset, verdict, coherence score, record counts, and timestamp. 1,008 artifacts shown with PASS, WARN, and FAIL verdicts — coherence scores make information loss immediately visible without opening a single file.

<p align="center">
  <img src="assets/aetheriaforge-brand-system/marketplace/screenshots/02-transformation-status.png" alt="Transformation Status tab showing 1008 evidence artifacts with PASS, WARN, and FAIL verdicts, coherence scores, and record counts" width="900"/>
</p>

### Exhibit 3: Full Evidence Artifact Inspection

The Evidence Explorer loads a single evidence artifact by filename and renders the complete JSON payload inline. The artifact shown is a forge operation with a FAIL verdict: coherence score 0.0, with the full transformation context including dataset identity, record counts, and gate results.

<p align="center">
  <img src="assets/aetheriaforge-brand-system/marketplace/screenshots/03-evidence-explorer.png" alt="Evidence Explorer showing full JSON payload for a forge FAIL artifact including coherence score and transformation context" width="900"/>
</p>

### Exhibit 4: Visual Analytics Across All Forge Operations

The Analytics tab scans the evidence directory and renders four charts: verdict distribution, coherence score distribution, daily activity volume, and coherence trend over time. Operators see transformation health at a glance across all registered datasets.

<p align="center">
  <img src="assets/aetheriaforge-brand-system/marketplace/screenshots/04-analytics.png" alt="Analytics tab showing verdict distribution, coherence score distribution, daily activity volume, and coherence trend charts" width="900"/>
</p>

## The Business Problem

Enterprises operating mature Lakehouse architectures face three transformation gaps that existing tools do not address:

- **Coherence loss is unmeasured.** Every transformation from Bronze to Silver to Gold discards, reshapes, or aggregates data. Without a coherence score, there is no way to know whether the output meets the target layer's quality threshold before writing it.

- **Entity resolution is ad-hoc.** Multiple source systems use different identifiers for the same entities. Lookup tables and manual mappings do not scale, version, or produce evidence of match decisions.

- **Temporal reconciliation is invisible.** CDC streams, SCD Type 2 dimensions, and batch loads create overlapping records with conflicting timestamps. Merge decisions happen silently with no audit trail.

## What This Repository Contains

| Surface | Purpose |
| ------- | ------- |
| `src/aetheriaforge/ingest/` | Enterprise file ingestion — CSV, Parquet, JSON, Excel, XML, ORC, Avro, and more |
| `src/aetheriaforge/forge/` | Coherence-scored transformation engine (Shannon entropy v1.x) |
| `src/aetheriaforge/resolution/` | Cross-source entity resolution using policy-driven exact matching in v1.x |
| `src/aetheriaforge/temporal/` | Temporal reconciliation using deterministic `latest_wins` conflict handling in v1.x |
| `src/aetheriaforge/schema/` | Schema-contract transformation metadata and schema enforcement with recorded contract versions |
| `src/aetheriaforge/evidence/` | Append-only transformation artifact writing shared across all modules |
| `src/aetheriaforge/orchestration/` | Workflow sequencing — runs all forge operations in order |
| `src/aetheriaforge/config/` | Forge contract and policy configuration |
| `src/aetheriaforge/integration/` | Optional DriftSentinel event emission and drift follow-up ingestion |
| `app/` | Databricks App (Gradio) — four-tab read-only operator dashboard |
| `notebooks/` | Onboarding, execution, and evidence-review notebooks for Databricks |
| `resources/` | Databricks Asset Bundle pipeline, job, and app resource definitions |
| `templates/` | Forge contract, resolution policy, and schema contract templates |
| `specs/` | Canonical SDLC documents governing the product |
| `tests/` | Pytest suite covering domain logic, packaging, and governance |

Every directory above contains a `README.md` describing its contents, including each submodule under `src/aetheriaforge/`.

## What This Repository Proves

| Verified outcome | Evidence from this repository |
| ---------------- | ----------------------------- |
| Coherence scoring measures information loss at every transformation | Forge engine computes Shannon entropy before and after, producing a 0.0–1.0 coherence score per operation with configurable thresholds |
| Entity resolution produces governed match evidence | Resolution module matches records across source systems using configured exact key rules, policy-driven ambiguity handling, and append-only evidence |
| Temporal reconciliation resolves conflicts with an audit trail | Temporal reconciler applies deterministic `latest_wins` selection, records duplicate-timestamp conflicts, and writes evidence for merge decisions |
| Schema enforcement is contract-driven and versioned | Schema enforcer validates output against YAML contracts, records the applied contract version in evidence, and honors contract enforcement settings |
| Enterprise file ingestion handles heterogeneous sources | Ingest module supports 10 formats with auto-detection, producing evidence metadata for every read operation |
| Evidence artifacts are queryable without writing scripts | Operator dashboard surfaces all artifacts with coherence scores, verdicts, timestamps, and provenance metadata across four tabs |
| Databricks deployment workflow is defined for a configured workspace | Asset Bundle resources and deployment docs cover validate, deploy, and app status checks; replay requires Databricks credentials and Unity Catalog |
| DriftSentinel integration is standalone-safe | Event emission and drift ingestion are optional — NullEventChannel is the default, and no runtime dependency on DriftSentinel exists |

## Decision / KPI Contract

**Business decision:** is the forged output trustworthy enough for the target Medallion layer?

| KPI | Meaning |
| --- | ------- |
| `coherence_score` | Information preservation ratio for the transformation (0.0–1.0), scored against the declared schema lineage for contract-backed runs |
| `resolution_confidence` | Match confidence for entity resolution decisions |
| `temporal_conflicts` | Count of temporal merge conflicts detected |
| `schema_conformance` | Percentage of records conforming to the target schema contract |
| `transformation_verdict` | PASS / WARN / FAIL for each forge operation |

**Control rule:** no transformation is assumed to have passed unless a PASS artifact exists in the evidence directory. FAIL artifacts carry measured values and thresholds so the platform team can triage without opening raw files.

## Why This Pattern

- **Gap 1.** Transformation quality must be measured, not assumed. Shannon entropy coherence scoring gives every operation a mathematical signal — not a boolean check, but a ratio that tells you how much declared source information the transformation preserved relative to the target contract.
- **Gap 2.** Entity resolution and temporal reconciliation must produce evidence. Match decisions and merge outcomes are written to the same append-only evidence directory as forge artifacts. A single dashboard surfaces all of them without special-casing any module.
- **Gap 3.** The operator dashboard must be read-only. The Gradio app exposes no write surfaces. Evidence is queried, never edited through the UI. This keeps the audit trail clean and the deployment governance simple.

## How It Works

1. **Register datasets and forge contracts.** Each dataset is registered with a YAML contract specifying source location, target schema, optional resolution rules, optional temporal merge policy, and coherence thresholds. File-backed datasets use a landing path; table-backed datasets use the `catalog/schema/table` triplet.
2. **Run the forge pipeline.** The orchestration layer loads the source surface, performs schema-contract transformation when no forged DataFrame is supplied, applies optional schema enforcement, optional exact-match entity resolution, and optional `latest_wins` temporal reconciliation, then scores the result for coherence. For schema-backed runs, coherence is measured against the contract's declared source lineage so column renames, multi-source derivations, and intentional projection are scored against the intended target shape rather than raw by-name overlap. Each module writes append-only evidence artifacts to the shared evidence directory.
3. **Inspect transformation evidence.** The Forge Registry tab shows all registered datasets with contract versions and locations. The Transformation Status tab surfaces artifacts with coherence scores, verdicts, and provenance. The Evidence Explorer loads any single artifact by filename and renders the full JSON payload. The Analytics tab renders verdict distribution, coherence trends, daily volume, and health over time.
4. **Integrate with DriftSentinel (optional).** When bundled, AetheriaForge emits transformation events that DriftSentinel can consume for smarter publication gating, and receives drift payloads that produce evidence-backed follow-up actions. When standalone, the NullEventChannel silently drops events with zero overhead.

## Databricks Fit

- **Databricks Asset Bundles** for source-controlled deployment of pipeline, job, and app resource definitions — validated and deployed from the repo with a single make target.
- **Databricks Apps (Gradio)** for a governed, read-only operator dashboard with no custom web infrastructure required.
- **Unity Catalog** for governed table publication and the evidence volume backing the operator dashboard.
- **Databricks Lakeflow / Jobs** for scheduled forge pipeline execution across registered datasets.
- AetheriaForge is contract-driven rather than domain-hardcoded. Registered datasets can vary by schema and source system, but supported execution depends on the declared file or table format, schema contract, and the forge logic implemented in this repository.

## Execution Profile

v1.x execution is pandas-based. File-backed datasets and Spark/Unity Catalog tables are materialized into pandas DataFrames on the driver before transformation, resolution, temporal reconciliation, and scoring run. Real workloads therefore need to fit driver memory or be pre-filtered and partitioned upstream; this repository does not claim cluster-distributed Spark-native execution.

## Quickstart

### Install via pip

The fastest way to get the AetheriaForge package into your environment:

```bash
pip install etherealogic-aetheriaforge
```

This installs the full AetheriaForge package — contract-driven transformation, coherence scoring, exact-match entity resolution, `latest_wins` temporal reconciliation, schema enforcement, evidence writing, orchestration, and bundled contract and policy templates.

For enterprise file ingestion (Excel, XML, ORC, Avro, Parquet):

```bash
pip install "etherealogic-aetheriaforge[ingest]"
```

### Clone and develop locally

To run the full test suite or contribute:

```bash
git clone https://github.com/Org-EthereaLogic/AetheriaForge.git
cd AetheriaForge

make sync   # installs runtime + dev dependencies via uv
make test   # runs the pytest suite
```

### Databricks Bundle and App Deployment

```bash
# First prove the catalog exists for your profile.
make bundle-catalog-check CATALOG=my_catalog PROFILE=<profile>

# Validate bundle wiring against that catalog.
make bundle-validate CATALOG=my_catalog PROFILE=<profile>

# Deploy bundle resources and start the Databricks App.
make app-deploy CATALOG=my_catalog PROFILE=<profile>
```

Direct CLI path:

```bash
databricks catalogs get my_catalog -p <profile>
databricks bundle validate -p <profile> --target dev --var="catalog=my_catalog"
databricks bundle deploy -p <profile> --target dev --var="catalog=my_catalog"
databricks apps start aetheriaforge -p <profile>
databricks apps deploy -p <profile> --target dev --var="catalog=my_catalog"
databricks apps get aetheriaforge -p <profile> -o json
```

`bundle validate` proves bundle, auth, and resource resolution. `databricks apps get` is the proof surface for `SUCCEEDED` plus `RUNNING`.

### Notebook Import

Import the `notebooks/` directory into your Databricks workspace to run the forge pipeline from the deployed bundle or standalone from GitHub. Notebooks prefer the workspace source tree when bundle-synced under `/Workspace/...` and otherwise install AetheriaForge from GitHub. Real dataset execution requires:

- a registered forge contract with source format and either a landing path or table name
- a schema contract defining the target layer's expected columns and types
- optional resolution and temporal policies for cross-source datasets
- for Databricks volume-backed files, use `/Volumes/...` notebook paths; avoid `/dbfs/Volumes/...`

## AI-Assisted Setup

If you use an AI coding agent (Claude Code, Cursor, GitHub Copilot Workspace, or similar), paste the prompt below directly into your agent session. The agent will clone the repository, install dependencies, run the full test suite, and walk you through the Databricks deployment — no manual steps required.

**Before you start, have these ready:**

- Python 3.11+
- `uv` package manager — install with `pip install uv`
- Databricks CLI configured with a valid profile — run `databricks auth login` if needed
- A Databricks workspace with Unity Catalog enabled
- The name of your Unity Catalog catalog

**Copy and paste this prompt into your AI coding agent:**

````
I want to set up AetheriaForge — a Databricks-deployable intelligent data
transformation engine that coherence-scores every Medallion layer transformation,
resolves entities across source systems, reconciles temporal conflicts, and
surfaces queryable evidence in a four-tab operator dashboard.

Repository: https://github.com/Org-EthereaLogic/AetheriaForge

Please complete these steps in order. Stop at any failure and report it before continuing.

1. Clone the repository:
   git clone https://github.com/Org-EthereaLogic/AetheriaForge.git
   cd AetheriaForge

2. Install dependencies (requires uv):
   make sync
   If uv is not installed: pip install uv

3. Run the full test suite. The full pytest suite must pass before proceeding:
   make test

4. Read these files to understand the configuration model before deployment:
   - README.md
   - templates/forge_contract.yml
   - templates/resolution_policy.yml
   - templates/schema_contract.yml

5. Ask me for my Databricks setup details:
   - My Unity Catalog catalog name
   - My Databricks CLI profile name
   Then confirm the catalog is reachable:
   make bundle-catalog-check CATALOG=<my_catalog> PROFILE=<my_profile>

6. Validate the Asset Bundle against my workspace:
   make bundle-validate CATALOG=<my_catalog> PROFILE=<my_profile>

7. If validation passes, deploy the bundle and start the Databricks App:
   make app-deploy CATALOG=<my_catalog> PROFILE=<my_profile>

8. Verify the deployment succeeded:
   databricks apps get aetheriaforge -p <my_profile> -o json
   Confirm the status shows SUCCEEDED and RUNNING, then report the app URL.

After every step, report what happened. Do not skip a step or proceed past any
error without explaining it and asking me how to continue.
````

## Scope Boundary

AetheriaForge validates the coherence-scored transformation model using registered datasets in a local and Databricks environment. It supports contract-driven execution for heterogeneous tabular schemas across common enterprise file formats (`csv`, `tsv`, `parquet`, `json`/`jsonl`, `excel`, `xml`, `orc`, `avro`, `fixed-width`) and Spark/Unity Catalog tables. It does not yet constitute production-scale proof across every schema shape, every data size, or every multi-workspace deployment pattern. The Databricks deployment path still requires a workspace with Unity Catalog enabled.

## Engineering Signals

- GitHub Actions workflow: [ci.yml](https://github.com/Org-EthereaLogic/AetheriaForge/actions/workflows/ci.yml)

## Additional Documentation

- [Architecture and design rationale](specs/AF-SDD-001_Architecture_Blueprint.md)
- [Implementation plan](specs/AF-IP-001_Implementation_Plan.md)

## Part of the EthereaLogic Databricks Suite

AetheriaForge is the second product in the **EthereaLogic Databricks Suite** — a portfolio of Databricks-deployable applications addressing the full lifecycle of data reliability in enterprise Lakehouse platforms.

| Product | Core Job | Primary Layer |
| ------- | -------- | ------------- |
| [DriftSentinel](https://github.com/Org-EthereaLogic/DriftSentinel) | Detect drift, block bad publishes | Bronze (detection) + Silver/Gold (gating) |
| **AetheriaForge** | Transform, reconcile, forge clean data | Silver (transformation engine) |
| EthereaLogic Suite (bundled) | Full governed Medallion pipeline | Bronze to Silver to Gold |

When both products are deployed, AetheriaForge emits transformation events that DriftSentinel consumes for smarter publication gating, and DriftSentinel can feed drift payloads back for evidence-backed follow-up actions. Each product operates independently when the other is absent.

---

MIT License. See [LICENSE](LICENSE) for details.
