Metadata-Version: 2.4
Name: serverless-data-mesh
Version: 0.2.0
Summary: Federated data mesh framework for proof-gated, exactly-once Iceberg data product publication on AWS Lambda.
Project-URL: Homepage, https://github.com/vaquarkhan/aws-serverless-datamesh-framework
Project-URL: Documentation, https://github.com/vaquarkhan/aws-serverless-datamesh-framework#readme
Project-URL: Repository, https://github.com/vaquarkhan/aws-serverless-datamesh-framework
Project-URL: Changelog, https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/CHANGELOG.md
Project-URL: PyPI, https://pypi.org/project/serverless-data-mesh/
Author: Serverless Data Mesh Contributors
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: aws,data-as-a-product,data-mesh,domain-oriented,durable-execution,federated-data-mesh,federated-governance,iceberg,iceguard,lakehouse,lambda,pyiceberg,rule-engine,serverless,sparkrules,vaquar-pattern,veridata-recon
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: System :: Distributed Computing
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: aws-durable-execution-sdk-python>=1.0.0
Requires-Dist: boto3>=1.34
Requires-Dist: iceguard>=1.0.0
Requires-Dist: pyiceberg[glue,rest-sigv4]>=0.7.0
Requires-Dist: veridata-recon>=0.1.0
Provides-Extra: all
Requires-Dist: pyspark>=3.5.0; extra == 'all'
Requires-Dist: sparkrules>=1.2.0; extra == 'all'
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == 'dev'
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pre-commit>=4; extra == 'dev'
Requires-Dist: pytest-cov>=5; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Requires-Dist: twine>=5; extra == 'dev'
Provides-Extra: publish
Requires-Dist: build>=1.2; extra == 'publish'
Requires-Dist: twine>=5; extra == 'publish'
Provides-Extra: rules
Requires-Dist: sparkrules>=1.2.0; extra == 'rules'
Provides-Extra: spark
Requires-Dist: pyspark>=3.5.0; extra == 'spark'
Requires-Dist: sparkrules>=1.2.0; extra == 'spark'
Description-Content-Type: text/markdown

# Serverless Data Mesh

**Federated data mesh lakehouse publication on AWS Lambda - with cryptographic proof, not just green job logs.**

An open Python framework for **domain-oriented**, **federated data mesh** teams on AWS. Implements **data as a product**, **self-serve write infrastructure**, and **federated computational governance** at the Iceberg lakehouse layer.

- **Producer** domains publish governed **data products**
- **Steward** notaries enforce mesh contracts with VRP proofs
- **Publisher** zones expose consumer-ready **Iceberg data products**

Introduces the **[Vaquar Pattern](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/vaquar-pattern.md)** (PVDM): Physical → Verify → Durable → Metadata. Invariant: `commit_metadata ⟹ VRP = PASS`.

## Install

```bash
pip install serverless-data-mesh
serverless-data-mesh demo    # <60s local PVDM demo, no AWS
pip install "serverless-data-mesh[rules]"   # + SparkRules on Lambda
pip install "serverless-data-mesh[spark]"   # + PySpark + SparkRules
pip install "serverless-data-mesh[all]"     # rules + spark
pip install "serverless-data-mesh[dev]"     # pytest, ruff, pre-commit
```

**Works on **Windows, Mac, and Linux**. Uses veridata-recon cryptographic VRP when wheels are available; **pure-Python fallback** otherwise (same PASS/FAIL gate).**

## What it solves

| Problem | Answer |
|---------|--------|
| Silent data loss on backfill | VRP `FAIL` blocks Iceberg snapshot |
| "Job succeeded" is not proof | Cryptographic multiset proof per chunk |
| Lambda 15-minute limit | Durable Execution + Step Functions (90+ min) |
| Retry duplicates data | IceGuard rollback + `workload_id` checkpoints |
| Central ETL bottleneck | Per-domain Lambda writer + transaction boundary |
| Federated blast radius | Producer · Steward · Publisher accounts |

## Building blocks

| Package | Role |
|---------|------|
| [iceguard](https://pypi.org/project/iceguard/) | Physical SafeWriter, timeout rollback, S3 resume |
| [veridata-recon](https://pypi.org/project/veridata-recon/) | VRP proof generation and validation |
| [aws-durable-execution-sdk-python](https://pypi.org/project/aws-durable-execution-sdk-python/) | Cross-invocation step replay |
| [pyiceberg](https://pypi.org/project/pyiceberg/) | Glue Iceberg REST metadata via `GlueCatalogConnector` |
| [sparkrules](https://pypi.org/project/sparkrules/) | Optional DRL business rules (`[rules]` extra) |

## Quick example

```python
from serverless_data_mesh import (
    IceGuardDurableCoordinator,
    DomainTransactionBoundary,
    VRPProofGenerator,
)

boundary = DomainTransactionBoundary(
    domain_id="orders-domain",
    source_namespace="raw_orders",
    target_table="orders_curated",
    partition_spec={"dt": "2026-06-14"},
)

coordinator = IceGuardDurableCoordinator(
    durable_context=durable_ctx,
    lambda_context=lambda_ctx,
    proof_generator=VRPProofGenerator(),
    catalog_adapter=glue_adapter,
)
outcome = coordinator.run_workload(workload)
```

## Optional extras

| Extra | Adds |
|-------|------|
| `rules` | SparkRules DRL on Lambda |
| `spark` | PySpark + SparkRules |
| `all` | `rules` + `spark` |
| `dev` | pytest, ruff, mypy, pre-commit |
| `publish` | build, twine |

## Documentation

- [GitHub README](https://github.com/vaquarkhan/aws-serverless-datamesh-framework#readme)
- [Vaquar Pattern](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/vaquar-pattern.md)
- [Getting started](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/getting-started.md)
- [PyPI guide](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/blob/main/docs/pypi.md)
- [Full documentation](https://github.com/vaquarkhan/aws-serverless-datamesh-framework/tree/main/docs)

## License

Apache-2.0
