Metadata-Version: 2.4
Name: lakelogic
Version: 1.40.6
Summary: A declarative, contract-driven medallion pipeline engine for data mesh architectures. Write once. Run on Spark, Polars, or DuckDB.
Author-email: LakeLogic Team <lakelogic@gmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: data-contracts,data-engineering,data-governance,data-pipeline,data-quality,delta-lake,duckdb,etl,lakehouse,lineage,medallion-architecture,polars,quarantine,schema-validation,spark
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: deltalake>=0.15.0
Requires-Dist: duckdb>=0.9.0
Requires-Dist: httpx<1,>=0.27.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: loguru>=0.7.0
Requires-Dist: polars>=0.20.0
Requires-Dist: pyarrow>=23.0.1
Requires-Dist: pydantic<3,>=2.0.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: sqlglot>=20.0.0
Requires-Dist: typer>=0.9.0
Provides-Extra: ai
Requires-Dist: anthropic>=0.18.0; extra == 'ai'
Requires-Dist: google-genai>=0.5.0; extra == 'ai'
Requires-Dist: instructor>=1.5.0; extra == 'ai'
Requires-Dist: openai>=1.0.0; extra == 'ai'
Requires-Dist: typing-extensions>=4.12.0; extra == 'ai'
Provides-Extra: all
Requires-Dist: deltalake>=0.15.0; extra == 'all'
Requires-Dist: duckdb>=0.9.0; extra == 'all'
Requires-Dist: lxml>=4.9.0; extra == 'all'
Requires-Dist: openpyxl>=3.1.0; extra == 'all'
Requires-Dist: polars>=0.20.0; extra == 'all'
Requires-Dist: pyarrow>=23.0.1; extra == 'all'
Requires-Dist: sqlglot>=20.0.0; extra == 'all'
Requires-Dist: typer>=0.9.0; extra == 'all'
Provides-Extra: api
Requires-Dist: requests>=2.31.0; extra == 'api'
Provides-Extra: aws-messaging
Requires-Dist: boto3>=1.28.0; extra == 'aws-messaging'
Provides-Extra: azure
Requires-Dist: adlfs>=2023.10.0; extra == 'azure'
Requires-Dist: azure-identity>=1.15.0; extra == 'azure'
Requires-Dist: azure-keyvault-secrets>=4.7.0; extra == 'azure'
Requires-Dist: azure-storage-blob>=12.19.0; extra == 'azure'
Requires-Dist: cryptography>=41.0.0; extra == 'azure'
Requires-Dist: databricks-sdk>=0.18.0; extra == 'azure'
Requires-Dist: fsspec>=2023.10.0; extra == 'azure'
Provides-Extra: azure-messaging
Requires-Dist: azure-eventgrid>=4.17.0; extra == 'azure-messaging'
Requires-Dist: azure-identity>=1.15.0; extra == 'azure-messaging'
Requires-Dist: azure-servicebus>=7.11.0; extra == 'azure-messaging'
Provides-Extra: azuresql
Requires-Dist: azure-identity>=1.15.0; extra == 'azuresql'
Requires-Dist: pyodbc>=5.0.0; extra == 'azuresql'
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery>=3.11.0; extra == 'bigquery'
Provides-Extra: cli
Requires-Dist: typer>=0.9.0; extra == 'cli'
Provides-Extra: cloud
Requires-Dist: azure-eventgrid>=4.17.0; extra == 'cloud'
Requires-Dist: azure-servicebus>=7.11.0; extra == 'cloud'
Requires-Dist: google-cloud-bigquery>=3.11.0; extra == 'cloud'
Requires-Dist: google-cloud-pubsub>=2.18.0; extra == 'cloud'
Requires-Dist: google-cloud-secret-manager>=2.16.0; extra == 'cloud'
Requires-Dist: google-cloud-storage>=2.10.0; extra == 'cloud'
Requires-Dist: google-genai>=0.5.0; extra == 'cloud'
Requires-Dist: snowflake-connector-python>=3.5.0; extra == 'cloud'
Provides-Extra: databases
Requires-Dist: azure-identity>=1.15.0; extra == 'databases'
Requires-Dist: psycopg2-binary>=2.9.0; extra == 'databases'
Requires-Dist: pymongo>=4.6.0; extra == 'databases'
Requires-Dist: pymysql>=1.1.0; extra == 'databases'
Requires-Dist: pyodbc>=5.0.0; extra == 'databases'
Provides-Extra: delta
Requires-Dist: azure-identity>=1.15.0; extra == 'delta'
Requires-Dist: azure-storage-blob>=12.19.0; extra == 'delta'
Requires-Dist: boto3>=1.28.0; extra == 'delta'
Requires-Dist: databricks-sdk>=0.18.0; extra == 'delta'
Requires-Dist: deltalake>=0.15.0; extra == 'delta'
Requires-Dist: google-cloud-storage>=2.10.0; extra == 'delta'
Provides-Extra: demo
Requires-Dist: deltalake>=0.15.0; extra == 'demo'
Requires-Dist: duckdb>=0.9.0; extra == 'demo'
Requires-Dist: faker>=24.0.0; extra == 'demo'
Requires-Dist: lxml>=4.9.0; extra == 'demo'
Requires-Dist: openpyxl>=3.1.0; extra == 'demo'
Requires-Dist: pdfplumber>=0.10.0; extra == 'demo'
Requires-Dist: polars>=0.20.0; extra == 'demo'
Requires-Dist: pyarrow>=23.0.1; extra == 'demo'
Requires-Dist: pypandoc-binary>=1.13; extra == 'demo'
Requires-Dist: rapidocr-onnxruntime>=1.4.0; extra == 'demo'
Requires-Dist: reportlab>=4.0.0; extra == 'demo'
Requires-Dist: spacy>=3.7.0; extra == 'demo'
Requires-Dist: sqlglot>=20.0.0; extra == 'demo'
Requires-Dist: textblob>=0.18.0; extra == 'demo'
Requires-Dist: unstructured[csv,doc,docx,md,pptx,rst,tsv,xlsx]>=0.12.0; extra == 'demo'
Provides-Extra: dev
Requires-Dist: black>=23.0.0; extra == 'dev'
Requires-Dist: commitizen>=3.0.0; extra == 'dev'
Requires-Dist: git-cliff>=2.0.0; extra == 'dev'
Requires-Dist: hypothesis>=6.100.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Provides-Extra: dlt
Requires-Dist: dlt[parquet]>=1.0; extra == 'dlt'
Requires-Dist: pyarrow>=23.0.1; extra == 'dlt'
Provides-Extra: docs
Requires-Dist: mkdocs-jupyter>=0.24.0; extra == 'docs'
Requires-Dist: mkdocs-material>=9.0.0; extra == 'docs'
Requires-Dist: mkdocstrings[python]>=0.20.0; extra == 'docs'
Provides-Extra: duckdb
Requires-Dist: deltalake>=0.15.0; extra == 'duckdb'
Requires-Dist: duckdb>=0.9.0; extra == 'duckdb'
Requires-Dist: lxml>=4.9.0; extra == 'duckdb'
Requires-Dist: openpyxl>=3.1.0; extra == 'duckdb'
Requires-Dist: pandas>=2.0.0; extra == 'duckdb'
Requires-Dist: pyarrow>=23.0.1; extra == 'duckdb'
Provides-Extra: engines
Requires-Dist: deltalake>=0.15.0; extra == 'engines'
Requires-Dist: duckdb>=0.9.0; extra == 'engines'
Requires-Dist: lxml>=4.9.0; extra == 'engines'
Requires-Dist: openpyxl>=3.1.0; extra == 'engines'
Requires-Dist: polars>=0.20.0; extra == 'engines'
Requires-Dist: pyarrow>=23.0.1; extra == 'engines'
Requires-Dist: sqlglot>=20.0.0; extra == 'engines'
Provides-Extra: enterprise
Requires-Dist: dataprofiler>=0.9.0; extra == 'enterprise'
Requires-Dist: nbclient>=0.9.0; extra == 'enterprise'
Requires-Dist: nbformat>=5.9.0; extra == 'enterprise'
Requires-Dist: presidio-analyzer>=2.2.0; extra == 'enterprise'
Requires-Dist: presidio-anonymizer>=2.2.0; extra == 'enterprise'
Requires-Dist: pyspark>=3.3.0; extra == 'enterprise'
Provides-Extra: extraction
Requires-Dist: pypandoc-binary>=1.13; extra == 'extraction'
Requires-Dist: spacy>=3.7.0; extra == 'extraction'
Requires-Dist: textblob>=0.18.0; extra == 'extraction'
Requires-Dist: unstructured[csv,doc,docx,md,pptx,rst,tsv,xlsx]>=0.12.0; extra == 'extraction'
Provides-Extra: extraction-ocr
Requires-Dist: pdfplumber>=0.10.0; extra == 'extraction-ocr'
Requires-Dist: pypandoc-binary>=1.13; extra == 'extraction-ocr'
Requires-Dist: rapidocr-onnxruntime>=1.4.0; extra == 'extraction-ocr'
Requires-Dist: spacy>=3.7.0; extra == 'extraction-ocr'
Requires-Dist: textblob>=0.18.0; extra == 'extraction-ocr'
Requires-Dist: unstructured[csv,doc,docx,md,pptx,rst,tsv,xlsx]>=0.12.0; extra == 'extraction-ocr'
Provides-Extra: gcp-messaging
Requires-Dist: google-cloud-pubsub>=2.18.0; extra == 'gcp-messaging'
Provides-Extra: integrations
Requires-Dist: azure-eventgrid>=4.17.0; extra == 'integrations'
Requires-Dist: azure-identity>=1.15.0; extra == 'integrations'
Requires-Dist: azure-servicebus>=7.11.0; extra == 'integrations'
Requires-Dist: boto3>=1.28.0; extra == 'integrations'
Requires-Dist: google-cloud-pubsub>=2.18.0; extra == 'integrations'
Requires-Dist: paramiko>=4.0.1; extra == 'integrations'
Requires-Dist: requests>=2.31.0; extra == 'integrations'
Provides-Extra: kafka
Requires-Dist: kafka-python>=2.0.2; extra == 'kafka'
Provides-Extra: mongodb
Requires-Dist: pymongo>=4.6.0; extra == 'mongodb'
Provides-Extra: mysql
Requires-Dist: pymysql>=1.1.0; extra == 'mysql'
Provides-Extra: nlp
Requires-Dist: spacy>=3.7.0; extra == 'nlp'
Requires-Dist: textblob>=0.18.0; extra == 'nlp'
Provides-Extra: notebook
Requires-Dist: nbclient>=0.9.0; extra == 'notebook'
Requires-Dist: nbformat>=5.9.0; extra == 'notebook'
Provides-Extra: notifications
Requires-Dist: apprise>=1.7.0; extra == 'notifications'
Requires-Dist: azure-identity>=1.15.0; extra == 'notifications'
Requires-Dist: azure-keyvault-secrets>=4.7.0; extra == 'notifications'
Requires-Dist: boto3>=1.28.0; extra == 'notifications'
Requires-Dist: cryptography>=41.0.0; extra == 'notifications'
Requires-Dist: google-cloud-secret-manager>=2.16.0; extra == 'notifications'
Requires-Dist: hvac>=2.0.0; extra == 'notifications'
Requires-Dist: jinja2>=3.1.0; extra == 'notifications'
Provides-Extra: notify
Requires-Dist: apprise>=1.7.0; extra == 'notify'
Requires-Dist: hvac>=2.0.0; extra == 'notify'
Provides-Extra: pdf
Requires-Dist: pdfplumber>=0.10.0; extra == 'pdf'
Provides-Extra: pii
Requires-Dist: dataprofiler>=0.9.0; extra == 'pii'
Requires-Dist: presidio-analyzer>=2.2.0; extra == 'pii'
Requires-Dist: presidio-anonymizer>=2.2.0; extra == 'pii'
Provides-Extra: polars
Requires-Dist: deltalake>=0.15.0; extra == 'polars'
Requires-Dist: lxml>=4.9.0; extra == 'polars'
Requires-Dist: openpyxl>=3.1.0; extra == 'polars'
Requires-Dist: polars>=0.20.0; extra == 'polars'
Provides-Extra: postgresql
Requires-Dist: azure-identity>=1.15.0; extra == 'postgresql'
Requires-Dist: psycopg2-binary>=2.9.0; extra == 'postgresql'
Provides-Extra: profiling
Requires-Dist: dataprofiler>=0.9.0; extra == 'profiling'
Requires-Dist: presidio-analyzer>=2.2.0; extra == 'profiling'
Requires-Dist: presidio-anonymizer>=2.2.0; extra == 'profiling'
Provides-Extra: review
Requires-Dist: datacontract-cli>=0.10; extra == 'review'
Requires-Dist: ruff<0.14,>=0.6; extra == 'review'
Requires-Dist: sqlfluff<5,>=4.2.1; extra == 'review'
Provides-Extra: sftp
Requires-Dist: paramiko>=4.0.1; extra == 'sftp'
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python>=3.5.0; extra == 'snowflake'
Provides-Extra: spark
Requires-Dist: pyspark>=3.3.0; extra == 'spark'
Provides-Extra: sse
Requires-Dist: sseclient-py>=1.8.0; extra == 'sse'
Provides-Extra: streaming
Requires-Dist: kafka-python>=2.0.2; extra == 'streaming'
Requires-Dist: sseclient-py>=1.8.0; extra == 'streaming'
Requires-Dist: websocket-client>=1.6.0; extra == 'streaming'
Provides-Extra: synthetic
Requires-Dist: faker>=24.0.0; extra == 'synthetic'
Requires-Dist: reportlab>=4.0.0; extra == 'synthetic'
Provides-Extra: websocket
Requires-Dist: websocket-client>=1.6.0; extra == 'websocket'
Description-Content-Type: text/markdown

# LakeLogic

**Define data trust once. Enforce it from development to production.**

[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://LakeLogic.github.io/LakeLogic/)
[![PyPI](https://img.shields.io/pypi/v/lakelogic?logo=pypi&logoColor=white)](https://pypi.org/project/lakelogic/)
[![CI](https://github.com/LakeLogic/LakeLogic/actions/workflows/ci-gate.yml/badge.svg)](https://github.com/LakeLogic/LakeLogic/actions/workflows/ci-gate.yml)
[![codecov](https://codecov.io/gh/LakeLogic/LakeLogic/graph/badge.svg)](https://codecov.io/gh/LakeLogic/LakeLogic)
[![Python](https://img.shields.io/badge/python-3.10+-blue?logo=python&logoColor=white)](https://www.python.org)
[![License](https://img.shields.io/badge/license-Apache%202.0-green)](LICENSE)

LakeLogic is an open-source framework for executable data contracts. Describe a data product's schema, quality rules, PII handling, lineage, and materialization in YAML, then run that contract with Polars, DuckDB, or Spark.

Use the same contract to get fast feedback locally, check changes in CI/CD, and govern pipeline execution in your lakehouse. Records that fail row-level rules can be retained with their failure reasons instead of being silently discarded.

[**Run the five-minute Colab quickstart**](https://colab.research.google.com/github/LakeLogic/LakeLogic/blob/main/examples/colab/00_quickstart.ipynb) · [**Read the documentation**](https://lakelogic.github.io/LakeLogic/) · [**Browse the examples**](https://lakelogic.github.io/LakeLogic/examples.html)

![LakeLogic Architecture](docs/assets/lakelogic_architecture.png)

> **Project status:** LakeLogic is currently beta software. Test contracts and engine-specific behaviour against representative data before production adoption.

## The Problem It Solves

Data teams repeatedly rebuild the same controls in notebooks and pipelines:

- schema checks and business rules;
- accepted and quarantined outputs;
- PII handling and lineage metadata;
- incremental processing and materialization;
- deployment checks for contract changes.

LakeLogic puts those expectations in a version-controlled contract and provides the execution machinery around it. Business meaning stays visible in YAML; complex transformations can remain in normal, testable Python or SQL.

## Quick Start

Install the base package:

```bash
pip install lakelogic
```

The fastest complete introduction is the [Google Colab quickstart](https://colab.research.google.com/github/LakeLogic/LakeLogic/blob/main/examples/colab/00_quickstart.ipynb). It creates sample data, executes a contract, and shows accepted and quarantined records without requiring a local Spark environment.

A contract starts with the fields and rules that matter to the data product:

```yaml
version: 1.0.0
dataset: orders

info:
  title: E-Commerce Orders
  owner: data-team@company.com
  target_layer: silver

model:
  fields:
    - name: order_id
      type: integer
      required: true
    - name: customer_email
      type: string
      required: true
      pii: true
      masking: partial
    - name: amount
      type: float
      required: true

quality:
  row_rules:
    - name: valid_email
      sql: "customer_email LIKE '%@%.%'"
    - name: positive_amount
      sql: "amount > 0"
```

Run it through the Python API:

```python
from lakelogic import DataProcessor

processor = DataProcessor("orders_contract.yaml", engine="polars")
result = processor.run_source("orders.csv")

print(f"Accepted: {result.good_count}")
print(f"Quarantined: {result.bad_count}")
print(f"Quality score: {result.quality_score:.1f}")
```

For a validation run, LakeLogic returns the accepted and quarantined rows with counts and diagnostic context. Materialization, alerts, retention, and catalog behaviour depend on the contract, selected engine, and connected infrastructure.

To check a contract before deployment:

```bash
lakelogic validate \
  --contract orders_contract.yaml \
  --gates breaking_change,pii_classification,lineage_break
```

CI gates analyse contract declarations and the comparison context supplied to them. Add the command to your pull-request workflow to reject a change when a configured gate fails.

## Core Capabilities

| Capability | What LakeLogic provides |
| :--- | :--- |
| **Executable contracts** | Strictly parsed YAML for schemas, row rules, dataset rules, service levels, lineage, and materialization. |
| **Quality and quarantine** | Accepted and failed records, rule-level diagnostics, run counts, and quality scores. |
| **Contract checks in CI/CD** | Static gates for breaking schema changes, PII declarations, and lineage changes when the required comparison context is available. |
| **Multiple execution engines** | A common contract model across Polars, DuckDB, and Spark, with documented engine-specific boundaries. |
| **Lakehouse patterns** | Incremental processing, Delta or Iceberg outputs, merge strategies, SCD Type 2, dependencies, and external transformation logic. |
| **Operational evidence** | Structured run logs, execution metadata, lineage evidence, and optional notification integrations. |

See the [complete capability matrix](docs/capabilities.md) before choosing an engine or storage format.

## Engine Support

| Engine | Best suited to | Installation and boundaries |
| :--- | :--- | :--- |
| **Polars** | Local development, notebooks, CI, and fast single-node processing | Included in the base package. Delta support uses delta-rs. |
| **DuckDB** | Local analytical SQL and embedded workflows | Included in the base package. Some catalog and materialization combinations differ from Spark. |
| **Spark** | Distributed lakehouse workloads and managed catalogs such as Unity Catalog | Install with `pip install "lakelogic[spark]"`. Managed catalog features depend on the Spark platform and its configuration. |

The contract model is shared, but engines are not identical. Review [engine and format capabilities](docs/capabilities.md) for supported combinations.

## Where Data Mesh Fits

LakeLogic can provide shared contract machinery for a data mesh while domain teams retain ownership of business meaning.

| Data-mesh principle | LakeLogic's role |
| :--- | :--- |
| **Domain ownership** | Domain teams version contracts alongside the data products they own. |
| **Data as a product** | Contracts make schemas, rules, service expectations, and dependencies explicit. |
| **Self-service platform** | Teams reuse common validation and execution interfaces across supported engines. |
| **Federated governance** | Platform standards can be expressed as shared defaults and checked alongside domain-specific rules. |

LakeLogic does not create organisational ownership, access policies, alert delivery, or regulatory compliance by itself. It supplies contract declarations, runtime controls, and evidence that can participate in those wider systems.

## Learn by Doing

| Guide | Use it to explore |
| :--- | :--- |
| [Quickstart](https://colab.research.google.com/github/LakeLogic/LakeLogic/blob/main/examples/colab/00_quickstart.ipynb) | Your first contract, generated data, validation, and quarantine. |
| [Data Quality and Trust](https://colab.research.google.com/github/LakeLogic/LakeLogic/blob/main/examples/colab/01_data_quality_trust.ipynb) | Schema rules, business rules, reconciliation, and medallion flows. |
| [Compliance and Governance](https://colab.research.google.com/github/LakeLogic/LakeLogic/blob/main/examples/colab/02_compliance_governance.ipynb) | PII-handling and governance patterns that must be combined with organisational controls. |
| [Engine and Scale](https://colab.research.google.com/github/LakeLogic/LakeLogic/blob/main/examples/colab/03_engine_scale.ipynb) | Polars, DuckDB, Spark, incremental execution, and dimensional modelling. |
| [Developer Experience](https://colab.research.google.com/github/LakeLogic/LakeLogic/blob/main/examples/colab/04_developer_experience.ipynb) | Validation, diagnostics, CI/CD, and development workflows. |
| [Data Generation and AI](https://colab.research.google.com/github/LakeLogic/LakeLogic/blob/main/examples/colab/05_data_generation_ai.ipynb) | Synthetic test data and optional AI-assisted workflows. |
| [Integrations](https://colab.research.google.com/github/LakeLogic/LakeLogic/blob/main/examples/colab/06_integrations.ipynb) | dbt, dlt, databases, streaming sources, and notifications. |

## Documentation

- [Installation and optional dependencies](docs/installation.md)
- [Complete contract reference](docs/contract_template.md)
- [Capabilities and engine boundaries](docs/capabilities.md)
- [Pipeline concepts](docs/pipelines.md)
- [Reconciliation](docs/reconciliation.md)
- [Notifications](docs/notifications.md)
- [Full documentation site](https://lakelogic.github.io/LakeLogic/)

## Contributing

Contributions and issue reports are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the development workflow and [developer installation](docs/installation.md#developer-installation) for environment setup.

## License

LakeLogic is available under the [Apache 2.0 License](LICENSE).
