Metadata-Version: 2.4
Name: delta-audit
Version: 0.1.0
Summary: A lightweight, Spark-free Python library for inspecting, validating, and diagnosing Delta Lake tables across local and cloud storage.
Author-email: Surender Elangovan <esurender99@gmail.com>
Maintainer-email: Surender Elangovan <esurender99@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/SurenderElangovan/delta-lens
Project-URL: Documentation, https://github.com/SurenderElangovan/delta-lens
Project-URL: Repository, https://github.com/SurenderElangovan/delta-lens
Project-URL: Issues, https://github.com/SurenderElangovan/delta-lens/issues
Project-URL: Changelog, https://github.com/SurenderElangovan/delta-lens/blob/main/CHANGELOG.md
Keywords: delta-lake,deltalake,delta,delta-log,lakehouse,diagnostics,inspection,validation,health-check,transaction-log,metadata,checkpoint,parquet,data-engineering,analytics,big-data,spark,databricks,azure,aws,gcp,adls,s3,gcs,filesystem,python
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: fsspec>=2024.6.0
Requires-Dist: pyarrow>=18.0.0
Requires-Dist: orjson>=3.10.0
Requires-Dist: pydantic>=2.8.0
Requires-Dist: typing_extensions>=4.12.2
Provides-Extra: azure
Requires-Dist: adlfs>=2024.7.0; extra == "azure"
Provides-Extra: aws
Requires-Dist: s3fs>=2024.7.0; extra == "aws"
Provides-Extra: gcp
Requires-Dist: gcsfs>=2024.7.0; extra == "gcp"
Provides-Extra: all
Requires-Dist: adlfs>=2024.7.0; extra == "all"
Requires-Dist: s3fs>=2024.7.0; extra == "all"
Requires-Dist: gcsfs>=2024.7.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.3; extra == "dev"
Requires-Dist: pytest-cov>=6.0; extra == "dev"
Requires-Dist: ruff>=0.9; extra == "dev"
Requires-Dist: mypy>=1.14; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: twine>=6.1; extra == "dev"

# Delta-audit

> **A Spark-free Python library for inspecting, validating, and diagnosing Delta Lake tables across local and cloud storage.**

[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/)
[![License](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
[![Status](https://img.shields.io/badge/status-alpha-orange.svg)]()

delta-audit is a production-oriented, read-only Python library that enables data engineers, platform engineers, and platform teams to inspect, validate, and diagnose Delta Lake tables without requiring Apache Spark.

Instead of launching a Spark cluster just to inspect a Delta table, Delta-audit reads the Delta transaction log directly from storage and provides meaningful diagnostics about table health, metadata, protocol compatibility, checkpoints, and transaction history.

---

# Features

- Spark-free Delta Lake inspection
- Read Delta transaction logs (`_delta_log`)
- Parse Delta checkpoint metadata
- Validate Delta table structure
- Detect missing transaction log versions
- Validate Delta protocol compatibility
- Inspect Delta metadata
- Calculate overall table health score
- Support Local Filesystem
- Support Amazon S3
- Support Azure Data Lake Storage Gen2
- Support Google Cloud Storage
- Lightweight and read-only
- Modular and extensible architecture

---

# Why Delta-audit?

Debugging Delta Lake tables typically requires running Apache Spark, even for simple metadata inspection.

Delta-audit removes that dependency by providing a lightweight library capable of reading and validating Delta Lake metadata directly from storage.

It helps answer questions such as:

- Is this a valid Delta table?
- What is the latest Delta version?
- Does the table contain a valid checkpoint?
- Are transaction log versions missing?
- Is the Delta protocol supported?
- Is metadata valid?
- Are referenced Parquet files missing?
- What is the overall health score?

---

# Installation

## Install from source

```bash
git clone https://github.com/SurenderElangovan/delta-lens.git

cd delta-lens

pip install -e .
```

## Install from PyPI

```bash
pip install delta-lens
```

## Optional Cloud Storage Support

### Amazon S3

```bash
pip install "delta-lens[aws]"
```

### Azure Data Lake Storage Gen2

```bash
pip install "delta-lens[azure]"
```

### Google Cloud Storage

```bash
pip install "delta-lens[gcp]"
```

### Install All Optional Dependencies

```bash
pip install "delta-lens[all]"
```

---

# Quick Start

```python
from delta_lens import scan

report = scan("/path/to/delta/table")

print(report.summary())
```

Example output:

```text
Table Health      : 96%

Latest Version    : 154

Checkpoint Version: 150

Missing Versions  : None

Metadata          : Valid

Protocol          : Reader=3 Writer=7

Warnings          : 0

Errors            : 0
```

---

# Supported Storage

| Storage | Supported |
|----------|-----------|
| Local Filesystem | Yes |
| Amazon S3 | Yes |
| Azure Data Lake Storage Gen2 | Yes |
| Google Cloud Storage | Yes |

---

# Project Structure

```text
Delta-audit/
│
├── src/
├── tests/
├── pyproject.toml
└── README.md
```

The project follows the modern **src layout** recommended for Python packages.

- **src/** – Delta-audit source code
- **tests/** – Unit and integration tests
- **pyproject.toml** – Project metadata, dependencies, and build configuration
- **README.md** – Project documentation

---

# Project Goals

Delta-audit is designed to become a production-ready diagnostics library for Delta Lake.

Primary goals include:

- Inspect Delta Lake tables without Apache Spark
- Detect common Delta Lake issues
- Validate transaction logs and checkpoints
- Analyze metadata and protocol compatibility
- Provide meaningful diagnostics
- Calculate table health scores
- Support multiple cloud storage providers
- Remain lightweight and read-only

---

# Roadmap

## Version 0.1

- Project foundation
- Modern package structure
- Storage abstraction
- Public API

## Version 0.2

- Local filesystem support
- Storage adapter architecture

## Version 0.3

- Delta transaction log parser
- Checkpoint parser

## Version 0.4

- Metadata validation
- Protocol validation

## Version 0.5

- Diagnostics engine
- Health score calculation

## Version 1.0

- Production-ready release
- Complete documentation
- Comprehensive testing
- Stable public API

---

# Contributing

Contributions are welcome.

If you would like to contribute:

1. Fork the repository.
2. Create a feature branch.
3. Add or update tests.
4. Submit a pull request.

Please ensure that new code includes appropriate documentation and follows the existing project structure.

---

# License

Delta-audit is licensed under the MIT License.

See the `LICENSE` file for additional information.

---

# Author

**Surender Elangovan**

Email: **esurender99@gmail.com**

GitHub: **https://github.com/SurenderElangovan**

---

# Support

If Delta-audit helps you in your work, consider:

- Starring the repository
- Reporting bugs
- Suggesting new features
- Contributing improvements

Community feedback and contributions help improve the project for everyone.
