Metadata-Version: 2.4
Name: ubunye-engine
Version: 0.9.2
Summary: Config-first, backend-neutral data and ML pipelines that you can check on a laptop and prove anywhere
Author-email: Thabang Mashinini-Sekgoto <thabangline@gmail.com>
Maintainer: Ubunye AI Ecosystems
License-Expression: MIT
Project-URL: Homepage, https://www.tmashininisekgoto.com
Project-URL: Documentation, https://ubunye-ai-ecosystems.github.io/ubunye_engine/
Project-URL: Repository, https://github.com/ubunye-ai-ecosystems/ubunye_engine
Project-URL: Issues, https://github.com/ubunye-ai-ecosystems/ubunye_engine/issues
Project-URL: Project page, https://www.tmashininisekgoto.com/work/ubunye-engine
Keywords: spark,pyspark,pandas,databricks,etl,mlops,data-pipelines,machine-learning,config-driven,reproducibility,lineage,kubernetes
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6
Requires-Dist: jinja2>=3
Requires-Dist: pydantic>=2
Requires-Dist: typer>=0.15.4
Requires-Dist: narwhals>=2.0
Provides-Extra: spark
Requires-Dist: pyspark>=3.5; extra == "spark"
Provides-Extra: delta
Requires-Dist: delta-spark>=4.0; extra == "delta"
Provides-Extra: pandas
Requires-Dist: pandas>=2.2; extra == "pandas"
Requires-Dist: pyarrow>=14; platform_system != "Windows" and extra == "pandas"
Requires-Dist: pyarrow>=24; platform_system == "Windows" and extra == "pandas"
Provides-Extra: ml
Requires-Dist: scikit-learn; extra == "ml"
Requires-Dist: mlflow; extra == "ml"
Provides-Extra: ml-torch
Requires-Dist: scikit-learn; extra == "ml-torch"
Requires-Dist: torch; extra == "ml-torch"
Requires-Dist: mlflow; extra == "ml-torch"
Provides-Extra: rest
Requires-Dist: requests>=2.28; extra == "rest"
Provides-Extra: objectstore
Requires-Dist: fsspec>=2023.1.0; extra == "objectstore"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov>=4; extra == "dev"
Requires-Dist: pytest-timeout; extra == "dev"
Requires-Dist: hypothesis>=6; extra == "dev"
Requires-Dist: requests>=2.28; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: pandas>=2.2; extra == "dev"
Requires-Dist: pyarrow>=14; platform_system != "Windows" and extra == "dev"
Requires-Dist: pyarrow>=24; platform_system == "Windows" and extra == "dev"
Requires-Dist: narwhals>=2; extra == "dev"
Requires-Dist: jsonschema>=4.18; extra == "dev"
Requires-Dist: opentelemetry-sdk>=1.24; extra == "dev"
Requires-Dist: mcp<3,>=2; extra == "dev"
Provides-Extra: databricks
Requires-Dist: databricks-sdk>=0.20; extra == "databricks"
Provides-Extra: otel
Requires-Dist: opentelemetry-sdk>=1.24; extra == "otel"
Requires-Dist: opentelemetry-exporter-otlp>=1.24; extra == "otel"
Provides-Extra: aws
Requires-Dist: boto3>=1.28; extra == "aws"
Provides-Extra: gcp
Requires-Dist: google-cloud-secret-manager>=2.16; extra == "gcp"
Provides-Extra: azure
Requires-Dist: azure-identity>=1.15; extra == "azure"
Requires-Dist: azure-keyvault-secrets>=4.7; extra == "azure"
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2; extra == "mcp"
Dynamic: license-file

<p align="center">
  <img src="docs/assets/ubunye-logo-white-v2.png" alt="Ubunye Engine" width="400"/>
</p>

<p align="center">
  <em>Ubunye (oo-BOON-yeh): isiZulu for "unity"</em>
</p>

<h3 align="center">One framework. Every pipeline. Any environment.</h3>

<p align="center">
  <a href="https://ubunye-ai-ecosystems.github.io/ubunye_engine/">Docs</a> •
  <a href="#docs">Docs map</a> •
  <a href="#try-it">Try it</a> •
  <a href="https://github.com/ubunye-ai-ecosystems/ubunye_engine/discussions">Community</a>
</p>

<p align="center">
  <img src="docs/assets/diagrams/home.svg" alt="Write a task once, run it unchanged on pandas, Spark, Databricks or any cloud; every run leaves a record, and ubunye prove checks the rows match" width="100%"/>
</p>

---

A **data pipeline** moves data from one place to another (a database to a file, a REST
API to a data warehouse) and usually changes it on the way. Most of the work is
plumbing: connections, credentials, a framework's quirks, the same read, transform,
write scaffold again and again.

**Ubunye Engine writes that plumbing for you.** You describe what to read and write in a
short YAML file and put your change to the data in a normal Python class. Ubunye does the
connections, the compute engine (Apache Spark, or pandas on a laptop with no Java) and the
read and write loop. The same folder runs on your laptop today and on a cluster tomorrow,
with no code changes.

What you get:

- **One standard layout.** Every project is use case, pipeline, task, run by one CLI.
  The words are explained in [Concepts](https://ubunye-ai-ecosystems.github.io/ubunye_engine/getting_started/concepts/).
- **A record of every run.** Each run can leave a run record with a hash of every row
  it read and wrote, so two runs can be compared exactly.
- **Checks before you run.** `ubunye plan` and `ubunye doctor` say what would fail, and
  why, before any data moves. Errors say what went wrong and suggest the fix.
- **Open plugins.** Connectors, backends, model registries and run record stores are
  added from outside, with no engine edits.
- **Types that ship.** The package carries its type information, and a type checker
  guards every merge.

---

## Try it

On a laptop, with no Java and no cloud account:

```bash
pip install "ubunye-engine[pandas]"
```

<!-- quickstart:begin -->
```bash
ubunye init -d pipelines -u demo -p starter -t filter_adults
ubunye plan -d pipelines -u demo -p starter -t filter_adults --backend pandas
ubunye run -d pipelines -u demo -p starter -t filter_adults --backend pandas --lineage
ubunye lineage list -d pipelines -u demo -p starter -t filter_adults
```
<!-- quickstart:end -->

`init` makes a task folder with a sample CSV, `plan` checks it without moving data, `run`
writes Parquet and keeps a run record, and `lineage list` shows that record. The test
suite runs these four commands exactly as written. What each one did, and how to run the
same folder on Spark: the [Quickstart](https://ubunye-ai-ecosystems.github.io/ubunye_engine/getting_started/quickstart/).

---

## Docs

The full docs live at
[ubunye-ai-ecosystems.github.io/ubunye_engine](https://ubunye-ai-ecosystems.github.io/ubunye_engine/).
These are the pages most people need:

| You want to | Read |
|---|---|
| Install it | [Installation](https://ubunye-ai-ecosystems.github.io/ubunye_engine/getting_started/install/) |
| Run a first task | [Quickstart](https://ubunye-ai-ecosystems.github.io/ubunye_engine/getting_started/quickstart/) |
| Learn the words (use case, pipeline, task, profile, run record) | [Concepts](https://ubunye-ai-ecosystems.github.io/ubunye_engine/getting_started/concepts/) |
| Write a `config.yaml` | [Config overview](https://ubunye-ai-ecosystems.github.io/ubunye_engine/config/overview/), then [Inputs and outputs](https://ubunye-ai-ecosystems.github.io/ubunye_engine/config/io/) |
| Check the data a task writes | [Expectations](https://ubunye-ai-ecosystems.github.io/ubunye_engine/config/expectations/) |
| Read what a run did (the run record) | [Lineage commands](https://ubunye-ai-ecosystems.github.io/ubunye_engine/cli/#ubunye-lineage) and [the run record](https://ubunye-ai-ecosystems.github.io/ubunye_engine/architecture/adr-006-run-record/) |
| Run on pandas or Spark | [Execution backends](https://ubunye-ai-ecosystems.github.io/ubunye_engine/backends/) |
| Look up a command | [CLI reference](https://ubunye-ai-ecosystems.github.io/ubunye_engine/cli/) |
| Call it from Python or a notebook | [Python API](https://ubunye-ai-ecosystems.github.io/ubunye_engine/api/) |
| Add a language model step | [Language model steps](https://ubunye-ai-ecosystems.github.io/ubunye_engine/patterns/llm/) |
| Copy a working example | [Examples](https://ubunye-ai-ecosystems.github.io/ubunye_engine/examples/) |
| Read from or write to a system | [Connectors](https://ubunye-ai-ecosystems.github.io/ubunye_engine/connectors/overview/) |
| Deploy it | [Deployment](https://ubunye-ai-ecosystems.github.io/ubunye_engine/deployment/) |
| Fix something that went wrong | [Troubleshooting](https://ubunye-ai-ecosystems.github.io/ubunye_engine/troubleshooting/), then [Errors](https://ubunye-ai-ecosystems.github.io/ubunye_engine/errors/) |
| Move from 0.8 to 0.9 | [Upgrading](https://ubunye-ai-ecosystems.github.io/ubunye_engine/upgrading/) |
| See what changed | [Changelog](https://ubunye-ai-ecosystems.github.io/ubunye_engine/changelog/) |

Worked examples, grouped by where they run (Databricks, a laptop, Docker, Kubernetes),
live in [**ubunye-examples**](https://github.com/ubunye-ai-ecosystems/ubunye-examples).

---

## Proof

One task, unchanged, gave the same rows on pandas (no Java), local Spark, Kubernetes,
AWS Glue, GCP Dataproc, Azure Container Apps and Databricks: the same data hash in all
seven, compared from their run records by `ubunye prove`. See the
[latest results](https://ubunye-ai-ecosystems.github.io/ubunye_engine/proving-ground/latest/).

---

## What Ubunye is not

- Not an agent framework: use LangChain or CrewAI for that.
- Not an orchestrator: use Airflow, Prefect or Dagster (`ubunye export` writes an Airflow
  DAG or a Databricks job for you).
- Not a compute engine: it runs on Spark, or on pandas for laptop-sized data.

Ubunye is the **standard layer** between your data sources and your applications. It
makes the plumbing boring so you can focus on what matters.

**Not yet:** data drift detection is a pattern you write in a task, not a built-in
feature. EMR Serverless is built but has not run the proving workload, and Hadoop/YARN is
not tested. Neither is claimed until a run says so.

---

## Get involved

New connectors, bug fixes, typos, or just telling us what you are building: all help.

- [Report a bug](https://github.com/ubunye-ai-ecosystems/ubunye_engine/issues)
- [Request a feature](https://github.com/ubunye-ai-ecosystems/ubunye_engine/discussions)
- [Read the contributing guide](CONTRIBUTING.md)
- Star the repo if you find it useful; it helps more than you'd think

## License

[MIT License](LICENSE)

---

<p align="center">
  Built with 🇿🇦 by <a href="https://github.com/ubunye-ai-ecosystems">Ubunye AI Ecosystems</a>
</p>
