Metadata-Version: 2.4
Name: ubunye-engine
Version: 0.7.0
Summary: Config-first, backend-neutral data and ML pipelines that you can check on a laptop and prove anywhere
Author-email: Thabang Mashinini-Sekgoto <thabangline@gmail.com>
Maintainer: Ubunye AI Ecosystems
License-Expression: MIT
Project-URL: Homepage, https://www.tmashininisekgoto.com
Project-URL: Documentation, https://ubunye-ai-ecosystems.github.io/ubunye_engine/
Project-URL: Repository, https://github.com/ubunye-ai-ecosystems/ubunye_engine
Project-URL: Issues, https://github.com/ubunye-ai-ecosystems/ubunye_engine/issues
Project-URL: Project page, https://www.tmashininisekgoto.com/work/ubunye-engine
Keywords: spark,pyspark,pandas,databricks,etl,mlops,data-pipelines,machine-learning,config-driven,reproducibility,lineage,kubernetes
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: Topic :: Software Development :: Libraries :: Application Frameworks
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6
Requires-Dist: jinja2>=3
Requires-Dist: pydantic>=2
Requires-Dist: typer>=0.15.4
Requires-Dist: narwhals>=2.0
Provides-Extra: spark
Requires-Dist: pyspark>=3.5; extra == "spark"
Provides-Extra: delta
Requires-Dist: delta-spark>=4.0; extra == "delta"
Provides-Extra: pandas
Requires-Dist: pandas>=2.2; extra == "pandas"
Requires-Dist: pyarrow>=14; platform_system != "Windows" and extra == "pandas"
Requires-Dist: pyarrow>=24; platform_system == "Windows" and extra == "pandas"
Provides-Extra: ml
Requires-Dist: scikit-learn; extra == "ml"
Requires-Dist: mlflow; extra == "ml"
Provides-Extra: ml-torch
Requires-Dist: scikit-learn; extra == "ml-torch"
Requires-Dist: torch; extra == "ml-torch"
Requires-Dist: mlflow; extra == "ml-torch"
Provides-Extra: rest
Requires-Dist: requests>=2.28; extra == "rest"
Provides-Extra: objectstore
Requires-Dist: fsspec>=2023.1.0; extra == "objectstore"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov>=4; extra == "dev"
Requires-Dist: pytest-timeout; extra == "dev"
Requires-Dist: hypothesis>=6; extra == "dev"
Requires-Dist: requests>=2.28; extra == "dev"
Requires-Dist: black; extra == "dev"
Requires-Dist: ruff; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: mypy>=1.8; extra == "dev"
Requires-Dist: pandas>=2.2; extra == "dev"
Requires-Dist: pyarrow>=14; platform_system != "Windows" and extra == "dev"
Requires-Dist: pyarrow>=24; platform_system == "Windows" and extra == "dev"
Requires-Dist: narwhals>=2; extra == "dev"
Requires-Dist: jsonschema>=4.18; extra == "dev"
Requires-Dist: opentelemetry-sdk>=1.24; extra == "dev"
Requires-Dist: mcp<3,>=2; extra == "dev"
Provides-Extra: databricks
Requires-Dist: databricks-sdk>=0.20; extra == "databricks"
Provides-Extra: otel
Requires-Dist: opentelemetry-sdk>=1.24; extra == "otel"
Requires-Dist: opentelemetry-exporter-otlp>=1.24; extra == "otel"
Provides-Extra: aws
Requires-Dist: boto3>=1.28; extra == "aws"
Provides-Extra: gcp
Requires-Dist: google-cloud-secret-manager>=2.16; extra == "gcp"
Provides-Extra: azure
Requires-Dist: azure-identity>=1.15; extra == "azure"
Requires-Dist: azure-keyvault-secrets>=4.7; extra == "azure"
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2; extra == "mcp"
Dynamic: license-file

<p align="center">
  <img src="docs/assets/ubunye-logo-white-v2.png" alt="Ubunye Engine" width="400"/>
</p>

<p align="center">
  <em>Ubunye (oo-BOON-yeh) — isiZulu for "unity"</em>
</p>

<h3 align="center">One framework. Every pipeline. Any environment.</h3>

<p align="center">
  <a href="https://ubunye-ai-ecosystems.github.io/ubunye_engine">Docs</a> •
  <a href="#quickstart">Quickstart</a> •
  <a href="#why-ubunye">Why Ubunye</a> •
  <a href="https://github.com/ubunye-ai-ecosystems/ubunye_engine/discussions">Community</a>
</p>

---

## Hey there 👋

A **data pipeline** is a program that moves data from one place to another — a database to a file, a REST API to a data warehouse — and usually reshapes the data along the way. Building one from scratch is mostly plumbing: wire up the connection, juggle credentials, learn a framework's quirks, write the same *"read → transform → write"* scaffold for the tenth time this year. It's a lot of glue code standing between you and the three lines that actually matter.

**Ubunye Engine writes that plumbing for you.** You describe the pipeline in a short YAML file and put your transformation in a normal Python class. Ubunye takes care of connections, the compute engine (Apache Spark), and the read/write loop.

Same pipeline runs on your laptop today and on a production cluster tomorrow, with no code changes. That sentence is tested, not hoped: a build job runs one pipeline on six environments and fails if the outputs differ by a single byte.

---

## What the engine gives you today

- **Proven portability.** The same task runs on a laptop, in Docker, on Kubernetes, against object storage, through the cloud submit path, and on Databricks. One output hash across all of them, checked on every change.
- **Models saved anywhere.** The model registry writes to a local folder, a Databricks volume, S3 or GCS, chosen purely by the path. New storage kinds are one class and one entry point.
- **A truly open plugin system.** Connectors, storage backends and registries are all added from the outside, with no engine edits and no inheritance required. A test proves it with a connector the engine has never seen.
- **Types that ship.** The package carries its type information, and a type checker guards every merge.
- **Errors that help.** Failures say what went wrong, show the context, and suggest the fix.
- **Nothing here is decorative.** Eleven worked examples have been run for real. Claims that could not be executed were removed from these docs rather than left to mislead.

---

## Quickstart

On a laptop, with no Java and no cloud account:

```bash
pip install "ubunye-engine[pandas]"
```

<!-- quickstart:begin -->
```bash
ubunye init -d pipelines -u demo -p starter -t filter_adults
ubunye plan -d pipelines -u demo -p starter -t filter_adults --backend pandas
ubunye run -d pipelines -u demo -p starter -t filter_adults --backend pandas --lineage
ubunye lineage list -d pipelines -u demo -p starter -t filter_adults
```
<!-- quickstart:end -->

These four commands are run by the test suite exactly as written. `init` makes a
folder, and the folder is the whole task:

```
pipelines/demo/starter/filter_adults/
  config.yaml            what to read and write
  transformations.py     your code
  data/people.csv        a small sample to start from
```

`plan` checks it without moving any data, `run` reads the CSV and writes
Parquet, and `lineage list` shows the record the run left, including a hash of
every row written. The code is one line:

```python
class FilterAdults(Task):
    def transform(self, sources):
        people = sources["people"]
        return {"adults": people[people["age"] >= 18]}
```

That line means the same thing in pandas and in Spark. With Java installed
(`pip install "ubunye-engine[spark]"`), leave out `--backend pandas` and the same
folder runs on Spark, leaving the same data hash. On Databricks, call it from a
notebook and the notebook's session is used:

```python
import ubunye
outputs = ubunye.run_task("pipelines/demo/starter/filter_adults")
```

Step by step: the [Quickstart](https://ubunye-ai-ecosystems.github.io/ubunye_engine/getting_started/quickstart/).
Realistic end to end examples live in
[**ubunye-examples**](https://github.com/ubunye-ai-ecosystems/ubunye-examples).

---

## Why Ubunye

We've all been there. You join a new team, open the repo, and find five Spark projects — each structured differently, each with its own way of handling configs, credentials, and deployment. One uses a JSON file, another has everything hardcoded, a third has a 300-line bash script that "Dave wrote and it just works."

Ubunye says: **let's agree on how pipelines look.** One folder structure. One config format. One CLI. Whether you're building an ETL job, a feature pipeline, or an ML training run.

| Without Ubunye | With Ubunye |
|---|---|
| Every project looks different | One standard: `use_case / pipeline / task` |
| Spark setup scattered everywhere | Engine handles it from YAML config |
| Credentials hardcoded or inconsistent | `{{ env.DB_PASSWORD }}` everywhere |
| "Works on my machine" | Same config runs local, YARN, K8s, Databricks |
| New teammate needs a week to onboard | `ubunye init` and they're running in minutes |

---

## How It Works

Three simple ideas:

**Config over code.** Your pipeline is a YAML file. Inputs, outputs, Spark settings, scheduling — all declared, not coded.

**Plugins for everything.** The `format` field in your config picks which *connector* to use. A connector is a small Python class that knows how to read from or write to one specific place (a database, a REST API, a cloud bucket). Built-ins include `hive`, `jdbc`, `delta`, `s3`, `unity`, and `rest_api`. Need a new data source? Write one and register it — Ubunye discovers plugins automatically.

**Folders as architecture.** Pipelines are organized as `project / use_case / pipeline / task`. The CLI uses this structure for scaffolding, execution, and discovery:

```
pipelines/
  fraud_detection/
    ingestion/
      claim_etl/
      policy_etl/
    feature_engineering/
      claim_features/
    risk_scoring/
      train_model/
      score_claims/
```

---

## What Can You Build With It

**ETL pipelines** — move data between Hive, JDBC databases, Delta Lake, S3, REST APIs. Config-driven, scheduled, reproducible.

**ML training and inference** — define your model behind a simple contract, let the engine handle versioning, storage, and deployment.

**RAG document pipelines** — ingest documents, extract text, chunk, compute embeddings, load into a vector store. All from YAML.

**Feature engineering** — compute features once, write to a shared table, reuse across use cases.

**Data drift detection** — monitor feature distributions between runs, flag when things shift.

Check out the [Patterns](https://ubunye-ai-ecosystems.github.io/ubunye_engine) section in our docs for full examples.

---

## Examples

All worked examples live in [**ubunye-examples**](https://github.com/ubunye-ai-ecosystems/ubunye-examples).
They install the engine from PyPI, so what you run there is what you get from `pip install`.

They are grouped by where they run. The same task folder is used in every environment.
Only environment variables change.

**Databricks** (free workspace is enough)

| Example | What it shows |
|---|---|
| Tables and SQL pushdown | read a table, push a join down as SQL, write with `merge` |
| REST API ingestion | the `rest_api` connector against a public weather API |
| Unstructured files | binary file reading and text chunking |
| The ML lifecycle | features, train, quality gate, registry, promote, score |
| RAG | embeddings and a chat model, retrieval, grounded answers |
| Fine-tune an open LLM | a large model labels data, DistilBERT learns from it |
| Data quality | a contract with severities, bad rows quarantined |
| Model monitoring | drift, decay, and a rollback that is a decision, not a reflex |

**Local machine, Docker, and Kubernetes** (no cloud account needed)

| Example | What it shows |
|---|---|
| Run anywhere | one task, identical output hash on local Spark, Docker, Kubernetes, object storage and Databricks |
| JDBC | a partitioned parallel read from a real public PostgreSQL database |
| RAG and fine-tuning on open models | the same pipelines with local open source models instead of hosted endpoints |
| The three-framework race | the same model in scikit-learn, PyTorch and TensorFlow behind identical configs |

**AWS and GCP**

Submit scripts and CI jobs exist for EMR Serverless and Dataproc Serverless.
They are written and documented but need a cloud account to run, and the CI
jobs say plainly when they were skipped rather than run.

## Connectors

| Format | Read | Write | Description |
|---|:---:|:---:|---|
| `hive` | ✓ | ✓ | Apache Hive tables |
| `jdbc` | ✓ | ✓ | PostgreSQL, MySQL, Teradata, and more |
| `delta` | ✓ | ✓ | Delta Lake (standalone or Unity Catalog) |
| `s3` | ✓ | ✓ | S3, HDFS, or local filesystem |
| `unity` | ✓ | ✓ | Databricks Unity Catalog |
| `binary` | ✓ | | Binary files (images, PDFs) |
| `rest_api` | ✓ | ✓ | REST APIs with pagination and auth |

Want to add one? See the [plugin guide](https://ubunye-ai-ecosystems.github.io/ubunye_engine).

---

## Run Anywhere

The same task folder runs on every environment below. This is tested, not claimed:
a CI job runs one task on each and fails the build if the output hashes differ.

| Environment | How you launch it |
|---|---|
| Your laptop | `ubunye run ...` with local Spark |
| Docker | one container image, same entry point |
| Kubernetes | the same image as a Job |
| Databricks | `ubunye.run_task()` from a notebook or a bundle job |
| AWS EMR Serverless, GCP Dataproc | `spark-submit` with `python -m ubunye` |

One rule makes this work: the task never chooses its own cluster. Which Spark
master to use belongs to whoever launches the job, so leave `spark.master` out
of your config. On a laptop the runner sets `local[*]`. On a cloud the platform
sets it, and the engine refuses a config that tries to override it, because a
silent single-node run on paid compute is worse than an error.

---------------------------------------|-----------------------------------------------|
| **Your laptop**                       | `spark.master: "local[*]"`                    |
| **Hadoop / YARN cluster**             | `spark.master: "yarn"`                        |
| **Kubernetes**                        | `spark.master: "k8s://..."`                   |
| **Databricks notebooks or jobs**      | Call `ubunye.run_task()` from Python — Ubunye picks up the active session |
| **AWS EMR**                           | Runs as an EMR Step                           |

Don't recognise some of these? That's fine — you only need one. If you're starting out, `local[*]` runs Spark on your own machine with no setup.

---

## Jinja Templating

Anywhere a string appears in your YAML, you can plug in a variable using `{{ … }}` syntax (this is called **Jinja templating**). That's how you keep secrets out of your config, change paths per environment, and inject the run date from the CLI:

```yaml
# Environment variables
password: "{{ env.DB_PASSWORD }}"

# CLI variables (--var ds=2025-01-01)
path: "s3a://bucket/{{ ds }}/"

# Defaults
path: "s3a://bucket/{{ ds | default('2025-01-01') }}/"
```

---

## CLI

```bash
ubunye init     -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # scaffold
ubunye validate -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # check config
ubunye plan     -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # preview plan
ubunye run      -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # execute
ubunye test run -d ./pipelines -u <use_case> -p <pipeline> -t <task>   # test mode
ubunye lineage list -d ./pipelines -u <use_case> -p <pipeline> -t <task>  # run history
ubunye models list -u <use_case> -m <model> -s <store>                 # model versions
```

## Python API

```python
import ubunye

# Run from Databricks or any Python environment
outputs = ubunye.run_task(task_dir="./pipelines/...", mode="DEV", dt="2024-06-01")

# Multiple tasks
results = ubunye.run_pipeline(
    usecase_dir="./pipelines", usecase="fraud", package="etl",
    tasks=["claim_etl", "features"], mode="DEV",
)
```

---

## What Ubunye Is Not

It's not an agent framework — use LangChain or CrewAI for that.
It's not an orchestrator — use Airflow, Prefect, or Dagster.
It's not a compute engine — it runs on Spark.

Ubunye is the **standardization layer** between your data sources and your applications. It makes the plumbing boring so you can focus on what matters.

---

## Roadmap

- [x] Config-driven ETL pipelines
- [x] Multi-environment profiles
- [x] Jinja templating
- [x] Plugin-based connectors
- [x] CLI scaffolding and execution
- [x] Pydantic config validation
- [x] ML model contract
- [x] Model registry with versioning
- [x] Lineage tracking
- [x] Python API for Databricks
- [x] Databricks Asset Bundles deployment
- [x] Dev notebook scaffolding
- [ ] Data drift detection
- [ ] `ubunye deploy` CLI command

---

## Get Involved

We'd love your help. Whether it's a new connector, a bug fix, a typo, or just telling us what you're building — all contributions matter.

- 🐛 [Report a bug](https://github.com/ubunye-ai-ecosystems/ubunye_engine/issues)
- 💡 [Request a feature](https://github.com/ubunye-ai-ecosystems/ubunye_engine/discussions)
- 📖 [Read the contributing guide](CONTRIBUTING.md)
- ⭐ Star the repo if you find it useful — it helps more than you'd think

---

## License

[MIT License](LICENSE)

---

<p align="center">
  Built with 🇿🇦 by <a href="https://github.com/ubunye-ai-ecosystems">Ubunye AI Ecosystems</a>
</p>
