Metadata-Version: 2.5
Name: redcliff
Version: 0.1.1
Summary: Open-source visual pipelines with lineage built in. Legacy ETL in, clean Spark out.
Project-URL: Homepage, https://github.com/Fullcountr/redcliff
Project-URL: Playground, https://fullcountr.github.io/redcliff/
Project-URL: Documentation, https://github.com/Fullcountr/redcliff/tree/main/docs
Project-URL: Issues, https://github.com/Fullcountr/redcliff/issues
Author: Harishwar Reddy Korpol
License-Expression: Apache-2.0
Keywords: data lineage,data quality,dbt,duckdb,etl,informatica,migration,openlineage,powercenter,pyspark,spark
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Code Generators
Requires-Python: >=3.11
Requires-Dist: sqlglot>=30
Provides-Extra: cli
Requires-Dist: duckdb>=1.4; extra == 'cli'
Requires-Dist: pyyaml>=6; extra == 'cli'
Provides-Extra: dev
Requires-Dist: duckdb>=1.4; extra == 'dev'
Requires-Dist: jsonschema>=4; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: pyyaml>=6; extra == 'dev'
Provides-Extra: yaml
Requires-Dist: pyyaml>=6; extra == 'yaml'
Description-Content-Type: text/markdown

# redcliff

The Python engine behind [Redcliff](https://github.com/Fullcountr/redcliff), an open-source visual
pipeline tool with lineage built in: **legacy ETL in, clean Spark out.**

```text
pip install "redcliff[cli]"
```

- **Import** legacy SQL (11 dialects, BTEQ scripts) and Informatica PowerCenter exports (mappings,
  mapplets, sessions, workflows, parameter files) into a small YAML pipeline spec, with a
  migration scorecard.
- **Compile** the spec to Spark SQL, PySpark, DuckDB SQL, a dbt project or a Lakeflow
  Declarative Pipeline.
- **Trace** column-level lineage, a column's blast radius, and the source rows behind any row;
  export OpenLineage.
- **Check** data quality expectations, **diff** two versions of a pipeline, and let AI agents
  edit specs through an **MCP server** that shows a diff before anything is written.

```text
redcliff import job.sql --dialect teradata -o pipeline.yaml
redcliff import wf_export.xml --params wf.par -o pipeline.yaml --scorecard scorecard.md
redcliff compile pipeline.yaml --to spark          # or pyspark, duckdb, dbt, lakeflow
redcliff lineage pipeline.yaml --impact orders.status
redcliff test pipeline.yaml                        # data quality expectations, in DuckDB
redcliff run pipeline.yaml --var start_date=2026-09-01
redcliff diff --git origin/main --data
redcliff mcp --root .
```

The core depends only on [sqlglot](https://github.com/tobymao/sqlglot), so it also runs in the
browser through Pyodide: try the [playground](https://fullcountr.github.io/redcliff/). The `cli`
extra adds PyYAML and DuckDB for running pipelines locally.

Documentation: [spec](https://github.com/Fullcountr/redcliff/blob/main/docs/spec.md),
[outputs](https://github.com/Fullcountr/redcliff/blob/main/docs/outputs.md),
[PowerCenter](https://github.com/Fullcountr/redcliff/blob/main/docs/powercenter.md),
[pull request bot](https://github.com/Fullcountr/redcliff/blob/main/docs/pr-bot.md),
[MCP server](https://github.com/Fullcountr/redcliff/blob/main/docs/mcp.md).
Apache 2.0.
