Metadata-Version: 2.4
Name: pipelx
Version: 1.0.1
Summary: Pipelx
Home-page: https://github.com/Pacome-Gapelbe/pipelinex
Author: Pacome
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Requires-Python: ==3.12.*
Description-Content-Type: text/markdown
Requires-Dist: argparse==1.4.0
Requires-Dist: tabulate==0.8.9
Requires-Dist: PyYAML==6.0.3
Requires-Dist: ansible-core==2.17.8
Requires-Dist: Jinja2==3.1.6
Requires-Dist: joblib==1.3.2
Requires-Dist: PyMySQL==0.7.11
Requires-Dist: psycopg2-binary==2.9.10
Requires-Dist: numpy==1.26.4
Requires-Dist: snowflake-connector-python[pandas]==3.15.0
Requires-Dist: pipelinewise-singer-python==3.0.2
Requires-Dist: python-pidfile==3.0.0
Requires-Dist: pymongo<4.12,>=4.7
Requires-Dist: tzlocal<4.1,>=2.0
Requires-Dist: slackclient==2.9.4
Requires-Dist: sqlparse==0.5.3
Requires-Dist: psutil==5.9.5
Requires-Dist: ujson==5.12.0
Requires-Dist: dnspython==2.1.*
Requires-Dist: boto3<1.27,>=1.21
Requires-Dist: chardet==4.0.0
Requires-Dist: backports.tarfile==1.2.0
Provides-Extra: test
Requires-Dist: pre-commit==2.21.0; extra == "test"
Requires-Dist: flake8==7.3.0; extra == "test"
Requires-Dist: pytest==9.0.3; extra == "test"
Requires-Dist: pytest-dependency==0.4.0; extra == "test"
Requires-Dist: pytest-cov==4.1.0; extra == "test"
Requires-Dist: python-dotenv==0.19.1; extra == "test"
Requires-Dist: pylint==4.0.5; extra == "test"
Requires-Dist: ruff==0.15.11; extra == "test"
Requires-Dist: pytest-timer~=0.0; extra == "test"
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Pipelx

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache2-yellow.svg)](https://opensource.org/licenses/Apache-2.0)

Pipelx is a community-maintained fork of PipelineWise, an open-source ELT (Extract, Load, Transform) framework built on the Singer specification for reliable data replication and analytics ingestion.

Pipelx enables organizations to replicate data from operational systems into data warehouses, analytics platforms, and cloud storage with minimal configuration while preserving the simplicity and flexibility that made PipelineWise successful.

## Why Pipelx?

PipelineWise has provided a robust foundation for data replication and analytics workloads for many years. As upstream maintenance slowed, many organizations continued to depend on the platform in production.

Pipelx was created to provide a sustainable continuation of the project by:

* Maintaining compatibility with existing PipelineWise deployments
* Supporting modern Python versions and dependencies
* Delivering bug fixes and security updates
* Preserving the existing Singer connector ecosystem
* Encouraging community-driven development and contributions
* Providing a long-term path forward for organizations using PipelineWise

## Key Features

* **ELT-first architecture** designed for modern analytics workflows
* **Singer-compatible ecosystem** of taps and targets
* **Automatic schema evolution** when source structures change
* **YAML-based configuration** for version-controlled pipelines
* **Incremental and full-table replication** support
* **Load-time transformations** for masking and data filtering
* **Lightweight deployment model** with no additional services required
* **Extensible connector framework** for custom integrations

## Compatibility

Pipelx is designed to remain highly compatible with PipelineWise.

Existing configurations, pipelines, Singer taps, Singer targets, operational workflows, and deployment patterns should continue to work with minimal or no modification.

Organizations currently running PipelineWise can migrate to Pipelx gradually while preserving their existing investments and operational processes.

## Installation

```bash
pip install pipelx
```

## Project Status

Pipelx is actively maintained as an independent fork of PipelineWise.

The project focuses on:

* Stability
* Compatibility
* Modernization
* Dependency maintenance
* Community contributions

Bug reports, feature requests, and contributions are welcome.

## Documentation

Project Repository:

https://github.com/Pacome-Gapelbe/pipelinex

The remainder of this document describes the supported connectors, deployment options, development workflow, and operational guidance inherited from the PipelineWise ecosystem.


## Connectors

Tap extracts data from any source and write it to a standard stream in a JSON-based format, and target
consumes data from taps and do something with it, like load it into a file, API or database


| Type      | Name       | Extra | Latest Version | Description                                          |
|-----------|------------|-------|----------------|------------------------------------------------------|
| Tap       | **[Postgres](https://github.com/transferwise/pipelinewise-tap-postgres)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-postgres.svg)](https://badge.fury.io/py/pipelinewise-tap-postgres) | Extracts data from PostgreSQL databases. Supporting Log-Based, Key-Based Incremental and Full Table replications |
| Tap       | **[MySQL](https://github.com/transferwise/pipelinewise-tap-mysql)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-mysql.svg)](https://badge.fury.io/py/pipelinewise-tap-mysql) | Extracts data from MySQL databases. Supporting Log-Based, Key-Based Incremental and Full Table replications |
| Tap       | **[Kafka](https://github.com/transferwise/pipelinewise-tap-kafka)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-kafka.svg)](https://badge.fury.io/py/pipelinewise-tap-kafka) | Extracts data from Kafka topics |
| Tap       | **[S3 CSV](https://github.com/transferwise/pipelinewise-tap-s3-csv)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-s3-csv.svg)](https://badge.fury.io/py/pipelinewise-tap-s3-csv) | Extracts data from S3 csv files (currently a fork of tap-s3-csv because we wanted to use our own auth method) |
| Tap       | **[Zendesk](https://github.com/singer-io/tap-zendesk)** | | [![PyPI version](https://badge.fury.io/py/tap-zendesk.svg)](https://badge.fury.io/py/tap-zendesk) | Extracts data from Zendesk using OAuth and Key-Based incremental replications |
| Tap       | **[Snowflake](https://github.com/transferwise/pipelinewise-tap-snowflake)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-snowflake.svg)](https://badge.fury.io/py/pipelinewise-tap-snowflake) | Extracts data from Snowflake databases. Supporting Key-Based Incremental and Full Table replications |
| Tap       | **[Salesforce](https://github.com/singer-io/tap-salesforce)** | | [![PyPI version](https://badge.fury.io/py/tap-salesforce.svg)](https://badge.fury.io/py/tap-salesforce) | Extracts data from Salesforce database using BULK and REST extraction API with Key-Based incremental replications |
| Tap       | **[Jira](https://github.com/singer-io/tap-jira)** | | [![PyPI version](https://badge.fury.io/py/tap-jira.svg)](https://badge.fury.io/py/tap-jira) | Extracts data from Atlassian Jira using Base auth or OAuth credentials |
| Tap       | **[MongoDB](https://github.com/transferwise/pipelinewise-tap-mongodb)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-mongodb.svg)](https://badge.fury.io/py/pipelinewise-tap-mongodb) | Extracts data from MongoDB databases. Supporting Log-Based and Full Table replications |
| Tap       | **[Google Analytics](https://github.com/transferwise/pipelinewise-tap-google-analytics)** | Extra | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-google-analytics.svg)](https://badge.fury.io/py/tap-adwords) | Extracts data from Google Analytics |
| Tap       | **[Oracle](https://github.com/transferwise/pipelinewise-tap-oracle)** | Extra | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-oracle.svg)](https://badge.fury.io/py/pipelinewise-tap-oracle) | Extracts data from Oracle databases. Supporting Log-Based, Key-Based Incremental and Full Table replications |
| Tap       | **[Zuora](https://github.com/transferwise/pipelinewise-tap-zuora)** | Extra | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-zuora.svg)](https://badge.fury.io/py/pipelinewise-tap-zuora) | Extracts data from Zuora database using AQAA and REST extraction API with Key-Based incremental replications |
| Tap       | **[GitHub](https://github.com/transferwise/pipelinewise-tap-github)** |       | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-github.svg)](https://badge.fury.io/py/pipelinewise-tap-github) | Extracts data from GitHub API using Personal Access Token and Key-Based incremental replications |
| Tap       | **[Shopify](https://github.com/singer-io/tap-shopify)** | Extra | [![PyPI version](https://badge.fury.io/py/tap-shopify.svg)](https://badge.fury.io/py/tap-shopify) | Extracts data from Shopify API using Personal App API Password and date based incremental replications |
| Tap       | **[Slack](https://github.com/transferwise/pipelinewise-tap-slack)** |       | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-slack.svg)](https://badge.fury.io/py/pipelinewise-tap-slack) | Extracts data from a Slack API using Bot User Token and Key-Based incremental replications |
| Tap       | **[Mixpanel](https://github.com/transferwise/pipelinewise-tap-mixpanel)** |       | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-mixpanel.svg)](https://badge.fury.io/py/pipelinewise-tap-mixpanel) | Extracts data from the Mixpanel API. |
| Tap       | **[Twilio](https://github.com/transferwise/pipelinewise-tap-twilio)** |       | [![PyPI version](https://badge.fury.io/py/pipelinewise-tap-twilio.svg)](https://badge.fury.io/py/pipelinewise-tap-twilio) | Extracts data from the Twilio API using OAuth and Key-Based incremental replications. |
| Target    | **[Postgres](https://github.com/transferwise/pipelinewise-target-postgres)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-target-postgres.svg)](https://badge.fury.io/py/pipelinewise-target-postgres) | Loads data from any tap into PostgreSQL database |
| Target    | **[Redshift](https://github.com/transferwise/pipelinewise-target-redshift)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-target-redshift.svg)](https://badge.fury.io/py/pipelinewise-target-redshift) | Loads data from any tap into Amazon Redshift Data Warehouse |
| Target    | **[Snowflake](https://github.com/transferwise/pipelinewise-target-snowflake)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-target-snowflake.svg)](https://badge.fury.io/py/pipelinewise-target-snowflake) | Loads data from any tap into Snowflake Data Warehouse |
| Target    | **[S3 CSV](https://github.com/transferwise/pipelinewise-target-s3-csv)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-target-s3-csv.svg)](https://badge.fury.io/py/pipelinewise-target-s3-csv) | Uploads data from any tap to S3 in CSV format |
| Transform | **[Field](https://github.com/transferwise/pipelinewise-transform-field)** | | [![PyPI version](https://badge.fury.io/py/pipelinewise-transform-field.svg)](https://badge.fury.io/py/pipelinewise-transform-field) | Transforms fields from any tap and sends the results to any target. Recommended for data masking/ obfuscation |

**Note**: Extra connectors are experimental connectors and written by community contributors. These connectors are not maintained regularly and not installed by default. To install the extra packages use the `--connectors=all` option when installing PipelineWise.

### Running from docker

If you have [Docker](https://www.docker.com/) installed then using docker is the recommended and easiest method to start using PipelineWise.

#### Use official image

PipelineWise images are built on each release and available on [Dockerhub](https://hub.docker.com/r/transferwiseworkspace/pipelinewise)

    ```sh
    $ docker pull transferwiseworkspace/pipelinewise
    ```

#### Build your own docker image

1. Build an executable docker image that has every required dependency and is isolated from your host system.

By default, the image will build with *all* connectors. In order to keep image size small, we strongly recommend you change it to just the connectors you need by supplying the `--build-arg` command:

    ```sh
    $ docker build --build-arg connectors=tap-mysql,target-snowflake -t pipelinewise:latest .
    ```

2. Once the image is ready, create an alias to the docker wrapper script:

    ```sh
    $ alias pipelinewise="$(PWD)/bin/pipelinewise-docker"
    ```

3. Check if the installation was successful by running the `pipelinewise status` command:

    ```sh
    $ pipelinewise status

    Tap ID    Tap Type      Target ID     Target Type      Enabled    Status    Last Sync    Last Sync Result
    --------  ------------  ------------  ---------------  ---------  --------  -----------  ------------------
    0 pipeline(s)
    ```

You can run any pipelinewise command at this point. Tutorials to create and run pipelines is at [creating pipelines](https://transferwise.github.io/pipelinewise/installation_guide/creating_pipelines.html).

**Running tests**:

* To run unit tests, follow the instructions in the [Building from source](#building-from-source) section.
* To run integration and unit tests, follow the instructions in the [Developing with Docker](#developing-with-docker) section.

### Building from source

1. Make sure that all dependencies are installed on your system:
    * Python 3.x
    * python3-dev
    * python3-venv
    * mongo-tools
    * mbuffer

2. Run the Makefile that installs the PipelineWise CLI and all supported singer connectors into separate virtual environments:

    ```shell
    $ make pipelinewise  all_connectors
    ```
    Press `Y` to accept the license agreement of the required singer components. To automate the installation and accept every license agreement run:
    ```shell
    $ make pipelinewise all_connectors -e pw_acceptlicenses=y
    ```
    And to install only a specific list of singer connectors:
    ```shell
    $ make connectors -e pw_connector=<connector_1>,<connector_2>
    ```

   Run `make` or `make -h` to see the help for Makefile and all options.

3. To start the CLI you need to activate the CLI virtual environment and set `PIPELINEWISE_HOME` environment variable:

    ```sh
    $ source {ACTUAL_ABSOLUTE_PATH}/.virtualenvs/pipelinewise/bin/activate
    $ export PIPELINEWISE_HOME={ACTUAL_ABSOLUTE_PATH}
    ```
    (The `ACTUAL_ABSOLUTE_PATH` differs on every system, running `make -h` prints the correct commands for CLI)

4. Check if the installation was successful by running the `pipelinewise status` command:

    ```sh
    $ pipelinewise status

    Tap ID    Tap Type      Target ID     Target Type      Enabled    Status    Last Sync    Last Sync Result
    --------  ------------  ------------  ---------------  ---------  --------  -----------  ------------------
    0 pipeline(s)
    ```

You can run any pipelinewise command at this point. Tutorials to create and run pipelines can be found here: [creating pipelines](https://transferwise.github.io/pipelinewise/installation_guide/creating_pipelines.html).

**To run unit tests**:

```sh
$ pytest --ignore tests/end_to_end
```

To run unit tests and generate code coverage:

```
$ coverage run -m pytest --ignore tests/end_to_end && coverage report
```

To generate code coverage HTML report.

```
$ coverage run -m pytest --ignore tests/end_to_end && coverage html -d coverage_html
```

**Note**: The HTML report will be generated in `coverage_html/index.html`

**To run integration and end-to-end tests**:

To run integration and end-to-end tests you need to use the [Docker Development Environment](dev-project/README.md). This will spin up a pre-configured PipelineWise project with pre-configured source and target databases in several docker containers which is required for the end-to-end test cases.

## Developing with Docker

If you have [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/) installed,
you can create a local development environment that includes not only the PipelineWise executables but also a
pre-configured development project with some databases as source and targets for a more convenient
development experience and to run integration and end-to-end tests.

For further instructions about setting up local development environment go to
[Test Project for Docker Development Environment](dev-project/README.md).


## Contribution

To add new taps and targets follow the instructions on
* [Contribution Page](https://transferwise.github.io/pipelinewise/project/contribution.html)
* [Code contribution guide](./CONTRIBUTING.md)


## Links

* [PipelineWise documentation](https://transferwise.github.io/pipelinewise/)
* [Singer ETL specification](https://github.com/singer-io/getting-started/blob/master/docs/SPEC.md)
* [Singer.io community slack channel](https://singer-slackin.herokuapp.com/)


## License

Apache License Version 2.0

See [LICENSE](LICENSE) to see the full text.

**Important Note:**

PipelineWise as a standalone software is licensed under Apache License Version 2.0 but bundled components can
use different licenses and may overwrite the terms and conditions detailed in Apache License Version 2.0.
You can customise which connectors you want to include into the final PipelineWise build and the final license of
your build depends on the included connectors. For further details please check the
[Licenses](https://transferwise.github.io/pipelinewise/project/licenses.html) section in the documentation.

