Metadata-Version: 2.4
Name: omniload
Version: 0.3.0
Summary: omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.
Author: Mustafa Ersan, Sanju Shah Gupta, Saravjeet 'Aman' Singh
Author-email: Burak Karakan <burak.karakan@getbruin.com>
Maintainer-email: Andreas Motl <andreas.motl@panodata.org>
License-Expression: MIT
Project-URL: Changelog, https://omniload.readthedocs.io/changelog.html
Project-URL: Documentation, https://omniload.readthedocs.io/
Project-URL: Issues, https://github.com/panodata/omniload/issues
Project-URL: Repository, https://github.com/panodata/omniload
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Customer Service
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: End Users/Desktop
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: Manufacturing
Classifier: Intended Audience :: Science/Research
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: Telecommunications Industry
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: OS Independent
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Unix
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Adaptive Technologies
Classifier: Topic :: Communications
Classifier: Topic :: Database
Classifier: Topic :: Documentation
Classifier: Topic :: Education
Classifier: Topic :: Internet
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Office/Business
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Software Development :: Object Brokering
Classifier: Topic :: Software Development :: Pre-processors
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Version Control
Classifier: Topic :: System :: Archiving
Classifier: Topic :: System :: Clustering
Classifier: Topic :: System :: Distributed Computing
Classifier: Topic :: System :: Logging
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Networking
Classifier: Topic :: System :: Systems Administration
Classifier: Topic :: Text Processing
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: LICENSE.Apache-2.0
License-File: NOTICE
Requires-Dist: aiohttp<4
Requires-Dist: asana<3.3
Requires-Dist: click<8.5
Requires-Dist: clickhouse-connect<2,>=0.8
Requires-Dist: clickhouse-driver<0.3
Requires-Dist: clickhouse-sqlalchemy<0.4
Requires-Dist: confluent-kafka<2.15
Requires-Dist: couchbase<5,>=4
Requires-Dist: databricks-sql-connector<5
Requires-Dist: databricks-sqlalchemy<3
Requires-Dist: dataclasses-json<0.7
Requires-Dist: dlt<1.29
Requires-Dist: dlt-cratedb<0.2
Requires-Dist: duckdb<1.6
Requires-Dist: duckdb-engine<0.18
Requires-Dist: elasticsearch<10
Requires-Dist: facebook-business<26
Requires-Dist: flatten-json<0.2
Requires-Dist: gcsfs
Requires-Dist: google-ads<32
Requires-Dist: google-analytics-data<0.24
Requires-Dist: google-api-python-client<3
Requires-Dist: google-auth<3
Requires-Dist: google-cloud-bigquery-storage<3
Requires-Dist: google-cloud-spanner<4
Requires-Dist: influxdb-client<2
Requires-Dist: mysql-connector-python<10
Requires-Dist: oracledb<4
Requires-Dist: orjson<4
Requires-Dist: pendulum<4,>=3
Requires-Dist: psycopg2-binary<3
Requires-Dist: py-machineid<2
Requires-Dist: pyairtable<4
Requires-Dist: pyarrow<25,>=18
Requires-Dist: pyathena<4
Requires-Dist: pymongo<5
Requires-Dist: pymongoarrow<2
Requires-Dist: pymysql<2
Requires-Dist: pyrate-limiter<4
Requires-Dist: python-quickbooks<0.10
Requires-Dist: requests<3
Requires-Dist: rudder-sdk-python<2.2
Requires-Dist: s3fs
Requires-Dist: simple-salesforce<2
Requires-Dist: smartsheet-python-sdk<5
Requires-Dist: snowflake-connector-python<5,>=3.18
Requires-Dist: snowflake-sqlalchemy<2
Requires-Dist: sqlalchemy<2.1
Requires-Dist: sqlalchemy-bigquery<2
Requires-Dist: sqlalchemy-cratedb<1
Requires-Dist: sqlalchemy-hana<5
Requires-Dist: sqlalchemy-redshift<2
Requires-Dist: sqlalchemy-spanner<2
Requires-Dist: stripe<16
Requires-Dist: tldextract<6
Requires-Dist: toolz<2
Requires-Dist: tqdm<5
Requires-Dist: trino[sqlalchemy]==0.337.0
Requires-Dist: typer<0.27
Requires-Dist: urllib3<2.8
Provides-Extra: develop
Requires-Dist: pkginfo<2; extra == "develop"
Requires-Dist: poethepoet<1; extra == "develop"
Requires-Dist: pyproject-fmt<3; extra == "develop"
Requires-Dist: pytest-repeat<1; extra == "develop"
Requires-Dist: ruff<0.16; extra == "develop"
Requires-Dist: ty==0.0.52; extra == "develop"
Requires-Dist: validate-pyproject<1; extra == "develop"
Provides-Extra: full
Requires-Dist: omniload[ibm-db,odbc,oracle]; extra == "full"
Provides-Extra: ibm-db
Requires-Dist: ibm-db<4; platform_machine != "aarch64" and extra == "ibm-db"
Requires-Dist: ibm-db-sa<0.5; platform_machine != "aarch64" and extra == "ibm-db"
Provides-Extra: odbc
Requires-Dist: pyodbc<5.4; extra == "odbc"
Provides-Extra: oracle
Requires-Dist: oracledb>=2.0.0; extra == "oracle"
Provides-Extra: release
Requires-Dist: build<2; extra == "release"
Requires-Dist: twine<7; extra == "release"
Provides-Extra: test
Requires-Dist: boto3<2; extra == "test"
Requires-Dist: pendulum[test]<4,>=3; extra == "test"
Requires-Dist: pytest<10; extra == "test"
Requires-Dist: pytest-asyncio<2; extra == "test"
Requires-Dist: pytest-cov<8; extra == "test"
Requires-Dist: pytest-xdist[psutil]<4; extra == "test"
Requires-Dist: testcontainers[mysql,postgres]<4.15,>=4.9.1; extra == "test"
Requires-Dist: types-requests<3; extra == "test"
Requires-Dist: verlib2; extra == "test"
Dynamic: license-file

<div align="center">

# omniload
<p>Copy data between any source and any destination.</p>
<img src="https://github.com/panodata/omniload/blob/main/resources/demo.gif?raw=true" width="750" />

[![License](https://img.shields.io/github/license/panodata/omniload.svg)](https://github.com/panodata/omniload/blob/main/LICENSE)
[![Downloads / month](https://pepy.tech/badge/omniload/month)](https://pepy.tech/project/omniload/)
[![Python versions](https://img.shields.io/pypi/pyversions/omniload.svg)](https://pypi.org/project/omniload/)

</div>

## About

omniload is a polyglot data loader framework based on [dlt], the
open-source Python library for building data pipelines. dlt does the
heavy lifting (schema inference, normalization, and incremental load
state); omniload wraps it behind a single CLI and a URI scheme, so you
can copy data between any source and any destination without writing
pipeline code, either from your shell or the Python API.

omniload provides the same efficient incremental data loading modes
inherited by dlt: `append`, `merge`, and `delete+insert`.

## Install

### Container

For running `omniload` in a container environment, use the provided OCI
images.

```shell
docker run --rm ghcr.io/panodata/omniload:latest version
```

### Package

For running or installing the `omniload` Python package,
we are recommending to use [uv]. [^1][^2]
```shell
uvx omniload
```

Alternatively, if you like to install it account-wide on your system:
```shell
uv tool install omniload
```

[^1]: You can install `uv` using `pip install uv`, or another method that matches your operating system.
[^2]: While installing `omniload` with vanilla `pip` is possible, it is an order of magnitude slower.

## Synopsis

### CLI

Instruct omniload to read from the PostgreSQL table `public.some_data`,
and write to the BigQuery warehouse table `omniload.some_data`.

```shell
omniload ingest \
    --source-uri 'postgresql://admin:admin@localhost:8837/web?sslmode=disable' \
    --source-table 'public.some_data' \
    --dest-uri 'bigquery://<your-project-name>?credentials_path=/path/to/service/account.json' \
    --dest-table 'omniload.some_data'
```

### Python API

Using `omniload.run_ingest`, you can run the same
ingestion from your own application. The keyword arguments map one-to-one to the
CLI options, and it returns the dlt `LoadInfo` for the run (or `None` for a dry run).

```python
from omniload import run_ingest

info = run_ingest(
    source_uri="sqlite:///./source.db",
    dest_uri="duckdb:///./warehouse.duckdb",
    source_table="main.some_table",
    dest_table="public.some_table",
)
print(info)
```

See the [Python API documentation] for dry runs, error handling, and passing
enum options as strings.

## Handbook

Please visit the [quickstart documentation], or inspect the
list of supported [sources and destinations].

## Project

### Contribute

Contributions are very much welcome. Please visit the [sandbox documentation]
to learn how to spin up a development environment on your workstation and submit
patches, or create a [ticket][Issues] to report a bug or propose a feature.

### Status

Breaking changes should be expected until a 1.0 release, so version pinning is
strongly recommended, especially when using this software as a library.
For example:
```shell
pip install 'omniload[full]==0.0.42'
```

### License

The project is licensed under the MIT License, see the [LICENSE] file for details.
Some components are licensed under the Apache 2.0 license, see the [NOTICE] file for details.

### Acknowledgements

This project would not have been possible without the amazing work by the
authors and contributors to [SQLAlchemy], [dlt], and [ingestr], turtles all
the way down. Kudos.


[dlt]: https://github.com/dlt-hub/dlt
[quickstart documentation]: https://omniload.readthedocs.io/getting-started/quickstart.html
[Python API documentation]: https://omniload.readthedocs.io/getting-started/python-api.html
[ingestr]: https://bruin-data.github.io/ingestr/
[Issues]: https://github.com/panodata/omniload/issues
[LICENSE]: https://github.com/panodata/omniload/blob/main/LICENSE
[NOTICE]: https://github.com/panodata/omniload/blob/main/NOTICE
[sandbox documentation]: https://omniload.readthedocs.io/sandbox.html
[sources and destinations]: https://omniload.readthedocs.io/supported-sources/
[SQLAlchemy]: https://www.sqlalchemy.org/
[uv]: https://docs.astral.sh/uv/
