Metadata-Version: 2.1
Name: dataops-testgen
Version: 5.189.4
Summary: Open-source data quality tool that profiles tables, generates tests from the profiling results, and monitors freshness, volume, and schema.
Author-email: "DataKitchen, Inc." <info@datakitchen.io>
Maintainer-email: "DataKitchen, Inc." <info@datakitchen.io>
Project-URL: Source Code, https://github.com/DataKitchen/dataops-testgen
Project-URL: Bug Tracker, https://github.com/DataKitchen/dataops-testgen/issues
Project-URL: Documentation, https://docs.datakitchen.io/testgen/what-is-testgen/
Project-URL: Release Notes, https://docs.datakitchen.io/testgen/release-notes/
Project-URL: Slack, https://data-observability-slack.datakitchen.io/join
Project-URL: Homepage, https://datakitchen.io/products/dataops-testgen/
Keywords: data-quality,data-testing,data-profiling,data-observability,data-validation,dataops,mcp
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Development Status :: 5 - Production/Stable
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Monitoring
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: PyYAML ==6.0.3
Requires-Dist: click ==8.3.3
Requires-Dist: sqlalchemy ==2.0.48
Requires-Dist: databricks-sql-connector ==4.7.0
Requires-Dist: databricks-sqlalchemy ==2.0.10
Requires-Dist: databricks-sdk >=0.20.0
Requires-Dist: snowflake-sqlalchemy ==1.11.1
Requires-Dist: sqlalchemy-bigquery ==1.16.0
Requires-Dist: oracledb ==4.0.2
Requires-Dist: hdbcli ==2.25.31
Requires-Dist: sqlalchemy-hana ==4.4.0
Requires-Dist: salesforce-cdp-connector >=1.0.19
Requires-Dist: pyodbc ==5.3.0
Requires-Dist: psycopg2-binary ==2.9.11
Requires-Dist: duckdb ==1.5.5
Requires-Dist: duckdb-engine ==0.17.0
Requires-Dist: pycryptodome ==3.21
Requires-Dist: prettytable ==3.18.0
Requires-Dist: requests-extensions ==1.1.3
Requires-Dist: numpy ==2.1.3
Requires-Dist: pandas ==2.2.3
Requires-Dist: streamlit ==1.66.1
Requires-Dist: streamlit-extras ==0.3.0
Requires-Dist: streamlit-authenticator ==0.2.3
Requires-Dist: streamlit-javascript ==0.1.5
Requires-Dist: beautifulsoup4 ==4.12.3
Requires-Dist: xlsxwriter ==3.2.0
Requires-Dist: concurrent-log-handler ==0.9.25
Requires-Dist: cryptography ==50.0.0
Requires-Dist: reportlab ==4.2.2
Requires-Dist: chardet <6,>=5.2
Requires-Dist: cron-converter ==1.2.1
Requires-Dist: cron-descriptor ==2.0.5
Requires-Dist: pybars3 ==0.9.7
Requires-Dist: azure-identity ==1.25.1
Requires-Dist: statsmodels ==0.14.6
Requires-Dist: holidays ~=0.89
Requires-Dist: packaging >=24.0
Requires-Dist: jsonschema[format-nongpl] ==4.26.0
Requires-Dist: sqlglot ==30.17.0
Requires-Dist: pyarrow ==23.0.1
Requires-Dist: matplotlib ==3.9.2
Requires-Dist: scipy ==1.14.1
Requires-Dist: jinja2 ==3.1.6
Requires-Dist: pillow ==13.0.0
Requires-Dist: protobuf ==6.33.5
Requires-Dist: mcp[cli] ==1.30.0
Requires-Dist: uvicorn ==0.41.0
Requires-Dist: PyJWT ==2.15.1
Requires-Dist: bcrypt ==5.0.0
Requires-Dist: authlib ~=1.9.5
Requires-Dist: fastapi ==0.135.1
Requires-Dist: anyio >=4.0
Requires-Dist: pywin32 >=311 ; sys_platform == "win32"
Provides-Extra: dev
Requires-Dist: invoke ==2.2.0 ; extra == 'dev'
Requires-Dist: ruff ==0.4.1 ; extra == 'dev'
Requires-Dist: mypy ==1.9.0 ; extra == 'dev'
Requires-Dist: pre-commit ==3.6.2 ; extra == 'dev'
Requires-Dist: types-PyYAML ==6.0.12.20240311 ; extra == 'dev'
Requires-Dist: types-setuptools ==69.2.0.20240317 ; extra == 'dev'
Requires-Dist: types-pywin32 ==312.0.0.20261104 ; extra == 'dev'
Requires-Dist: semver ==3.0.2 ; extra == 'dev'
Requires-Dist: pytest ==8.1.1 ; extra == 'dev'
Requires-Dist: pytest-cov ==4.1.0 ; extra == 'dev'
Requires-Dist: tomli ==2.0.1 ; extra == 'dev'
Provides-Extra: release
Requires-Dist: build ==1.2.1 ; extra == 'release'
Requires-Dist: bumpver ==2023.1129 ; extra == 'release'
Requires-Dist: twine ==5.1.1 ; extra == 'release'
Provides-Extra: standalone
Requires-Dist: pixeltable-pgserver <0.7.0,>=0.6.0 ; extra == 'standalone'

# DataOps TestGen
![apache 2.0 license Badge](https://img.shields.io/badge/License%20-%20Apache%202.0%20-%20blue) ![PRs Badge](https://img.shields.io/badge/PRs%20-%20Welcome%20-%20green) [![Latest Version](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2Ftags%2F&query=results%5B0%5D.name&label=latest%20version&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Docker Pulls](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fhub.docker.com%2Fv2%2Frepositories%2Fdatakitchen%2Fdataops-testgen%2F&query=pull_count&style=flat&label=docker%20pulls&color=06A04A)](https://hub.docker.com/r/datakitchen/dataops-testgen) [![Documentation](https://img.shields.io/badge/docs-On%20datakitchen.io-06A04A?style=flat)](https://docs.datakitchen.io/testgen/what-is-testgen/) [![Static Badge](https://img.shields.io/badge/Slack-Join%20Discussion-blue?style=flat&logo=slack)](https://data-observability-slack.datakitchen.io/join)

DataOps TestGen is an open-source data quality tool that profiles tables, generates tests from the profiling results, and runs them on a schedule with freshness, volume, and schema monitoring.

The open-source edition includes the full web UI, not just a library. You also get a REST API and an MCP server, so you can drive TestGen from your pipelines or an AI assistant. It is Apache 2.0 licensed and runs the same core engine and UI as TestGen Enterprise.

[Interactive Product Tour](https://datakitchen.storylane.io/share/byag8vimd5tn) | [Product page](https://datakitchen.io/products/dataops-testgen/) | [Documentation](https://docs.datakitchen.io/testgen/what-is-testgen/)

## What it does

- Profiles every column in a table group and records type-specific statistics.
- Flags hygiene issues in new data, so you can review a dataset before it reaches production and send problems back to the supplier.
- Generates data quality validation tests from the profile. You can add your own SQL tests too.
- Runs tests on a schedule, as SQL inside your database. Your data stays where it is.
- Watches each table between test runs with freshness, volume, and schema monitors.
- Gives you one place to manage data quality across datasets, locations, and teams.

<p align="center">
<img alt="DataOps TestGen: hygiene review of new data" src="https://raw.githubusercontent.com/DataKitchen/dataops-testgen/main/docs/assets/images/readme/feature-new-data-hygiene.png" width="70%">
</p>
<p align="center">
<img alt="DataOps TestGen: continuous monitoring and quality testing" src="https://raw.githubusercontent.com/DataKitchen/dataops-testgen/main/docs/assets/images/readme/feature-continuous-monitoring.png" width="70%">
</p>
<p align="center">
<img alt="DataOps TestGen: one place to manage data quality" src="https://raw.githubusercontent.com/DataKitchen/dataops-testgen/main/docs/assets/images/readme/feature-centralized-quality.png" width="70%">
</p>

## Supported databases

As of TestGen 5.134.0, TestGen connects to 14 databases:

- Snowflake
- Databricks SQL
- DuckDB
- Azure Synapse Analytics
- Azure SQL Database
- Microsoft SQL Server
- Microsoft OneLake
- Google BigQuery
- Amazon Redshift
- Amazon Aurora PostgreSQL
- Oracle Database 12c and later
- SAP HANA
- PostgreSQL
- Salesforce Data 360

TestGen also reads CSV, Parquet, and JSON files on a local path, in Amazon S3, or in Azure Blob Storage, with no database in front. Every source works in both the open-source and Enterprise editions.

To set up a connection, see [Connect your data](https://docs.datakitchen.io/testgen/connect-your-data/).

## Quick start

The [dk-installer](https://github.com/DataKitchen/data-observability-installer/?tab=readme-ov-file#install-the-testgen-application) program installs TestGen in either Docker or pip mode.

You need 2 CPUs, 8 GB memory, 20 GB disk space. pip mode needs [Python](https://www.python.org/downloads/) 3.9+ to run the installer. Docker mode also needs [Docker](https://docs.docker.com/get-docker/) 27+ and [Docker Compose](https://docs.docker.com/compose/install/) 5.0+.

Download the installer into a new, empty directory:

```shell
curl -o dk-installer.py 'https://raw.githubusercontent.com/DataKitchen/data-observability-installer/main/dk-installer.py'
```

On Windows, you can instead download [`dk-installer.exe`](https://github.com/DataKitchen/data-observability-installer/releases/download/latest/dk-installer.exe) and double-click it.

Install TestGen:

```shell
python3 dk-installer.py tg install
```

The installer checks for Docker and asks you to pick Docker or pip. Pass `--docker` or `--pip` to skip the prompt. When it finishes, it writes credentials to `dk-tg-credentials.txt`, generates demo data, and opens the TestGen UI in your browser. Use `--no-demo` to skip the demo data.

Upgrade later with `python3 dk-installer.py tg upgrade`. Restart with `python3 dk-installer.py tg start`.

For complete instructions, see the documentation:

* [Install on Mac/Linux](https://docs.datakitchen.io/testgen/get-started/install-on-mac-linux/)
* [Install on Windows](https://docs.datakitchen.io/testgen/get-started/install-on-windows/)

Then walk through the [Data Observability Overview Demo](https://docs.datakitchen.io/tutorials/quickstart-demo/), and [connect your own database](https://docs.datakitchen.io/testgen/connect-your-data/).

## Use it from an AI assistant (MCP)

TestGen ships a Model Context Protocol (MCP) server in the open-source edition. Connect Claude, Claude Code, Cursor, GitHub Copilot, or any MCP client, and ask about your data in plain English. The assistant can profile tables, generate and run tests, and triage failures. Every tool runs under your own TestGen role, so the assistant can do only what you can do.

The installer serves the API and MCP server on port 8530 by default. Change it with `--api-port`.

Setup instructions: [docs.datakitchen.io/testgen/mcp](https://docs.datakitchen.io/testgen/mcp/)

## Open source vs Enterprise

The core profiling, testing, and monitoring engine is free and open source, with unlimited tables and unlimited data volume. Enterprise adds team access, security, Commercial Pharma tests, source-data controls for AI, and support.

| | Open source | Enterprise |
|---|---|---|
| License | Apache 2.0 | Commercial |
| Users | 1 | Unlimited, $100 per user per month |
| Database connections | 1 | Unlimited, $100 per connection per month |
| Projects | 1 | Unlimited |
| Enterprise security | | Included |
| Commercial Pharma test types (12) | | Included |
| Hide source data from AI clients (MCP) | | Included |
| Support | Community Slack | DataKitchen support |

See [datakitchen.io/pricing](https://datakitchen.io/pricing/) for current pricing.

## Documentation

* [DataOps TestGen documentation](https://docs.datakitchen.io/testgen/what-is-testgen/)
* [DataOps TestGen product page](https://datakitchen.io/products/dataops-testgen/)
* [Configuration (environment variables)](docs/configuration.md)
* [Local development](docs/local_development.md)

## Community

Talk with other data practitioners who build with DataKitchen. Share what you know, get help, and contribute.

* [Join us on Slack](https://data-observability-slack.datakitchen.io/join). This is also how you get support: post in the `#support` channel.
* [Star us on GitHub](https://github.com/DataKitchen/dataops-testgen)
* [Follow us on X](https://x.com/datakitchen_io)
* [Follow us on LinkedIn](https://www.linkedin.com/company/datakitchen)
* [Get free DataOps certification](https://datakitchen.io/certifications/)
* [Read our blog](https://datakitchen.io/blog/)
* [Sign The DataOps Manifesto](https://DataOpsManifesto.org)
* [Sign The Data Journey Manifesto](https://DataJourneyManifesto.org)

## Contributing

For details on contributing or running the project for development, see the [contributing guide](CONTRIBUTING.md).

## License

DataOps TestGen is Apache 2.0 licensed. See [LICENSE](LICENSE).
