Metadata-Version: 2.4
Name: evaldata
Version: 0.3.0
Summary: AI evals framework for data & analytics engineering teams.
Keywords: ai,evals,evaluation,sql,data,testing,snowflake,bigquery,databricks,duckdb,postgres,pytest,llm,text-to-sql
Author: monospaceai
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Database
Classifier: Typing :: Typed
Requires-Dist: duckdb>=1.5.3
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pytest>=9.0.3
Requires-Dist: rich>=15.0.0
Requires-Dist: sqlglot>=30.9.0
Requires-Dist: typer>=0.25.1
Requires-Dist: evaldata[all-platforms,litellm,tracing] ; extra == 'all'
Requires-Dist: evaldata[postgres,snowflake,bigquery,databricks] ; extra == 'all-platforms'
Requires-Dist: google-cloud-bigquery>=3.41.0 ; extra == 'bigquery'
Requires-Dist: google-cloud-bigquery-storage>=2.38.0 ; extra == 'bigquery'
Requires-Dist: databricks-sql-connector>=4.2.6 ; extra == 'databricks'
Requires-Dist: databricks-sdk>=0.30.0 ; extra == 'databricks'
Requires-Dist: litellm>=1.85.1 ; extra == 'litellm'
Requires-Dist: psycopg[binary]>=3.3.4 ; extra == 'postgres'
Requires-Dist: snowflake-connector-python>=4.5.0 ; extra == 'snowflake'
Requires-Dist: openinference-instrumentation>=0.1.52 ; extra == 'tracing'
Requires-Dist: opentelemetry-exporter-otlp>=1.42.1 ; extra == 'tracing'
Requires-Dist: opentelemetry-sdk>=1.42.1 ; extra == 'tracing'
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/monospaceai/evaldata
Project-URL: Issues, https://github.com/monospaceai/evaldata/issues
Project-URL: Repository, https://github.com/monospaceai/evaldata
Provides-Extra: all
Provides-Extra: all-platforms
Provides-Extra: bigquery
Provides-Extra: databricks
Provides-Extra: litellm
Provides-Extra: postgres
Provides-Extra: snowflake
Provides-Extra: tracing
Description-Content-Type: text/markdown

# evaldata

[![CI](https://github.com/monospaceai/evaldata/actions/workflows/ci.yml/badge.svg)](https://github.com/monospaceai/evaldata/actions/workflows/ci.yml)
[![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen.svg)](https://github.com/monospaceai/evaldata/actions/workflows/ci.yml)
[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)

**The evaluation framework for AI-generated SQL.**

Evaluate AI-generated SQL in your warehouse and catch regressions when prompts, models, or code change.

Pytest-native. CI-friendly. Built for data teams.

## Install

```bash
uv add evaldata                # core (includes the DuckDB adapter)
uv add "evaldata[postgres]"    # + Postgres adapter
uv add "evaldata[databricks]"  # + Databricks adapter
uv add "evaldata[litellm]"     # + litellm, to call a model as the AI under test
```

DuckDB, Postgres, and Databricks are the adapters available today. Snowflake and
BigQuery are planned.

## Examples

Runnable examples in [`examples/`](examples/):

| Example | Shows |
| --- | --- |
| [Deterministic](examples/01_deterministic/test_golden_questions.py) | Every expected-type and scorer with fixed SQL — no model or network |
| [Local AI](examples/02_local_ai/test_text_to_sql.py) | A self-hosted Ollama model as the AI under test, via litellm |
| [Hosted AI](examples/03_hosted_ai/test_text_to_sql.py) | Hosted-model plumbing, mocked so it runs without a live call or key |

See [`examples/README.md`](examples/README.md) for details.

## Develop locally

```bash
git clone https://github.com/monospaceai/evaldata.git
cd evaldata
uv sync                       # core + dev tooling
uv run pre-commit install
just check                    # lint + typecheck + tests with coverage (runs everything)
```

`just check` runs lint, typecheck, and tests with coverage (held at 100%). See the
`justfile` for the full set of commands.

### Platform e2e tests

Adapter conformance for real platforms is marked `e2e`. CI provisions Postgres as a
service container and runs the suite on every push, so the Postgres adapter is exercised
against a real engine on every change.

Run it locally against Postgres with:

```bash
docker compose up -d                  # postgres:17 on localhost:5432
uv run --extra postgres pytest -m e2e # connection via POSTGRES_TEST_* env (defaults match compose)
```
