Metadata-Version: 2.4
Name: evaldata
Version: 0.4.0
Summary: AI evals framework for data & analytics engineering teams.
Keywords: ai,evals,evaluation,sql,data,testing,snowflake,bigquery,databricks,duckdb,postgres,pytest,llm,text-to-sql
Author: monospaceai
License-Expression: Apache-2.0
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Database
Classifier: Typing :: Typed
Requires-Dist: duckdb>=1.5.3
Requires-Dist: pyarrow>=24.0.0
Requires-Dist: pydantic>=2.13.4
Requires-Dist: pytest>=9.0.3
Requires-Dist: rich>=15.0.0
Requires-Dist: sqlglot>=30.9.0
Requires-Dist: typer>=0.25.1
Requires-Dist: evaldata[all-platforms,litellm,tracing,benchmarks,dbt,dbt-sl] ; extra == 'all'
Requires-Dist: evaldata[postgres,snowflake,bigquery,databricks] ; extra == 'all-platforms'
Requires-Dist: platformdirs>=4.0 ; extra == 'benchmarks'
Requires-Dist: google-cloud-bigquery>=3.41.0 ; extra == 'bigquery'
Requires-Dist: google-cloud-bigquery-storage>=2.38.0 ; extra == 'bigquery'
Requires-Dist: databricks-sql-connector>=4.2.6 ; extra == 'databricks'
Requires-Dist: databricks-sdk>=0.30.0 ; extra == 'databricks'
Requires-Dist: pyyaml>=6 ; extra == 'dbt'
Requires-Dist: dbt-metricflow>=0.13 ; extra == 'dbt-sl'
Requires-Dist: litellm>=1.85.1 ; extra == 'litellm'
Requires-Dist: psycopg[binary]>=3.3.4 ; extra == 'postgres'
Requires-Dist: snowflake-connector-python>=4.5.0 ; extra == 'snowflake'
Requires-Dist: openinference-instrumentation>=0.1.52 ; extra == 'tracing'
Requires-Dist: opentelemetry-exporter-otlp>=1.42.1 ; extra == 'tracing'
Requires-Dist: opentelemetry-sdk>=1.42.1 ; extra == 'tracing'
Requires-Python: >=3.11
Project-URL: Homepage, https://github.com/monospaceai/evaldata
Project-URL: Issues, https://github.com/monospaceai/evaldata/issues
Project-URL: Repository, https://github.com/monospaceai/evaldata
Provides-Extra: all
Provides-Extra: all-platforms
Provides-Extra: benchmarks
Provides-Extra: bigquery
Provides-Extra: databricks
Provides-Extra: dbt
Provides-Extra: dbt-sl
Provides-Extra: litellm
Provides-Extra: postgres
Provides-Extra: snowflake
Provides-Extra: tracing
Description-Content-Type: text/markdown

# evaldata

[![CI](https://github.com/monospaceai/evaldata/actions/workflows/ci.yml/badge.svg)](https://github.com/monospaceai/evaldata/actions/workflows/ci.yml)
[![Coverage](https://img.shields.io/badge/coverage-100%25-brightgreen.svg)](https://github.com/monospaceai/evaldata/actions/workflows/ci.yml)
[![License: Apache 2.0](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)

**The evaluation framework for AI-generated SQL.**
`pytest`-native. CI-friendly. Built for data teams.

`evaldata` catches regressions on every prompt and model change, before they reach production.

## Why evaldata

`evaldata` can prove two queries are equivalent without executing them or asking an LLM
to judge.

MLflow, Ragas, and DeepEval reach for an LLM even when the answer is exact and provable
— a slow, costly guess at something you can settle for free.

- **Semantic equivalence.** Confirm two queries have the same meaning by comparing their
  structure. No execution, no guessing — when it can't confirm, it returns `unknown`.
- **Execution in your warehouse.** Run the query on DuckDB, Postgres, or Databricks and
  compare the results, accounting for row order, NULLs, float tolerance, and types.
- **It's just `pytest`.** Every eval is a test, run in your suite and your CI on every PR.
  No new runner, notebook, or dashboard.
- **An LLM judge when you need one.** For ambiguous questions, missing reference answers,
  or an explanation to grade: the right tool for the job, fully supported.

evaldata reproduces dbt's own Semantic Layer benchmark locally on DuckDB — same dataset, questions,
and model — scoring 96.4% with `gpt-5.3-codex`, as pytest and with no dbt Cloud. See
[Reproduce dbt's Semantic Layer benchmark](docs/guides/dbt-semantic-layer-benchmark.md).

## Quickstart

```bash
uv add evaldata   # core, includes the DuckDB adapter
```

An eval is a `pytest` test: a **case** (a question and its expected answer), a **solver**
(the system under test that writes the SQL), and a **scorer** (how the answer is judged).

Below, the AI's SQL is written differently from the reference query — reordered predicates,
different casing — but means the same thing. `observed_equivalence()` proves the match from
the query structure alone; no query runs.

```python
from evaldata import CallableSolver, EvalCase, assert_eval, eval_case, observed_equivalence
from evaldata.platforms import duckdb_platform

platform = duckdb_platform(name="shop", path="shop.duckdb")


@eval_case(
    input="Name the US customers with an id above 1.",
    expected={"kind": "gold_query", "sql": "SELECT name FROM customers WHERE country = 'US' AND id > 1"},
    platform=platform,
)
def test_us_customers(case: EvalCase) -> None:
    solver = CallableSolver(lambda c: "select NAME from customers where id > 1 and country = 'US'")
    assert_eval(case, solver, scorers=[observed_equivalence()])
```

```bash
uv run pytest
```

```
 case               result   detail
 ──────────────────────────────────
 test_us_customers  PASS

 1 passed, 0 failed
```

The full runnable version is in
[`examples/01_deterministic/test_showcase.py`](examples/01_deterministic/test_showcase.py).

To test a real model instead of fixed SQL, swap the solver for
`PromptSolver(model="openai/gpt-4o-mini")` (needs the `evaldata[litellm]` extra). To judge
equivalence without a warehouse, swap the scorer for `judged_equivalence(model)`.

## Install

```bash
uv add evaldata                # core (includes the DuckDB adapter)
uv add "evaldata[postgres]"    # + Postgres adapter
uv add "evaldata[databricks]"  # + Databricks adapter
uv add "evaldata[litellm]"     # + litellm, to call a model as the AI under test
```

DuckDB, Postgres, and Databricks are the adapters available today. Snowflake and
BigQuery are planned.

## Documentation

Full documentation: **[monospaceai.github.io/evaldata](https://monospaceai.github.io/evaldata/)**

- [Getting started](https://monospaceai.github.io/evaldata/getting-started/) — write and run your first eval.
- [Concepts](https://monospaceai.github.io/evaldata/concepts/) — cases, solvers, scorers, and platforms.
- Guides — [semantic equivalence](https://monospaceai.github.io/evaldata/guides/semantic-equivalence/), [LLM judge](https://monospaceai.github.io/evaldata/guides/llm-judge/), [local Ollama](https://monospaceai.github.io/evaldata/guides/local-ollama/), [hosted model](https://monospaceai.github.io/evaldata/guides/hosted-model/), [Databricks](https://monospaceai.github.io/evaldata/guides/databricks/).
- [API reference](https://monospaceai.github.io/evaldata/reference/) — the public API, generated from docstrings.

## Examples

Runnable examples in [`examples/`](examples/):

| Example | Shows |
| --- | --- |
| [Showcase](examples/01_deterministic/test_showcase.py) | Semantic equivalence with an execution fallback — no setup |
| [Deterministic](examples/01_deterministic/test_golden_questions.py) | Every expected-type and scorer, with fixed SQL |
| [Local AI](examples/02_local_ai/test_text_to_sql.py) | A self-hosted Ollama model as the AI under test |
| [Hosted AI](examples/03_hosted_ai/test_text_to_sql.py) | A hosted model, mocked so it runs without a key |
| [Databricks](examples/04_databricks/test_deterministic.py) | The same cases on a live Databricks SQL Warehouse |
| [LLM judge](examples/05_llm_judge/test_judged_equivalence.py) | Judged equivalence, mocked so it runs without a key |
| [Benchmark](examples/06_benchmark/test_benchmark.py) | Load a Spider/BIRD dataset and measure execution accuracy |

See [`examples/README.md`](examples/README.md) for details.

## Contributing

```bash
git clone https://github.com/monospaceai/evaldata.git
cd evaldata
uv sync                       # core + dev tooling
uv run pre-commit install
just check                    # lint + typecheck + tests with coverage (runs everything)
```

`just check` runs lint, typecheck, and tests with coverage (held at 100%). See the
`justfile` for the full set of commands.

### Platform e2e tests

Adapter conformance for real platforms is marked `e2e`. CI provisions Postgres as a
service container and runs the suite on every push, so the Postgres adapter is exercised
against a real engine on every change.

Run it locally against Postgres with:

```bash
docker compose up -d                  # postgres:17 on localhost:5432
uv run --extra postgres pytest -m e2e # connection via POSTGRES_TEST_* env (defaults match compose)
```
