Metadata-Version: 2.5
Name: mindzeval
Version: 0.1.0
Summary: Evaluation framework for SQL agents: result-based accuracy, behaviour scoring, multi-turn and golden-answer validation.
Project-URL: Homepage, https://github.com/MindzKonnected/mindzeval
Project-URL: Documentation, https://github.com/MindzKonnected/mindzeval#readme
Project-URL: Repository, https://github.com/MindzKonnected/mindzeval
Project-URL: Issues, https://github.com/MindzKonnected/mindzeval/issues
Project-URL: Changelog, https://github.com/MindzKonnected/mindzeval/blob/main/CHANGELOG.md
Author: MindzKonnected
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: agents,evaluation,llm,sql,testing,text-to-sql
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: httpx<1,>=0.27
Requires-Dist: pydantic<3,>=2.7
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: sqlalchemy<3,>=2.0
Requires-Dist: sqlglot<31,>=30
Provides-Extra: dev
Requires-Dist: mypy==2.4.0; extra == 'dev'
Requires-Dist: pytest==9.1.1; extra == 'dev'
Requires-Dist: ruff==0.17.1; extra == 'dev'
Requires-Dist: types-pyyaml==6.0.12.20260906; extra == 'dev'
Provides-Extra: postgres
Requires-Dist: psycopg2-binary<3,>=2.9; extra == 'postgres'
Requires-Dist: psycopg[binary]<4,>=3.1; extra == 'postgres'
Description-Content-Type: text/markdown

# MindzEval

MindzEval proves your SQL agent is accurate and safe before it reaches users, and keeps proving it on every change.

> **Status: release `0.1.0`.** New here? Start with [getting started](docs/getting-started.md). Feedback and bug reports are welcome in the [issues](https://github.com/MindzKonnected/mindzeval/issues).

## What it does

- **Grades the data, not the SQL.** The agent's results are compared with the results of a reference query, so correct SQL written a different way still passes.
- **Checks the answer key first.** Reference queries are validated against the live database (empty, drift, unstable, NULL groups, orphan-inflated) before any agent is graded.
- **Scores behaviour both ways.** Asking for clarification or declining is rewarded when it's right and penalised when it isn't.
- **Separates display limits from logic errors.** The agent's SQL is re-run without its display `LIMIT`, so a UI row cap isn't mistaken for a wrong query.
- **Checks safety.** Write attempts, forbidden tables and PII in results or answers.
- **Works with any agent.** LangChain, LangGraph, CrewAI, LlamaIndex, the OpenAI Agents SDK, plain Python, or any HTTP service.
- **Compares models and prompts.** Matrix runs evaluate every combination side by side.
- **Read-only by design.** Every query MindzEval runs is parsed and refused unless it is a single read, then executed in a read-only transaction with a timeout.

## Installation

MindzEval requires Python 3.11 or newer:

```bash
# SQLite only
pip install mindzeval

# With PostgreSQL support
pip install "mindzeval[postgres]"
```

## Quickstart

```bash
mindzeval quickstart demo
cd demo
mindzeval run
```

The example agent fails some cases on purpose so you can see what each verdict means.

## Your own agent

`mindzeval.yaml`:

```yaml
project: shop-agent
agent:
  type: python
  target: my_project.eval_adapter:answer     # or type: http with a URL
database:
  url: postgresql://reader:${DB_PASSWORD}@db.internal/shop
datasets: cases/*.yaml
```

`cases/customers.yaml`:

```yaml
- id: customers_in_germany
  question: Which customers are in Germany?
  reference_sql: SELECT id FROM customers WHERE country = 'DE'
  key: id

- id: vague_request
  question: Show me the data
  expect: {behavior: clarify}
```

```bash
mindzeval lint       # check config and cases (no connections, no secrets)
mindzeval validate   # check reference queries against the database
mindzeval run        # evaluate; exits 1 if thresholds aren't met
```

## Documentation

- [Getting started](docs/getting-started.md): install, try the example, connect your agent
- [Integration examples](https://github.com/MindzKonnected/mindzeval-integration-examples): runnable examples per framework, starting with LangGraph
- [Connecting your agent](docs/guides/connecting-agents.md): Python, LangGraph, LangChain, CrewAI, HTTP, saved outputs, model comparison
- [Case format](docs/reference/case-format.md)
- [Config file](docs/reference/config.md)
- [Agent result record](docs/reference/agent-response.md)
- [Reading results](docs/reference/results.md): verdicts, numbers, failure causes, exit codes
- [Custom scorers](docs/guides/custom-scorers.md)

## Development

```bash
git clone https://github.com/MindzKonnected/mindzeval.git && cd mindzeval
python -m venv .venv && .venv\Scripts\activate      # macOS/Linux: source .venv/bin/activate
pip install -e ".[dev,postgres]"
pytest              # set MINDZEVAL_TEST_POSTGRES_URL to include Postgres
ruff check src tests && ruff format --check src tests && mypy
```

Work happens on `development` through pull requests. `main` is the default branch and `production` holds released versions.

## License

MindzEval is licensed under the [Apache License 2.0](LICENSE).
