Metadata-Version: 2.4
Name: isitright
Version: 0.1.0.dev5
Summary: Evidence-first stress testing for text-to-SQL evaluations
Project-URL: Repository, https://github.com/isitright-ai/isitright
Author: isitright contributors
License: MIT License
        
        Copyright (c) 2026 isitright contributors
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
License-File: LICENSE
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: jsonschema<5,>=4.23
Requires-Dist: pydantic<3,>=2.10
Requires-Dist: pyyaml<7,>=6.0.2
Requires-Dist: sqlglot<29,>=27
Requires-Dist: typer<1,>=0.15
Provides-Extra: dev
Requires-Dist: build<2,>=1.2.2; extra == 'dev'
Requires-Dist: hypothesis<7,>=6.115; extra == 'dev'
Requires-Dist: mypy<2,>=1.14; extra == 'dev'
Requires-Dist: pytest-cov<8,>=6; extra == 'dev'
Requires-Dist: pytest<9,>=8.3; extra == 'dev'
Requires-Dist: ruff<1,>=0.9; extra == 'dev'
Requires-Dist: twine<7,>=6.1; extra == 'dev'
Description-Content-Type: text/markdown

# isitright

`isitright` is an evidence-first tool for stress-testing text-to-SQL evaluations.
Given reference and candidate SQL, the v0.1 engine will search for a valid
database state on which their outputs differ and emit a minimized, replayable
counterexample. A disagreement is evidence, not automatic attribution of fault.

## Current status: Stage 4 review gate

The v0.1 candidate safely compares queries on the supplied SQLite database and runs
a deterministic, schema-aware generated-state search. Search harvests referenced
tables, columns, and literals and tries boundary, NULL, duplicate, join, filtering,
and aggregation mutations in priority order.

Every generated state is checked against SQLite integrity and foreign-key rules,
an optional JSON Schema constraints document, and an optional trusted validator.
Finding no divergence is reported as `AGREEMENT_ON_TESTED_STATES`; it is not a
claim of equivalence. Findings are minimized across generated and background rows,
revalidated, and exported as deterministic `counterexample.sql` files. Suite mode,
ranked reports, and opt-in `--fail-on` policies are active.

## Quick start

```python
from isitright import check

result = check(
    database="example.sqlite",
    reference_sql="SELECT province FROM tribute ORDER BY province",
    candidate_sql="SELECT DISTINCT province FROM tribute ORDER BY province",
)
print(result.headline)
print(result.original_state.outputs_match)
print(result.search.model_dump())
```

The CLI reads each query from a UTF-8 file and writes versioned JSON plus a concise
Markdown rendering by default:

```console
isitright check --database example.sqlite --reference gold.sql \
  --candidate agent.sql --output-dir artifacts/check
```

When a counterexample is found, replay the emitted fixture and verify both recorded
outputs without touching the source database:

```console
cd artifacts/check
python -m isitright.replay --report case_report.json --fixture counterexample.sql
```

## Development

Python 3.11 or newer is required. With `uv` installed:

```console
uv sync --extra dev
uv run ruff check .
uv run mypy src
uv run pytest
uv build
```

The package exposes single-case and JSONL suite commands:

```console
isitright --version
isitright check --help
isitright suite --help
```

```console
isitright suite --database example.sqlite --cases cases.jsonl \
  --output-dir artifacts/suite --fail-on domain-counterexample
```

Suite cases run independently and are ranked with domain-validated findings first,
then schema-only findings, inconclusive/unsupported cases, and observed agreement.
Both commands accept `--fail-on none`, `domain-counterexample`, or
`any-counterexample`; a matched opt-in policy exits with code `3` after artifacts
have been written.

The Stage 4 preview is versioned `0.1.0.dev5`. It is intentionally a development
release and must not be presented as the complete v0.1 search engine.

## Contract highlights

- Result statuses keep observed equivalence, state validity, and attribution separate.
- Unordered query results are compared as multisets, preserving duplicates.
- Automatic ordering honors a top-level `ORDER BY` on either query; otherwise results
  are unordered.
- Engine-native values are normalized into explicit, portable canonical kinds.
- JSON Schema and trusted callback validation are applied before generated states
  are admitted, and every rejection is counted by reason.
- Reports use schema version `0.1`; package versioning remains pre-release until the
  v0.1 engine is complete.

Generation is intentionally bounded and lightweight. It creates small
insertions by adapting existing rows or synthesizing complete rows for empty
referenced tables, including required foreign-key parent rows. It does not perform
AST-diff-guided search or prove that exhausted candidates cover every admissible
state. Minimization is also bounded; if its budget is exhausted, the smallest
verified state found so far is returned with `minimization_complete: false`.

See [the contract reference](docs/contracts.md) and
[the fixture proposal](tests/fixtures/proposals/README.md). The precise supported
SQL surface and known limitations are documented in
[Supported SQL and limitations](docs/supported-sql.md).

Contributions are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md). Security issues
should follow the private reporting process in [SECURITY.md](SECURITY.md).

## License

MIT

