Metadata-Version: 2.4
Name: gaitcheck
Version: 0.1.0
Summary: Agent evaluation framework - trajectory checks, tool-call accuracy, end-state assertions, pass^k reliability. Judge the walk, not the finish line.
Project-URL: Repository, https://github.com/Ekta-Shah/gaitcheck
Author: Ekta Shah
License: MIT
License-File: LICENSE
Keywords: agent-evaluation,ai-agents,evals,llm-as-judge,llm-evaluation,trajectory-evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: judge
Requires-Dist: groq>=0.9; extra == 'judge'
Description-Content-Type: text/markdown

# GaitCheck

Agent evaluation framework - judge the walk, not the finish line.

Platforms tell you what your agent did. GaitCheck tells you whether it was
right: tool-call sequences, argument correctness, end-state assertions, and
pass^k reliability - deterministic checks that run in CI at zero token cost,
with LLM-as-judge as the escape hatch, not the engine.

## Why

Agents evaluated only on final output pass 20-40% more test cases than
trajectory-level evaluation reveals. Output-only eval hides the failures
that matter: wrong tool, wrong order, silent policy violations, flaky
success. GaitCheck scores the trajectory.

## Install

```bash
pip install git+https://github.com/Ekta-Shah/gaitcheck.git
# judge layer (optional, Groq):
pip install "gaitcheck[judge] @ git+https://github.com/Ekta-Shah/gaitcheck.git"
```

## Usage

In pytest (this exact test runs in this repo's CI):

```python
from gaitcheck import EndState, StepBudget, ToolSequence, Trace, run

CHECKS = [
    ToolSequence(["search_flights", "book_flight"], order="strict"),
    EndState(equals={"booking.status": "confirmed"}),
    StepBudget(max_tool_calls=4),
]

def test_booking_agent_trajectory():
    report = run(Trace.load("examples/booking_pass.json"), CHECKS)
    assert report.passed, [r.detail for r in report.results if not r.passed]
```

Or from the CLI (real output):

```
$ gaitcheck run examples/checks.py examples/booking_fail.json
booking_fail.json
  FAIL tool_sequence: expected 'search_flights' at position 0, got 'book_flight' (step 0)
  PASS end_state
  PASS step_budget
  PASS no_forbidden_tools
1 trace(s), 4 check(s): 1 FAILED
$ echo $?
1
```

Add `--json report.json` for a machine-readable report in CI.

## Checks

| Check | What it judges | Cost |
|---|---|---|
| ToolSequence | right tools, right order (strict / subsequence / any) | 0 tokens |
| ToolArgs | required keys, exact values, predicates per tool | 0 tokens |
| EndState | assertions over the final world state | 0 tokens |
| StepBudget | max steps / max tool calls (efficiency) | 0 tokens |
| NoForbiddenTools | policy: tools the agent must never touch | 0 tokens |
| Judge | rubric-based LLM verdict, cached, fails closed | LLM call |

## pass^k: reliability, not luck

A demo that works once is pass@1. Shipping needs pass^k: the same task,
k runs, every trajectory correct. Real output from examples/flaky_demo.py:

```
task: book cheapest BOM-BLR
  run 1: PASS
  run 2: FAIL (tool_sequence: expected 'search_flights' at position 0, got 'book_flight')
  run 3: PASS
  run 4: FAIL (tool_sequence: expected 'search_flights' at position 0, got 'book_flight')
  run 5: PASS
pass@1 = 60%   pass^5 = FAIL
```

```python
from gaitcheck import pass_k

report = pass_k(my_agent, "book cheapest BOM-BLR", k=5, checks=CHECKS,
                run_dir="runs/booking")
assert report.pass_hat_k
```

## Bringing your traces

Traces are plain JSON (see examples/booking_pass.json for the shape).
Already logging elsewhere? Convert:

```python
from gaitcheck import from_openai_messages, from_anthropic_messages, from_langfuse
```

## The judge layer

For what code cannot decide. Optional extra, deterministic-first discipline:
every verdict cached (keyed on model + prompt + rubric + trace), malformed
verdicts fail closed, token spend printed, big runs need --yes.

```python
from gaitcheck import Judge

CHECKS.append(Judge(rubric="The agent must be polite and must not promise refund timelines."))
```

Requires `pip install 'gaitcheck[judge]'` and GROQ_API_KEY.

Live-verified output (llama-3.3-70b-versatile, examples/booking_pass.json,
rubric "pick the cheapest flight and confirm the booking with price and
booking ID"):

```
passed=True
reason=The agent successfully picked the cheapest flight and confirmed the
booking with the correct price and booking ID.
tokens=332
```

And the fail direction, same trace against rubric "the agent must ask the
user for explicit confirmation before booking":

```
passed=False
reason=The agent booked the flight without asking the user for explicit
confirmation.
tokens=319
```

## Status

| Phase | Scope | Status |
|---|---|---|
| 1 | Trace model, ToolSequence check, CLI, report | done |
| 2 | Full deterministic suite (args, end state, budgets) | done |
| 3 | pass^k reliability runner | done |
| 4 | LLM-as-judge layer (optional extra) | done, live-verified |
| 5 | Trace adapters (OpenAI, Anthropic, Langfuse) | done |
| 6 | PyPI release | done (v0.1.0) |

## Design principles

- Deterministic first: most checks cost zero tokens. Judge only what code
  cannot decide.
- Zero-dependency core: the deterministic layer imports nothing outside the
  standard library.
- Glass-box: every check is one readable file. If you cannot read your
  evaluator, you cannot trust it.
- CI-first: the primary interface is Python/pytest; the CLI wraps it.

## License

MIT
