Metadata-Version: 2.4
Name: eval-mock
Version: 0.0.1
Summary: A flight simulator for AI agents - deterministic, stateful simulated worlds for agent evaluation.
Project-URL: Homepage, https://github.com/qianmoai/eval-mock
Author-email: QIANMO AI <qianmoai@hotmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: ai-agents,evaluation,mcp,simulation,testing
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: rfc8785>=0.1.4
Description-Content-Type: text/markdown

# Eval Mock

English | [简体中文](README.zh-CN.md)

> **A flight simulator for AI agents.** A deterministic, stateful simulated business world where your agent is tested — before it ever touches a real customer, a real order, or a real inbox.

**Status**: design complete, implementation starting — no release yet (pre-0.1.0).

## The 60-second version

You own an AI agent — say, customer support. Tomorrow you ship a new version (new model, new prompt). Your boss asks: **"Will it mishandle refunds?"**

Today you have three bad options:

- **Test against real systems** — you can't. A test refund is a real refund; a test email lands in a real customer's inbox.
- **Build a staging environment** — expensive, slow, and still incomplete: the agent needs orders *and* email *and* CRM behaving as one coherent world.
- **Click through by hand and pray** — and a pass today proves nothing, because you can't tell *agent flakiness* from *environment flakiness*.

Eval Mock gives your agent an exam hall instead:

- **Fake, but alive.** A simulated inbox, order system, and CRM that actually behave: when the agent sends an email, the simulated customer receives it; when it cancels an order, the order state changes. Nothing real is touched. No API bills.
- **Game saves.** Every test starts from an identical save. Run the same scenario 100 times — the world is exactly the same every time, so failures measure *your agent's* variance (`pass^k`), not infrastructure noise.
- **Auto-grading.** Task completed? Touched data it shouldn't have? Retried after the payment API "timed out"? Graded from the world's final state and the full call journal — no human watching replays, no second LLM as judge.
- **Disaster drills.** Declaratively inject timeouts, rate limits, and permission errors, and see whether the agent recovers — before your customers run that experiment for you.

What you get before every release is a scorecard:

> 50 scenarios × 10 runs each: v1 passes 92%, v2 passes 88% — the regression is entirely "forgets to retry after a timeout", first diverging at step 3.

Ship or hold, you now decide with numbers.

## Why a third path

Evaluating an agent requires a world it can act in. The existing paths all fall short:

- **Real systems**: side effects, cost, and no reproducibility.
- **Record & replay**: only replays trajectories that already happened; breaks the moment the agent takes a different (equally valid) path.
- **Traditional API mocks**: stateless canned responses — an email "sent" through the Email tool can never be found by the CRM tool; the same query returns the same answer before and after a cancellation. There is no world, only isolated answers.

Eval Mock is the third path: **the world comes first; tools are projections of the world.** The agent acts through standard protocols (MCP, plus a native function-calling adapter); every step is journaled, assertable, and exactly replayable.

## How it works

1. **World Store** — one entity store shared by all tools. `seed / snapshot / restore / diff` are first-class: the same seed rebuilds the same world, any number of times, verified by canonical digests across machines and OSes.
2. **Scenario** — one declarative YAML file = one exam question: the task given to the agent, the initial world, enabled tools, injected faults, scripted multi-turn user follow-ups, and assertions. Versioned together; distributable; a colleague reruns your exact evaluation with one command.
3. **Trajectory & Verdict** — a complete tool-call journal plus a machine-readable verdict: final-state assertions, collateral-damage checks (nothing changed outside an allowlist), earliest-failure attribution, aggregated pass rates across repeated runs. OTel span export optional.

**Design rule**: LLMs may help *author* environments; at runtime the environment is **zero-LLM** — pure deterministic code, so results are reproducible by construction. Scripted multi-turn is deterministic too: no LLM plays the user.

An illustrative scenario (DSL may evolve until 0.1.0):

```yaml
task:
  prompt: "The customer wants a refund for order #1042."
world:
  shop:
    orders:
      - { id: 1042, status: delivered, days_since_delivery: 3 }
tools: [shop, email]
faults:
  - { tool: shop.refund, error: timeout, after_calls: 1 }   # first attempt times out
turns:
  - user: "Actually, store credit is fine too."
assertions:
  - id: refund-completed
    expr: world.shop.orders.exists(o, o.id == 1042 && o.status == "refunded")
  - id: no-collateral
    collateral: { allow_changes: [shop.orders, shop.refunds, email.sent] }
```

## What Eval Mock is not

- **Not a model mock.** We simulate the tools and the world around your agent; the model under test is your real model.
- **Not a code-execution sandbox.** SWE-bench-style "run the repo's tests in a container" is a different, well-served problem.
- **Not an LLM user simulator.** Multi-turn users are scripted and deterministic — by design.
- **Not a scoring platform.** Verdicts are plain files that feed pytest, CI gates, DeepEval, LangSmith, or your own judge.

## Roadmap (v1 scope)

- [ ] Determinism kernel: seeded world, virtual clock, canonical digests, snapshot/restore
- [ ] Core loop: scenario → MCP server + function-calling adapter → journal → verdict
- [ ] Domain packs: Email / Calendar / CRM — plus **declarative BYO packs** (define your own domain in YAML, no code)
- [ ] Fault injection, scripted multi-turn, repeat/`pass^k` CI gating, third-party `verify` replay
- [ ] 0.1.0 on PyPI

## Promises

- **Zero telemetry.** Runs fully offline — no accounts, no API keys, no network required.
- **The open-source version is the full product.** No feature-gated enterprise edition.

## License

TBD (Apache 2.0 planned).
