How ai-evals works

A deep-dive into the codebase — from a first ai-evals init all the way to how every module connects.

What is ai-evals?

ai-evals is a zero-config, repo-aware, model-agnostic AI evaluation CLI. You point it at any Python project that uses OpenAI, LangChain, ChromaDB, or similar frameworks, and it:

1. Reads your code (AST, never executes it) to infer what AI tasks exist.
2. Scaffolds an eval/ directory with a rubrics file, a golden-set stub, and a pytest entry point.
3. Captures real I/O traces while your tests run (Phase 3).
4. Scores each trace through a configurable judge LLM and surfaces regressions (Phase 4).

The key insight: you do not write eval harnesses by hand. The tool reads your repo and generates them for you, then updates them as your code evolves.

Big picture — end-to-end flow

  Your Python repo
        │
        ▼  ast.parse every .py file
  ┌─────────────────────────────┐
  │  inference/                 │  ← Phase 1 ✅
  │  ast_scan  ▸  detectors     │
  │  signatures ▸ synthesize    │
  └────────────┬────────────────┘
               │  list[DetectedTask]  →  RubricsConfig
               ▼
  ┌─────────────────────────────┐
  │  scaffold/                  │  ← Phase 1 ✅
  │  rubrics_writer             │  →  eval/rubrics.yaml
  │  golden_writer              │  →  eval/golden_set.json  (stub)
  │  tests_writer               │  →  eval/tests.py
  └────────────┬────────────────┘
               │
               ▼  user runs  ai-evals bootstrap -- pytest
  ┌─────────────────────────────┐
  │  bootstrap/  (Phase 3 🔜)  │  →  eval/golden_set.json  (real captures)
  │  wrappers  ▸  tracer        │
  └────────────┬────────────────┘
               │
               ▼  user runs  ai-evals run
  ┌─────────────────────────────┐
  │  judge/     (Phase 2 🔜)   │  →  LiteLLM  →  Ollama / OpenAI / …
  │  runner/    (Phase 4 🔜)   │  →  metrics  →  .ai-evals/history.json
  └────────────┬────────────────┘
               │
               ▼
  ┌─────────────────────────────┐
  │  insights/  (Phase 4 🔜)   │  →  ai-evals diff  →  regression report
  └─────────────────────────────┘

4-phase roadmap

Repository layout

Files ai-evals creates in your repo

After a successful ai-evals init your project gets two new top-level directories:

CLI commands

All commands follow the shape ai-evals <command> [flags] — max one level of nesting.

Global flags

These flags apply to every command and are resolved in cli/app.py → _root() before the subcommand runs. They are stored on ctx.obj as a GlobalOptions dataclass.

CI auto-mode

When the environment variable CI=true is set, the root callback in app.py:158 automatically applies:

• --no-input is forced on
• --no-color is forced on
• --format auto resolves to json

Exit codes

Every error printed to stderr follows the pattern from plan §1.7:

error: <one-line summary>
  what: <what went wrong>
  why:  <root cause if known>
  fix:  <suggested next step>

Configuration resolution order

Implemented in config/loader.py → load_resolved(). The five layers are deep-merged from lowest to highest priority. Each key tracks its winning source for ai-evals config --print.

Inference engine

The inference layer lives in ai_eval/inference/ and answers: "What AI tasks does this repo contain, and what type are they?" It does this with pure static analysis — no importing, no executing.

scan_repo — the main entry point

inference/ast_scan.py → scan_repo(root) is the top-level function called by ai-evals init and ai-evals analyze. Here is what it does in order:

Detectors

Every detector subclasses inference/detectors/base.py → Detector and implements two methods:

class Detector(ABC):
    framework: str  # "openai" | "langchain" | "chromadb" | …

    def matches(self, tree, imports) -> bool: ...
        # Quick check: does this file import the relevant framework?

    def extract(self, tree, imports, file_path, project_root,
               *, calls, defs) -> list[DetectedTask]: ...
        # Detailed extraction: which calls, which enclosing function?

Partition invariant — OpenAI chat vs tools

Two detectors target the OpenAI SDK. They must be mutually exclusive — the same call cannot produce two tasks. The partition is enforced through shared predicates in signatures.py:

OPENAI_TOOL_KWARGS = frozenset({"tools", "functions", "tool_choice"})

def is_openai_completions_create(call): ...
# True for .chat.completions.create(…) or .ChatCompletion.create(…)

def has_openai_tool_kwarg(call): ...
# True when call has tools=, functions=, or tool_choice= kwarg

Built-in detector summary

How attr_chain works

The key primitive used by all detectors is signatures.py → attr_chain(node). It walks an ast.Attribute chain upward and returns a list of names. The tricky case is a chained receiver like OpenAI().chat.completions.create(…) where the root is a call expression, not a name:

attr_chain( OpenAI().chat.completions.create.func )
  # → ['<call>', 'chat', 'completions', 'create']

chain[-3:] == ['chat', 'completions', 'create']  # → True ✓

The leading sentinel <call> means detectors only need to match the tail of the chain, regardless of what the receiver expression is.

Synthesize — DetectedTask → RubricsConfig

inference/synthesize.py → build_rubrics(scan) converts the raw list of detected tasks into a validated RubricsConfig Pydantic model ready to be serialized to YAML.

It does three things:

1. Assigns default metrics per task type. tool_calling tasks get argument_accuracy + hallucination_rate. rag tasks get context_precision + faithfulness. These thresholds are editable in rubrics.yaml after init.
2. Resolves name collisions. If two files both have a def run() that uses LLMs, synthesize gives the second one the name run_2, then run_3, etc.
3. Classifies the project type. Counts task types and picks: rag / tool_calling / agent / rag_and_tools / chat / custom. Written into rubrics.yaml as metadata.

Scaffold writers

The scaffold/ package contains the code that actually writes files to disk. Three important invariants:

rubrics_writer

Converts a RubricsConfig to YAML via yaml.safe_dump and prepends a header comment. The YAML is deterministic (no random ordering) because Pydantic serializes in field declaration order.

tests_writer

Reads scaffold/templates/tests_py.tmpl from the installed package using importlib.resources and writes it as eval/tests.py. The template is a pytest file that calls python -m ai_eval run --fail-on-regression as a single test.

gitignore_patch

Reads the existing .gitignore, checks line-by-line for .ai-evals/, and appends the block only when absent. Never touches any existing content.

Data contracts (on-disk JSON/YAML schemas)

Every file written by ai-evals embeds schema_version: 1. Breaking changes bump the version and require a migration path in config/loader.py.

rubrics.yaml — the editable source of truth

schema_version: 1
project_type: rag_and_tools        # inferred, editable
judge:
  default: ollama/qwen2.5-coder:7b
  regression_check: openai/gpt-4o-mini
  fallback: []
defaults:
  parallel: 4
  cache: true
  tolerance: 0.02
tasks:
  customer_support_agent:
    file_path: src/agents/support.py
    entry: customer_support_agent     # enclosing function
    type: tool_calling
    inputs: [messages, tools]
    outputs: [tool_calls, content]
    metrics:
      - name: argument_accuracy
        threshold: 0.9
        weight: 1.0

golden_set.json — captured I/O examples

{
  "schema_version": 1,
  "tasks": {
    "customer_support_agent": [
      {
        "id": "gs_a3f8",
        "captured_at": "2026-06-25T08:00:00Z",
        "input": { "messages": [...] },
        "trace": { "calls": [...] }
      }
    ]
  }
}

Pydantic schema hierarchy

Telemetry — logging & progress

Two separate channels, as required by the design:

The status() context manager from progress.py shows a Rich spinner when stderr is a TTY and enabled=True. It silently yields (no-op) when piped, in CI, or when --quiet is set.

Security model (plan §2.12)

• Write boundary. The tool never writes outside <cwd>/eval/, <cwd>/.ai-evals/, or ~/.config/ai-evals/ unless --out is passed.
• Read boundary. ast_scan respects .gitignore via git check-ignore. Files the user has gitignored (e.g. secrets modules, generated artifacts) are never scanned and never appear in rubrics.yaml.
• doctor is read-only. It never creates directories or writes files. The writability check uses os.access on the nearest existing ancestor.
• No outbound telemetry. No data is sent anywhere. The --telemetry on hook is a design placeholder for anonymous version pings only, not implemented.
• Trace redaction (Phase 3). The bootstrap layer (not yet implemented) will redact env-var-looking strings (OPENAI_API_KEY=…, Bearer …) from captured traces, and honor a per-task redact: list in rubrics.yaml.

Test suite

49 tests in tests/, run with pytest -q.

$ .venv/bin/python -m pytest -q
.................................................
49 passed in 0.38s

What comes next (Phase 2–4)

The CLI surface for all future commands is already live as stubs. Each stub parses its flags per the design spec and exits 2 with a clear "Phase X" message. When a phase is implemented, its module replaces the stub without changing the command name or any flag names.

Generated from source — ai-evals v0.1.0 · Phase 1 complete · .kilo/plans/ai-evals-cli-and-system-design.md