Metadata-Version: 2.4
Name: eval-doctor
Version: 0.1.1
Summary: Know what you are not testing before you ship - an eval coverage auditor for RAG and agent systems.
Author: Sneha Upadhyaula
License: MIT
Keywords: evals,llm,rag,agents,testing,evaluation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer>=0.9
Requires-Dist: pydantic>=2.0
Requires-Dist: rich>=13.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: tomli>=2.0; python_version < "3.11"
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == "anthropic"
Provides-Extra: openai
Requires-Dist: openai>=1.60; extra == "openai"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.15; extra == "dev"
Dynamic: license-file

# Eval Doctor

> Know what you are not testing before you ship.

Eval Doctor is a CLI that helps AI builders find blind spots in their eval suite — a linter for AI evals.

Most AI projects start with architecture and implementation. Evals come later, once the system already "kind of works." But for RAG systems and agents, the hardest part is not writing test cases — it's knowing **what failure modes matter**. Eval Doctor scans your repo, detects your AI system's capabilities, maps them to expected evaluation dimensions, and tells you what your current eval suite is missing.

```text
You are testing faithfulness and answer relevance.

You are not testing:
- empty retrieval
- stale documents
- tool timeout
- hallucinated tool arguments
- prompt injection
- ambiguous user requests
```

## Install

```bash
pip install eval-doctor               # static analysis, no API keys needed
pip install "eval-doctor[anthropic]"  # + optional LLM-powered capability detection (Claude API)
pip install "eval-doctor[openai]"     # + optional LLM-powered capability detection (OpenAI API)
```

Already have [Claude Code](https://claude.com/claude-code) installed? No extra or API key needed — `eval-doctor audit --llm` uses your Claude subscription through the `claude` CLI.

## Usage

```bash
cd your-ai-project
eval-doctor audit
```

You get a console report plus two files:

- `eval_doctor_report.md` — human-readable coverage report
- `eval_doctor_findings.json` — machine-readable findings for automation

Options:

```bash
eval-doctor audit --repo path/to/repo      # audit a different directory
eval-doctor audit --evals path/to/evals    # eval files living outside the repo
eval-doctor audit --output-dir reports/    # where to write report files
eval-doctor audit --llm                    # force LLM enrichment (also enables the claude CLI path)
eval-doctor audit --no-llm                 # force pure static analysis
eval-doctor audit --force                  # overwrite existing report files
eval-doctor interview                      # short risk interview; audit picks up the answers
eval-doctor show-taxonomy                  # print the built-in taxonomy
```

### Risk interview

Severity defaults are generic; your risks are not. `eval-doctor interview` asks a few multiple-choice questions keyed to what the scanner actually found in your repo — retrieval, web grounding, tool calling, SQL, memory — plus where the system will run, and writes `eval_doctor_risk.json`. The next `eval-doctor audit` picks it up automatically (or point at one with `--risk path/to/eval_doctor_risk.json`) and elevates the severity of the dimensions you said would hurt most, re-ordering the report. Answers only ever raise severity — the interview never argues a risk away.

## How it works

```text
Repo Scanner ──► Capability Extractor ──► Coverage Engine ──► Report
(deterministic)  (static rules,           (expected vs
                  optional LLM enrich)     observed dimensions)
```

1. **Scan** — a deterministic scanner walks your repo and detects signals: frameworks (LangChain, LlamaIndex, CrewAI, ...), vector DBs, embeddings, chunking, reranking, tool definitions, SQL, browser automation, memory, prompt templates, and existing eval/test files. Nothing leaves your machine.
2. **Extract** — signals are mapped to a capability profile (`rag`, `tool_calling_agent`, `sql_agent`, ...).
3. **Compare** — the profile selects applicable dimensions from a built-in taxonomy of 40 eval dimensions across RAG, tool calling, SQL agents, memory, and general AI risks. Your existing eval files are checked for evidence of each.
4. **Report** — every applicable dimension is rated `covered`, `weak`, or `missing`, with a severity and a concrete recommendation.

### Optional LLM enhancement

Static analysis sees imports and patterns, not intent. With an LLM available, Eval Doctor sends its collected evidence snippets (never your whole repo) to a model to correct and complete the capability profile — catching things like web-search-grounded generation that deserves faithfulness evals, hand-rolled RAG with no framework, and pruning false positives like an sqlite cache being flagged as a SQL agent. Two ways to enable it:

- **API key** — set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` and install the matching extra. Used automatically when present.
- **Claude subscription** — have [Claude Code](https://claude.com/claude-code) installed and signed in, then pass `--llm`. Eval Doctor shells out to `claude` on your machine; no API key or extra needed. This path is opt-in only — an installed CLI is never used unless you ask.

Without either, everything still works; the report just notes it ran in static-only mode.

Additional providers are welcome contributions: implement the two-method `LLMProvider` protocol in [`eval_doctor/llm.py`](eval_doctor/llm.py) — the existing providers are small, self-contained classes.

## Example

See [`examples/sample_report.md`](examples/sample_report.md) for the report produced by auditing a small RAG + tool-calling fixture project.

## Roadmap

- **Candidate eval generation** — generate golden eval cases for your missing dimensions
- **Coverage diff** — `eval-doctor diff` to score an eval suite against recommended coverage
- **CI / PR mode** — comment on PRs that add AI capabilities without corresponding evals

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

MIT
