Metadata-Version: 2.4
Name: fred-deepeval-cli
Version: 0.1.3
Summary: External CLI for evaluating Fred agent turns via /agents/evaluate
License: Apache-2.0
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
Requires-Dist: fred-sdk>=2.0.7
Requires-Dist: fred-runtime>=2.0.8
Provides-Extra: dev
Requires-Dist: bandit>=1.8.6; extra == "dev"
Requires-Dist: basedpyright==1.31.0; extra == "dev"
Requires-Dist: detect-secrets>=1.5.0; extra == "dev"
Requires-Dist: pytest>=8.4.2; extra == "dev"
Requires-Dist: pytest-cov>=6.2.1; extra == "dev"
Requires-Dist: pytest-socket>=0.7.0; extra == "dev"
Requires-Dist: ruff>=0.12.5; extra == "dev"
Provides-Extra: eval
Requires-Dist: deepeval; extra == "eval"
Requires-Dist: litellm; extra == "eval"
Requires-Dist: python-dotenv; extra == "eval"
Requires-Dist: rich>=13.0; extra == "eval"
Requires-Dist: temporalio; extra == "eval"

# fred-deepeval-cli

External CLI for evaluating one Fred agent turn through `POST /agents/evaluate`.

## Purpose

This project provides a small external CLI that:
- calls a Fred pod `/agents/evaluate` endpoint
- receives an `EvalTrace`
- classifies the turn outcome
- resolves an evaluation preset from `agent_tags`
- computes structural checks
- scores the trace with DeepEval

## Commands

```bash
make dev
make eval-dev
make test
make code-quality
make cli
make score BASE_URL=http://127.0.0.1:8000/fred/agents/v2 AGENT_ID=fred.test.assistant INPUT="echo bonjour" SESSION_ID=eval-001 USER_ID=alice
make sql-scenarios BASE_URL=http://127.0.0.1:8000/fred/agents/v2

## Documentation

| Topic | File |
| --- | --- |
| Evaluate any Fred agent pod | `docs/evaluating-any-fred-agent.md` |
| RAG evaluation — approach and metrics | `docs/rag-evaluation-rfc.md` |
| RAG local setup guide | `docs/rag-local-setup.md` |
| SQL evaluation | `docs/sql-evaluation.md` |
| OTel export strategy | `fred/docs/swift/rfc/AGENT-EVALUATION-RFC.md §13` |

## Architecture — EVAL-01 Phase 1

This CLI is being restructured into a reusable library core so the Fred
Control Plane evaluation worker can call it directly without spawning a subprocess.

- `fred_deepeval_cli/core/` — callable library (models, evaluator, profiles, scorer, judge factory)
- `fred_deepeval_cli/cli/` — thin CLI adapter over the core
- `fred_deepeval_cli/worker_adapter.py` — public entrypoint for the Control Plane worker

The CLI interface and JSON output remain unchanged.
See EVAL-01 Phase 1 issue and `fred/docs/swift/rfc/AGENT-EVALUATION-RFC.md §7.3`.
