Metadata-Version: 2.4
Name: python-yama
Version: 0.5.0
Summary: A reproducible evaluation runner for tool-using Agent skills
Project-URL: Repository, https://github.com/world-sim-dev/yama
Project-URL: Issues, https://github.com/world-sim-dev/yama/issues
Author-email: Farmer Sun <podpodiumapp@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: agent,evaluation,llm,skill,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Requires-Python: >=3.12
Requires-Dist: fastapi>=0.139.0
Requires-Dist: jinja2>=3.1
Requires-Dist: litellm<2.0,>=1.80
Requires-Dist: orjson>=3.11.9
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich<15.0,>=14.1
Description-Content-Type: text/markdown

# yama

English | [简体中文](https://github.com/world-sim-dev/yama/blob/main/README.zh-CN.md)

`yama` is a reproducible evaluation framework for tool-using Agent Skills: a declarative YAML Case describes what the model sees — the system prompt, Skill metadata, tool schemas, and multi-turn user messages — then yama calls the LLM through LiteLLM, drives the tool loop, and scores the transcript with deterministic hard checks or an LLM judge. See the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese) for the full Case schema and execution contract.

## Quick start

```bash
# Run from a plugin root; collects __evals__/cases/**/*.yaml by default
cd dingding-simple
OPENAI_API_KEY=... uv run --project yama yama

# Or, from a workspace root (a directory with yama.toml), pick a plugin by name
OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple
```

Instead of prefixing every command, you can put credentials in a `.env` file: on startup the CLI loads the nearest `.env` found from the working directory upward (variables already set in the environment are not overridden).

## Writing a Case

A Case is a YAML file organized as `context` (what the model sees) → `mocks` (how tool calls get executed) → `steps` (user turns sent in order, each with assertions) → `outcome` (the pass bar for the whole Case):

```yaml
description: The model reads the DSL before proposing creative directions

context:
  system_prompt: { default: true }     # use SYSTEM.md from the plugin root

  tools:
    - file: tools/read-dsl.yaml         # tool schema, resolved relative to __evals__

mocks:
  tools:
    read_dsl:
      respond:
        result:
          visualStyle: { name: Vintage Film }

steps:
  - id: request-directions
    user: Give me a few creative directions
    assert:
      hard:
        - tool_called: { name: read_dsl }
        - assistant_contains: creative direction

outcome:
  require:
    hard_checks: all_pass
```

- `description` (optional) is a one-line summary of what the Case tests; it is shown in the CLI output and the HTML report.
- `context.tools` is the tool schema sent to the model; `mocks.tools` is how the runner responds when a tool call arrives. The two sets of tool names must match exactly.
- To use a Skill, declare it by name under `context.skills` and also declare `{builtin: skill}` under `context.tools`; the model reads the Skill body via `skill(name, file?, call_reason?)`.
- To simulate command-line tools, declare `{builtin: bash}` and configure per-command output under `mocks.cli`.
- `steps[].assert.hard` are deterministic checks (nine types, including `tool_called`, `tool_arguments`, and `assistant_contains`); add `assert.judge` for LLM scoring.

For the full schema (every Skill/tool/mock form, the `bash` sandbox, message injection, judge configuration, and more), see the [design doc](https://github.com/world-sim-dev/yama/blob/main/docs/agent-skill-test-framework-design.md) (Chinese).

## CLI usage

```bash
uv run --project yama yama --plugin dingding-simple --report
```

| Flag | Meaning |
| --- | --- |
| `paths` (positional, repeatable) | Explicit Case YAML paths to run |
| `--plugin-root PATH` | Use the given path as the single plugin root |
| `--plugin NAME` (repeatable) | Select plugins by name from `yama.toml` |
| `--all-plugins` | Run every plugin configured in `yama.toml` |
| `-k, --filter PATTERN` (repeatable) | Only run cases whose case key (path relative to the plugin root) matches — a substring, or an fnmatch glob when the pattern contains `*` `?` `[`; with several patterns a case runs if any matches |
| `--result-dir PATH` | Override the artifact root (default `<plugin_root>/.yama/runs`) |
| `--concurrency N` | Maximum number of cases running in parallel (default 10; `1` runs cases one at a time) |
| `--report [PATH]` | Also generate a single-file HTML report. Without PATH it writes a timestamped `.yama/reports/report-<ts>.html` (history accumulates) and keeps `.yama/reports/latest.html` as a symlink to the newest one; an explicit PATH writes exactly that file |
| `--preview` | Don't run anything (no LLM calls): resolve the collected Cases and write an HTML case preview (default `.yama/reports/preview.html`; `--report PATH` overrides), or print a JSON preview to stdout when combined with `--json` |
| `--list` | Only print the collected Cases, without running them |
| `--no-artifacts` | Write no artifact files at all |
| `--json` | Emit machine-readable JSON instead of Rich tables |

`--plugin`, `--all-plugins`, and `--plugin-root` are mutually exclusive. When none of them is given, yama walks up from the current directory to the nearest directory containing `yama.toml` and uses it as the workspace root (falling back to the cwd), running it as a single plugin root.

### Dashboard

```bash
uv run --project yama yama dashboard --plugin dingding-simple
```

`yama dashboard` starts a local HTTP server (default `http://127.0.0.1:8765/`, change with `--host`/`--port`; it also accepts the same plugin selection flags as the main command plus `-k/--filter` and `--result-dir`). The index page lists the historical HTML reports accumulated under `.yama/reports/` (newest first, with the `latest` one flagged), `/reports/<name>` serves each report, and `/preview` renders the same case preview as `--preview` — re-collecting and re-resolving the case files on every refresh (collection errors render as an error page), so you can edit a Case YAML and just reload the browser. No LLM calls are ever made by the dashboard.

### Output

Cases run in parallel, up to 10 at a time by default (`--concurrency` changes the cap); runs within one case stay sequential, and results keep collection order regardless of completion order. While several cases run concurrently, a transient dashboard at the bottom of the terminal shows one row per case, updated in place — waiting, then a spinner with the run/step currently executing, then its ✓/✗ verdict — and each case's full run/step detail is printed above the dashboard as one block the moment it finishes, so blocks of different cases never interleave. With `--concurrency 1` (or a single collected case) it streams instead — a spinner marks the step currently executing (on a terminal), and each case, run, and assertion result is printed the moment it completes. Either way a per-case summary table closes the output, and with `--json` the progress stream goes to stderr so stdout stays valid JSON.
