Admin Login

Stored in localStorage on this browser only. Use Logout to clear.

Activity
Eval
Subscribers
Messages
Usage
Providers
Logs
Loading…

LLM Coach Eval

Headline scorecard — coach vs real players

The single number to watch: contested match%. Match% on every replay decision can be inflated by easy moves (every winning player attacks all-out). Contested match% filters to decisions where the player had a real choice — multiple plays available, or the player held back attackers, or actively chose to block. That's the call where coaching matters.

No replay-match data yet. Run tools.eval.replay.run, then tools.eval.upload_results.

Cell colors: green = strong; yellow = ok; red = weak. Latency thresholds: <1500 ms green, 1500-3000 ms yellow, >3000 ms red.

1. Replay match-arc — vs actual recorded matches

For every decision point in a real .rply file, the coach gets the live game state + legal actions and is scored against what the player actually did. Covers ActionsAvailable (what to cast), DeclareAttackers, DeclareBlockers, and Mulligan. The contested-only chart filters out the easy moves so you can see which backend is picking real coaching plays right.

No replay-match results uploaded yet.
Contested-decisions match% appears once results land.
Match% by ActionType appears once results land.
Per-replay distribution appears once results land.

2. 17lands turn-action — vs winning Diamond+ players

For each turn of a winning Diamond+ game (sampled early/mid/late), asks the coach which action categories to take. Ground truth = what the player actually did that turn. Mean Jaccard is the headline — it measures set similarity (recommending 3 of 4 right tags scores 0.75). Exact-match% requires every tag right and no extras (very strict). ACTIVATE is excluded because the prompt doesn't surface activatable abilities.

No turn-action results uploaded yet.
Per-category F1 appears once results land.

Per-set latest run

No per-set data yet — uploads with set_code show one row per set here.
Lines = exact-match% and mean Jaccard over time.
Trend appears once 2+ runs are uploaded.

3. 17lands mulligan — keep/mull on real opening hands

Asks each backend to KEEP or MULLIGAN a real Diamond+ opening hand. Ground truth = empirical higher-WR option in the bucket. Balanced accuracy is the headline — average of keep_acc + mull_acc, so models can't game it by always picking keep. The Diamond+ dataset is ~96% keep-favored, so raw higher_wr_rate rewards always-keep baselines. 50% balanced = coin flip; 65%+ = real signal.

No 17lands-mulligan results uploaded yet.
Decision share appears once results land.

Per-set latest run

No per-set data yet — uploads with set_code show one row per set here.
Higher_wr% over time. 50% = coin flip.
17lands trend appears once 2+ runs are uploaded.

4. General LLM eval — vs judge model (advice quality)

Replays captured coach prompts through one or more backends; scored by an online judge model on a 1-5 rubric (correctness, reasoning, conciseness, legality). Useful for catching regressions in advice voice and hallucinated cards, but the judge is itself an LLM — not a substitute for the player-grounded evals above.

No general-eval results uploaded yet.
Score histograms appear once results land.

General eval — trend over time

Y-axis is 1-5 (higher = better). Click legend entries to hide/show lines.
Trend appears once 2+ runs are uploaded.

Create Subscriber

Subscribers

KeyEmailNameStatusPinned modelLatest FrontendLatest VersionInstalls (30d)Last SeenRequests (30d)Tokens (30d)Actions

Send Message to All Subscribers

Message History

Usage (Last 30 Days)

KeyRequestsPrompt TokensCompletion TokensTotal Tokens

Backend Providers

Live routing config. Edits are persisted in the proxy database and override the baked-in config.yaml. Click Save to apply (router reloads in-process — no container restart needed).

Server Logs

Loading...