# Evaluation rubrics — follow-up (saved plan)

**Status:** Idea backlog — not implemented. When you are ready to test whether Furoshiki (including user needs) is *actually* working, use this as the starting point.

**Related:** `docs/USER-NEEDS-TIME-SERIES-PLAN.md` (Phase D — “Was the need met?”), `furoshiki trace --simulate` for prompt-path audits.

---

## 1. What to measure (layers)

| Layer | Question the rubric answers |
|--------|-----------------------------|
| **Instrumentation** | Are signals computed, stored, and visible? (heartbeat, session-context, `user_needs_history`, dashboard) |
| **Prompt path** | Do the right pieces reach the model? (`furoshiki trace --simulate`, piece labels) |
| **Behavior** | Does the *next reply* match the intent implied by state? (human or LLM-as-judge) |
| **Outcome** | Over time, does the user experience improve on what you care about? (lighter-weight, later) |

Start with **instrumentation + prompt path** (cheap, objective), then add **behavior** (small scripted scenarios).

---

## 2. Scenario bank (10–20 rows to start)

Each scenario is one row with:

- **Setup** — e.g. user says they want playfulness but only discusses work stress.
- **Expected signals** — e.g. conflict or elevated dimensions in snapshots after post-conversation inference runs.
- **Expected prompt** — e.g. `user_need_conflict` or `user_need_directives` non-empty with the right *shape* (not exact wording).
- **Pass criteria** — checklist (see §3).

Add separate scenarios for **outreach** (space veto, no spam, cadence).

---

## 3. Checklist rubric per scenario

Example dimensions (pick 5–8):

1. **Grounding** — Reply stays on the user’s last message; no invented facts.
2. **Directive use** — Tone matches elevated user-need dimensions without naming internal scores.
3. **Conflict handling** — When the scenario implies mismatch, the model briefly names tension or asks one clarifying question.
4. **Safety / boundaries** — No leaking internals (health scores, raw need numbers), per existing prompt rules.
5. **No spam** — Outreach scenarios: at most one proactive send in the test window; respects high `space` when applicable.

Score each **pass / partial / fail** with a one-line note.

---

## 4. How to run tests (semi-automated)

**Automated (scriptable today):**

- `furoshiki trace --simulate "<user message>"` — assert expected **piece labels** exist (`user_need_directives`, `user_need_conflict`, `behavioral_cues`, `narrator_briefing`, …).
- After a controlled post-conversation run in a test instance, inspect `user_needs_history` and heartbeat JSON for expected structure.

**Manual (gold standard for “does it feel right?”):**

- Same scenarios in Telegram with a short rubric (spreadsheet or markdown table).

**LLM-as-judge (optional):**

- Input: user message, assistant reply, rubric dimension definitions.
- Output: pass/fail per dimension with short evidence quotes.
- Version the judge prompt so scores are comparable over time.

---

## 5. Baseline and regression

- Snapshot **day zero** when defaults are acceptable: heartbeat, sample `user_needs_history` rows, a few full trace outputs.
- After changes, re-run the **same scenario bank** and compare pass rates and trace pieces.
- Treat **missing prompt pieces** as regressions before debating prose quality.

---

## 6. Tie-in to Phase D

The long-term “compare inferred `user_needs(t)` to assistant behavior labels” path in the user-needs plan is the automated version of this rubric. Until labels exist in the DB, the checklist + scenario bank is the practical substitute.

---

## 7. Suggested first week (when you pick this up)

1. Write **8 scenarios** (2 instrumentation-only, 4 behavior, 2 outreach).
2. Add a **one-page rubric template** (checkboxes + notes).
3. Run each scenario once; log gaps (missing piece → engineering bug; wrong tone → inference or prompt tuning).
