# Analytics & historical state — implementation roadmap

**Status:** Core instrumentation implemented (soul tick extras, `health_history`, outreach transitions, `emotion_signal_events`, `session_context_snapshots`, nullable `trace_id`/`session_id` on select tables, dashboard **Emotions → Mind state trends**). Phase 7–8 helpers and full trace threading remain optional — see `docs/ANALYTICS-CORRELATION.md`.

**Goals:** Time-aligned data for rubrics, regression detection, dashboards, and research joins; minimal duplication with existing tables (`emotion_history`, `needs_history`, `user_needs_history`, `llm_call_log`, `system_events`, `conversation_turns`).

**User needs (live):** `user_needs_history` is populated on soul tick and after post-conversation inference; the operator dashboard **Emotions → Mind state trends** page plots **User needs over time** alongside Furoshiki’s five derived needs. **`furoshiki mind-state`**, **`GET /api/v1/state-series`**, and the **Emotions → Mind analytics** report surface the same windowed counts and trigger histograms — use `docs/ANALYTICS-CORRELATION.md` for join keys.

**Principles:**

- **Append-only** or **rolling retention** per series class; avoid unbounded growth.
- **Correlate** with existing identifiers (`trace_id`, `session_id`, soul tick timestamps) where possible.
- **Dashboard + API + CLI** parity for anything operators need to inspect (`furoshiki` subcommand or `GET` mirror).
- **Design doc + `furoshiki design update`** when schema or major flows change (see repo rules).

---

## Phase 1 — Extend soul-tick snapshots (highest leverage)

**Problem:** `urgency_tier`, `emotional_intensity`, and `emotional_coherence` live in heartbeat and prompts but are not stored next to `emotion_history` rows, so trend and rubric joins are incomplete.

**Deliverables:**

1. **Schema:** Add columns to `emotion_history` **or** create `mind_tick_extras` keyed by `emotion_history.id` / `recorded_at` (prefer **columns on `emotion_history`** to keep one row per tick: `urgency_tier INTEGER`, `emotional_intensity REAL`, `emotional_coherence REAL` — all nullable for backfill).
2. **Write path:** `soul_engine.update_emotions()` — same transaction as existing inserts.
3. **Read path:** `collect_mind_state_series()` / `collect_emotion_history()` include new fields on each history row.
4. **Dashboard:** Emotions → Mind state trends — optional extra lines or a small secondary chart for intensity/coherence; show `urgency_tier` in table/tooltip.
5. **Retention:** Same 7-day prune as `emotion_history` today.
6. **Tests:** Unit tests for serialization; optional migration test on existing DBs.

**Acceptance:** `furoshiki mind-state --json` and `/state-series` show the new fields per timestamp; charts or summary expose them.

**Estimated dependency:** None (extends current pipeline).

---

## Phase 2 — Health time series (operational + confound control)

**Problem:** `health` in heartbeat is **live-computed** (`live_health.py`); on-disk narrative is stale. You cannot plot “pipeline health vs mood” without periodic captures.

**Deliverables:**

1. **Schema:** `health_history` — `recorded_at`, `health_json TEXT` (full `compute_live_health` output), optional `brain_age_sec REAL`, `source TEXT` default `'read_heartbeat'` or `'soul_tick'` (pick one writer to avoid duplicate storms).
2. **Write path:** Prefer **one writer** — e.g. append on **soul_engine tick** after heartbeat write (same cadence as Phase 1), **or** sample every N minutes from a single scheduled job — document choice.
3. **Retention:** 14–30 days rolling (longer than emotions if useful for ops); configurable constant.
4. **API/dashboard:** `collect_health_history(hours)` + dashboard card or sparkline; optional CLI `furoshiki health-history --json`.
5. **Tests:** Round-trip JSON; prune behavior.

**Acceptance:** Queryable series correlating with Phase 1 timestamps (same wall-clock bucket).

**Dependency:** Prefer after Phase 1 so dashboards can combine “tick extras + health” in one view later.

---

## Phase 3 — Structured emotion-signal events (testing + analytics)

**Problem:** `emotion_signals.py` logs human-readable lines only; rubrics cannot assert “signal X fired on turn Y” without log scraping.

**Deliverables:**

1. **Schema:** `emotion_signal_events` — `id`, `recorded_at`, `source TEXT` (e.g. `tier1_keywords`, `tier2_micro`), `signals_json TEXT` (list of signal names), optional `turn_context TEXT` hash or `conversation_turns` id if available, optional `trace_id`.
2. **Write path:** After classification in `process_turn_emotions` / equivalent — **best-effort insert**, never block reply path (async queue or same pattern as `llm_call_log` batch if needed).
3. **Retention:** 30 days default.
4. **API/query:** `furoshiki db`-style query or `GET /api/v1/emotion-signals?hours=` for dashboard debug tab.
5. **Tests:** Golden tests for known transcripts → expected signal rows.

**Acceptance:** Automated test can assert rows exist for a fixture turn without parsing logs.

**Dependency:** Can run in parallel with Phase 2 after Phase 1 is stable (different subsystem).

---

## Phase 4 — Session-context metadata snapshots (prompt-path rubrics)

**Problem:** `session-context.json` is **overwritten**; rubrics need “what pieces were on at time T” without storing full multi-KB prompts.

**Deliverables:**

1. **Schema:** `session_context_snapshots` — `recorded_at`, `generated_at` from payload if present, `content_hash TEXT`, `piece_flags_json TEXT` (booleans / counts: e.g. `has_user_need_directives`, `behavioral_cues_chars`, `narrator_briefing_chars`, zone counts), optional `trace_id`, optional `session_id`.
2. **Write path:** End of `refresh_session_context.py` (or single exit path that writes the file) — compute hash + flags from structured data **before** write.
3. **Retention:** 7–14 days (high row count if frequent); consider sampling (e.g. one row per refresh **or** only when `generated_at` changes materially).
4. **Dashboard:** Debug panel row “last N snapshots” with flags; link to trace when `trace_id` present.
5. **Privacy:** Do **not** store full prompt text by default; hash + flags only.

**Acceptance:** `furoshiki trace --simulate` scenarios can be cross-checked against snapshot flags for the same window.

**Dependency:** None hard; soft dependency on consistent `trace_id` propagation (Phase 6) for best joins.

---

## Phase 5 — Outreach state history (behavioral + rubrics)

**Problem:** Outreach gating (`outreach`, `last_proactive_outreach`) is merged in heartbeat but not **historically** visible — hard to validate cadence, space veto, and streak resets.

**Deliverables:**

1. **Schema:** `outreach_state_history` — `recorded_at`, `proactive_sends_since_user INTEGER`, `pause_until_iso TEXT`, optional `last_proactive_surfaced_at`, `trigger TEXT` (`user_message`, `voice_send`, `manual`, `soul_tick`), optional JSON for `last_proactive_outreach` subset.
2. **Write path:** On **state transitions** (increment send, reset on user message, pause change) in `db.py` / `telegram_listener.py` / `voice_dispatcher.py` — **not** every read; avoid noise.
3. **Retention:** 30–90 days.
4. **Dashboard:** Time series or event table; export JSON for tests.
5. **Tests:** Simulate send/user message → expected rows.

**Acceptance:** Rubric “at most one proactive in window” can be checked from SQL.

**Dependency:** Coordinate with Phase 2 (health) only for dashboard layout, not schema.

---

## Phase 6 — Identifier normalization (`session_id`, `trace_id`)

**Problem:** Joins from LLM spend → emotional moments → post-conversation → evaluation are harder when IDs are optional or inconsistent.

**Deliverables:**

1. **Inventory:** `emotional_weights`, `self_observations`, `llm_call_log`, post-conversation writes, `user_needs_history` (where applicable).
2. **Schema:** Add nullable `trace_id` / `session_id` columns where missing; backfill **best-effort** from existing `llm_call_log` or session logs.
3. **Write paths:** Thread `trace_context` from `telegram_listener` / `call_llm` into all reflection and weight writers.
4. **Docs:** Short “correlation guide” for eval harness authors.
5. **Tests:** One integration test proving a single conversation produces rows shareable by `trace_id`.

**Acceptance:** One SQL query can aggregate cost + weights + user_needs inference for one session.

**Dependency:** Phases 1–5 benefit; implement when churn from Phases 1–4 stabilizes (avoid constant migration churn).

---

## Phase 7 — Long-horizon links & derived metrics (lower urgency)

**Deliverables:**

1. **Deep reflection ↔ emotion window:** Optional script or view: “personality_drift_log row + nearest `emotion_history` aggregate for ±24h.”
2. **Diagnosis ↔ health_history:** Compare `diagnosis_reports` with `health_history` same day (sanity check).
3. **Chroma / user_facts:** Optional **aggregate** table or cron job: counts by category per day (not per-fact embedding — too heavy).

**Acceptance:** Research / one-off notebooks can join without ad-hoc file parsing.

**Dependency:** Phases 1–2 provide the emotion/health series to join.

---

## Phase 8 — Legacy cleanup & single source of truth for tokens

**Deliverables:**

1. Reconcile **`token-usage.json`** vs **`llm_call_log`** — document canonical source; deprecate or auto-sync legacy file.
2. **`query_db.py` / operator docs** updated so analytics paths are obvious.

**Acceptance:** No conflicting numbers between dashboard budget and raw SQL for the same day.

**Dependency:** Independent; safe after Phase 6 if cost joins matter.

---

## Cross-cutting checklist (every phase)

| Item | Action |
|------|--------|
| `init_db.py` | `CREATE TABLE` / `ALTER` + indexes |
| Operators | `furoshiki db migrate` |
| Dashboard | Tab, chart, or table as appropriate |
| API v1 + dashboard routes | Read endpoints with `hours` / `limit` |
| CLI | New subcommand or flags on `mind-state` / `health` family |
| `docs/SELF-AWARENESS-DESIGN.md` | Prose + table list |
| `furoshiki design update` | After design edits |
| Tests | Pytest + optional rubric fixture |

---

## Suggested sequencing summary

| Order | Phase | Focus |
|------|--------|--------|
| 1 | Soul-tick extras | `urgency_tier`, intensity, coherence |
| 2 | Health history | Live health JSON over time |
| 3 | Emotion signal events | Structured test assertions |
| 4 | Session-context snapshots | Flags + hash, not full prompt |
| 5 | Outreach history | Gating & cadence proof |
| 6 | ID normalization | Joins across subsystems |
| 7 | Long-horizon joins | Drift/diagnosis/Chroma aggregates |
| 8 | Token legacy cleanup | Single cost source of truth |

---

## Out of scope for this roadmap (explicit)

- Storing **full** prompts or entire `session-context.json` in SQLite (privacy, size) — use hashes + flags unless a separate encrypted research export is requested.
- Replacing **markdown session logs** — keep as narrative; derive features in Phase 7 if needed.
