LIBRARY-OK
Group A (Offline Library Claims, Checks 1–10): 10 / 10 PASS.
Group B (v0.1.0 Release Gate, Checks 11–17): 1 / 7 PASS, 6 / 7 FAIL.
Per the verdict rule (LIBRARY-OK if all Group A items pass and any Group B item fails), the library satisfies its offline behavioral, invariant, privacy, CLI, and simulator claims, while the v0.1.0 release gate items remain incomplete.
Evaluated UTC: 2026-10-23T02:05:59Z – 2026-10-23T02:14:35Z HEAD: c97122d704414a00f118f93cfb5242aa5073d9f1 Live API Spend During Eval: $0.00 (0 network calls) Terminal Recording: hearth-v0.1-proof.typescript (via /usr/bin/script; asciinema not installed; no video/screenshot fabricated)

Summary Checklist

Check Group Requirement Exit Code Status
#1 Group A make check exits 0 (ruff, ruff format, mypy --strict src/hearth, pytest -m "not contract"). Passed: 56, Failed: 0, Deselected: 3. 0 PASS
#2 Group A All named invariant tests exist and pass (15 listed + test_no_content_by_default = 16 in tests/test_invariants.py) + tests/test_named_tests_exist.py asserts list exists (17/17 passed). 0 PASS
#3 Group A Local offline demo with FakeProvider in observe mode: outgoing request is byte-identical to input (293 bytes in == 293 bytes out). 0 PASS
#4 Group A Local offline demo in safe mode: shuffled tool set in, first-seen order out, tool text unchanged, trace records tool_order_restored. 0 PASS
#5 Group A Local offline demo in optimize mode: volatile ISO timestamp line in system prompt moved to latest user message, trace records volatile_relocated. 0 PASS
#6 Group A read + write + uncached == total_input verified for Anthropic-shaped (3500 + 1000 + 500 = 5000) and OpenAI-shaped (2800 + 500 + 700 = 4000) usage objects. 0 PASS
#7 Group A FakeProvider session with record=False: SQLite ledger row has token counts and cost (0.00368700, 0.00373500) and zero prompt text; JSONL trace has no "body" key and zero prompt text. 0 PASS
#8 Group A hearth report --session <id> --ledger <path> and hearth diagnose <trace.jsonl> both exit 0 and name real cause codes from src/hearth/data/breaks.yaml (tools_reordered, below_minimum). 0 PASS
#9 Group A hearth replay <trace.jsonl> --provider gemini --model gemini-2.5-flash exits 0 and prints a cost for observe, safe, and optimize. 0 PASS
#10 Group A python -m hearth.bench.runner --m0 --verify, --m4 --verify, and --m6 --verify exit 0. In docs/REPORT.md, all 4 framework rows per provider have identical read share (0.0% / 48.9%) and cost ($0.0156 / $0.0093 on Gemini; $0.3201 / $0.1854 on OpenAI) → 42.0% figure labeled SIMULATOR. 0 PASS
#11 Group B Version in pyproject.toml and src/hearth/__init__.py is 0.1.0 (actual: 0.1.0b1); git tag -l has v0.1.0/0.1.0 (actual: only 0.1.0a1); git status clean (actual: untracked hearth-synthesis.html); HEAD == origin/main (true). 0 FAIL
#12 Group B A GitHub Actions workflow exists and runs make check (actual: .github/workflows does not exist). 1 FAIL
#13 Group B docs/BENCH.md has at least one spend-log row with a non-zero token total from a live provider, dated, within $25 budget (actual: Spend Log table is empty, Total Spent $0.00). 0 FAIL
#14 Group B Contract tests have evidence of five successful nightly runs (actual: not run, would spend money; zero nightly run evidence in repo or CI). 0 FAIL
#15 Group B gtm/TRIALS.md exists and contains three outsider install notes (actual: gtm/TRIALS.md does not exist). 1 FAIL
#16 Group B src/hearth/data/providers.toml source_date values are within 14 days of today (all three are 2026-10-19, 4 days prior to 2026-10-23). 0 PASS
#17 Group B PyPI trusted publishing workflow configured in repo + https://pypi.org/pypi/hearth-ai/json check (actual: no workflow exists; PyPI returns HTTP 404 "not published"). 0 FAIL

Group A — Offline Library Verification (Checks 1–10)

PASS Check 1 · make check (ruff, ruff format, mypy --strict, pytest -m "not contract")
UTC: 2026-10-23T02:07:05Z Exit Code: 0 Passed: 56 · Failed: 0 · Deselected: 3
$ make check
uv run ruff check src tests bench
All checks passed!
uv run ruff format --check src tests bench
72 files already formatted
uv run mypy --strict src/hearth
Success: no issues found in 39 source files
uv run pytest -m "not contract" tests/
============================= test session starts ==============================
platform darwin -- Python 3.12.10, pytest-9.1.1, pluggy-1.6.0
rootdir: /Users/jishmisc_pro/Documents/hearth
configfile: pyproject.toml
plugins: hypothesis-6.151.13
collected 59 items / 3 deselected / 56 selected

tests/test_breaks_doc.py ....                                            [  7%]
tests/test_cli.py ....                                                   [ 14%]
tests/test_compaction.py ....                                            [ 21%]
tests/test_FakeProvider.py ....                                          [ 28%]
tests/test_guard.py ....                                                 [ 35%]
tests/test_integrations.py ....                                          [ 42%]
tests/test_invariants.py ................                                [ 71%]
tests/test_memory.py .....                                               [ 80%]
tests/test_named_tests_exist.py .                                        [ 82%]
tests/test_nfr_latency.py .                                              [ 83%]
tests/test_offload.py ...                                                [ 89%]
tests/test_planner_ttl.py ...                                            [ 94%]
tests/test_route.py ...                                                  [100%]

======================= 56 passed, 3 deselected in 0.97s =======================
PASS Check 2 · Named Invariant Tests & Meta-Test (tests/test_named_tests_exist.py)
UTC: 2026-10-23T02:07:08Z Exit Code: 0 Passed: 17 · Failed: 0
$ .venv/bin/pytest -v tests/test_invariants.py tests/test_named_tests_exist.py
============================= test session starts ==============================
platform darwin -- Python 3.12.10, pytest-9.1.1, pluggy-1.6.0 -- /Users/jishmisc_pro/Documents/hearth/.venv/bin/python
cachedir: .pytest_cache
hypothesis profile 'default'
rootdir: /Users/jishmisc_pro/Documents/hearth
configfile: pyproject.toml
plugins: hypothesis-6.151.13
collecting ... collected 17 items

tests/test_invariants.py::test_observe_passthrough_bytes PASSED          [  5%]
tests/test_invariants.py::test_no_content_by_default PASSED              [ 11%]
tests/test_invariants.py::test_safe_no_content_change PASSED             [ 17%]
tests/test_invariants.py::test_fingerprint_determinism PASSED            [ 23%]
tests/test_invariants.py::test_append_only PASSED                        [ 29%]
tests/test_invariants.py::test_changes_reported PASSED                   [ 35%]
tests/test_invariants.py::test_usage_reconciles_anthropic PASSED         [ 41%]
tests/test_invariants.py::test_usage_reconciles_openai PASSED            [ 47%]
tests/test_invariants.py::test_usage_reconciles_gemini PASSED            [ 52%]
tests/test_invariants.py::test_fail_open PASSED                          [ 58%]
tests/test_invariants.py::test_no_extra_network PASSED                   [ 64%]
tests/test_invariants.py::test_break_attribution PASSED                  [ 70%]
tests/test_invariants.py::test_planner_limits PASSED                     [ 76%]
tests/test_invariants.py::test_memory_supersede PASSED                   [ 82%]
tests/test_invariants.py::test_memory_no_session_mutation PASSED         [ 88%]
tests/test_invariants.py::test_cost_decimal PASSED                       [ 94%]
tests/test_named_tests_exist.py::test_all_named_invariant_tests_exist PASSED [100%]

============================== 17 passed in 0.69s ==============================
PASS Check 3 · Offline Demo in observe Mode (Byte-Identical Outgoing Request)
UTC: 2026-10-23T02:08:56Z Exit Code: 0
$ .venv/bin/python -c ' import json, hearth from hearth.testing import FakeProvider from hearth.trace import TraceRecorder fake = FakeProvider(provider="anthropic", model="claude-3-5-sonnet-20241022") orig_create = fake.messages.create captured = {} def spy_create(**kw): captured["out"] = kw return orig_create(**kw) fake.messages.create = spy_create client = hearth.wrap(fake, mode="observe", recorder=TraceRecorder("/tmp/hearth_eval/traces_c3")) payload = { "model": "claude-3-5-sonnet-20241022", "system": "You are a precise assistant.\nTimestamp: 2026-10-23T02:00:00Z", "tools": [{"name": "z_tool", "description": "Z"}, {"name": "a_tool", "description": "A"}], "messages": [{"role": "user", "content": "Hello world"}], "session_id": "demo-observe", } in_bytes = json.dumps(payload, sort_keys=True).encode("utf-8") client.messages.create(**payload) out_bytes = json.dumps(captured["out"], sort_keys=True).encode("utf-8") print("IN_BYTES :", in_bytes.decode()) print("OUT_BYTES:", out_bytes.decode()) print("BYTE_IDENTICAL:", in_bytes == out_bytes, "LEN:", len(in_bytes), "vs", len(out_bytes)) assert in_bytes == out_bytes '
IN_BYTES : {"messages": [{"content": "Hello world", "role": "user"}], "model": "claude-3-5-sonnet-20241022", "session_id": "demo-observe", "system": "You are a precise assistant.\nTimestamp: 2026-10-23T02:00:00Z", "tools": [{"description": "Z", "name": "z_tool"}, {"description": "A", "name": "a_tool"}]}
OUT_BYTES: {"messages": [{"content": "Hello world", "role": "user"}], "model": "claude-3-5-sonnet-20241022", "session_id": "demo-observe", "system": "You are a precise assistant.\nTimestamp: 2026-10-23T02:00:00Z", "tools": [{"description": "Z", "name": "z_tool"}, {"description": "A", "name": "a_tool"}]}
BYTE_IDENTICAL: True LEN: 293 vs 293
PASS Check 4 · Offline Demo in safe Mode (Tool Order Restored, Text Unchanged, Trace Recorded)
UTC: 2026-10-23T02:09:18Z Exit Code: 0
$ .venv/bin/python -c ' import json, hearth from hearth.testing import FakeProvider from hearth.trace import TraceRecorder recorder = TraceRecorder("/tmp/hearth_eval/traces_c4") fake = FakeProvider(provider="anthropic", model="claude-3-5-sonnet-20241022") orig_create = fake.messages.create sent_payloads = [] def spy_create(**kw): sent_payloads.append(kw) return orig_create(**kw) fake.messages.create = spy_create client = hearth.wrap(fake, mode="safe", recorder=recorder) t_read = {"name": "read_file", "description": "Read a file from disk"} t_grep = {"name": "grep_search", "description": "Search regex in repo"} t_test = {"name": "run_tests", "description": "Execute pytest suite"} client.messages.create(model="claude-3-5-sonnet-20241022", system="Static system prompt.", tools=[t_read, t_grep, t_test], messages=[{"role": "user", "content": "Turn 1"}], session_id="demo-safe") shuffled_in = [t_test, t_read, t_grep] client.messages.create(model="claude-3-5-sonnet-20241022", system="Static system prompt.", tools=shuffled_in, messages=[{"role": "user", "content": "Turn 1"}, {"role": "assistant", "content": "Ack 1"}, {"role": "user", "content": "Turn 2"}], session_id="demo-safe") sent_tools_2 = sent_payloads[1]["tools"] trace_entries = recorder.read_session("demo-safe") print("CALL 2 TOOLS IN (shuffled):", json.dumps(shuffled_in)) print("CALL 2 TOOLS OUT (restored):", json.dumps(sent_tools_2)) print("CALL 2 TRACE CHANGES:", json.dumps(trace_entries[1]["changes"])) print("CALL 2 TRACE BREAKS :", json.dumps(trace_entries[1]["breaks"])) '
CALL 2 TOOLS IN (shuffled): [{"name": "run_tests", "description": "Execute pytest suite"}, {"name": "read_file", "description": "Read a file from disk"}, {"name": "grep_search", "description": "Search regex in repo"}]
CALL 2 TOOLS OUT (restored): [{"name": "read_file", "description": "Read a file from disk"}, {"name": "grep_search", "description": "Search regex in repo"}, {"name": "run_tests", "description": "Execute pytest suite"}]
CALL 2 TRACE CHANGES: [{"kind": "tool_order_restored", "first_seen_order": ["read_file", "grep_search", "run_tests"], "reordered_order": ["run_tests", "read_file", "grep_search"], "details": "Restored first-seen tool order to protect prefix cache"}]
CALL 2 TRACE BREAKS : [{"cause": "tools_reordered", "user_caused": true, "level": "tools", "details": "Restored first-seen tool order in safe mode"}, {"cause": "below_minimum", "user_caused": false, "level": "system", "details": "Context tokens (83) below provider minimum (2048)"}]

Observed Call 2 Input Bytes (Shuffled Tools)

[{"name": "run_tests", "description": "Execute pytest suite"}, {"name": "read_file", "description": "Read a file from disk"}, {"name": "grep_search", "description": "Search regex in repo"}] system: "Static system prompt." messages: [{"role": "user", "content": "Turn 1"}, {"role": "assistant", "content": "Ack 1"}, {"role": "user", "content": "Turn 2"}]

Observed Call 2 Outgoing Bytes (First-Seen Order Restored)

[{"name": "read_file", "description": "Read a file from disk"}, {"name": "grep_search", "description": "Search regex in repo"}, {"name": "run_tests", "description": "Execute pytest suite"}] system: "Static system prompt." messages: [{"role": "user", "content": "Turn 1"}, {"role": "assistant", "content": "Ack 1"}, {"role": "user", "content": "Turn 2"}]
PASS Check 5 · Offline Demo in optimize Mode (Volatile ISO Timestamp Relocated to User Tail)
UTC: 2026-10-23T02:09:18Z Exit Code: 0
$ .venv/bin/python -c ' import json, hearth from hearth.testing import FakeProvider from hearth.trace import TraceRecorder recorder = TraceRecorder("/tmp/hearth_eval/traces_c5") fake = FakeProvider(provider="anthropic", model="claude-3-5-sonnet-20241022") orig_create = fake.messages.create sent = {} def spy_create(**kw): sent["kw"] = kw return orig_create(**kw) fake.messages.create = spy_create client = hearth.wrap(fake, mode="optimize", recorder=recorder) in_system = "You are a senior Python engineer.\nCurrent time: 2026-10-23T02:09:00Z\nFollow PEP 8 strictly." in_messages = [{"role": "user", "content": "Refactor src/hearth/layout.py"}] client.messages.create(model="claude-3-5-sonnet-20241022", system=in_system, messages=in_messages, session_id="demo-optimize") out_system = sent["kw"]["system"] out_messages = sent["kw"]["messages"] trace_entries = recorder.read_session("demo-optimize") print("BEFORE (input system) :", repr(in_system)) print("BEFORE (input messages):", json.dumps(in_messages)) print("AFTER (sent system) :", repr(out_system)) print("AFTER (sent messages) :", json.dumps(out_messages)) print("TRACE CHANGES :", json.dumps(trace_entries[0]["changes"])) '
BEFORE (input system)  : 'You are a senior Python engineer.\nCurrent time: 2026-10-23T02:09:00Z\nFollow PEP 8 strictly.'
BEFORE (input messages): [{"role": "user", "content": "Refactor src/hearth/layout.py"}]
AFTER  (sent system)   : 'You are a senior Python engineer.\nFollow PEP 8 strictly.'
AFTER  (sent messages) : [{"role": "user", "content": "Refactor src/hearth/layout.py\n\n[Runtime Context:\nCurrent time: 2026-10-23T02:09:00Z]"}]
TRACE CHANGES          : [{"kind": "volatile_relocated", "source": "system", "target_message_index": 0, "snippets": ["Current time: 2026-10-23T02:09:00Z"]}]

Observed Input Prompt Bytes (Before optimize)

system[0..33]: "You are a senior Python engineer.\n" system[34..69]: "Current time: 2026-10-23T02:09:00Z\n" ← volatile ISO timestamp in prefix system[70..91]: "Follow PEP 8 strictly." messages[0].content: "Refactor src/hearth/layout.py"

Observed Outgoing Prompt Bytes (After optimize)

system[0..55]: "You are a senior Python engineer.\nFollow PEP 8 strictly." ← stable prefix preserved messages[0].content[0..28]: "Refactor src/hearth/layout.py" messages[0].content[29..84]: "\n\n[Runtime Context:\nCurrent time: 2026-10-23T02:09:00Z]" ← relocated to latest user message
PASS Check 6 · Usage Reconciliation (read + write + uncached == total_input for Anthropic & OpenAI)
UTC: 2026-10-23T02:11:34Z Exit Code: 0
$ .venv/bin/python -c ' import json from hearth.adapters.anthropic import parse_anthropic_usage from hearth.adapters.openai import parse_openai_usage anth_raw = {"input_tokens": 5000, "cache_read_input_tokens": 3500, "cache_creation_input_tokens": 1000, "output_tokens": 200} oai_raw = {"prompt_tokens": 4000, "prompt_tokens_details": {"cached_tokens": 2800}, "cache_write_tokens": 500, "completion_tokens": 150} anth_u = parse_anthropic_usage(anth_raw) oai_u = parse_openai_usage(oai_raw) print("ANTHROPIC FIXTURE:", json.dumps(anth_raw)) print(f"ANTHROPIC RECON : read({anth_u.read}) + write({anth_u.write}) + uncached({anth_u.uncached}) = {anth_u.read + anth_u.write + anth_u.uncached} == total_input({anth_u.total_input})") assert anth_u.read + anth_u.write + anth_u.uncached == anth_u.total_input == anth_raw["input_tokens"] print("OPENAI FIXTURE :", json.dumps(oai_raw)) print(f"OPENAI RECON : read({oai_u.read}) + write({oai_u.write}) + uncached({oai_u.uncached}) = {oai_u.read + oai_u.write + oai_u.uncached} == total_input({oai_u.total_input})") assert oai_u.read + oai_u.write + oai_u.uncached == oai_u.total_input == oai_raw["prompt_tokens"] '
ANTHROPIC FIXTURE: {"input_tokens": 5000, "cache_read_input_tokens": 3500, "cache_creation_input_tokens": 1000, "output_tokens": 200}
ANTHROPIC RECON  : read(3500) + write(1000) + uncached(500) = 5000 == total_input(5000)
OPENAI FIXTURE   : {"prompt_tokens": 4000, "prompt_tokens_details": {"cached_tokens": 2800}, "cache_write_tokens": 500, "completion_tokens": 150}
OPENAI RECON     : read(2800) + write(500) + uncached(700) = 4000 == total_input(4000)
PASS Check 7 · Privacy Invariant (record=False: Token Counts & Cost in SQLite, No Prompt Text or Body in SQLite/JSONL)
UTC: 2026-10-23T02:11:59Z Exit Code: 0
$ HEARTH_HOME=/tmp/hearth_eval/check7 .venv/bin/python -c ' import json, sqlite3, os, shutil from pathlib import Path import hearth from hearth.testing import FakeProvider from hearth.trace import TraceRecorder base = Path("/tmp/hearth_eval/check7") if base.exists(): shutil.rmtree(base) (base / "traces").mkdir(parents=True, exist_ok=True) os.environ["HEARTH_HOME"] = str(base) ledger_path = base / "ledger.sqlite" trace_dir = base / "traces" recorder = TraceRecorder(trace_dir) fake = FakeProvider(provider="anthropic", model="claude-3-5-sonnet-20241022") client = hearth.wrap(fake, mode="safe", record=False, recorder=recorder) secret_prompt = "TOP_SECRET_DO_NOT_RECORD_994812_SYSTEM_PROMPT " + ("padding token " * 250) tools_t1 = [{"name": "alpha", "description": "Alpha tool"}, {"name": "beta", "description": "Beta tool"}] tools_t2 = [{"name": "beta", "description": "Beta tool"}, {"name": "alpha", "description": "Alpha tool"}] client.messages.create(model="claude-3-5-sonnet-20241022", system=secret_prompt + "\nTimestamp: 2026-10-23T02:00:00Z", tools=tools_t1, messages=[{"role": "user", "content": "CONFIDENTIAL_USER_MSG_TURN_1"}], session_id="sess-check7") client.messages.create(model="claude-3-5-sonnet-20241022", system=secret_prompt + "\nTimestamp: 2026-10-23T02:01:00Z", tools=tools_t2, messages=[{"role": "user", "content": "CONFIDENTIAL_USER_MSG_TURN_1"}, {"role": "assistant", "content": "ack"}, {"role": "user", "content": "CONFIDENTIAL_USER_MSG_TURN_2"}], session_id="sess-check7") conn = sqlite3.connect(ledger_path) conn.row_factory = sqlite3.Row rows = [dict(r) for r in conn.execute("SELECT * FROM calls WHERE session_id = ?", ("sess-check7",)).fetchall()] db_dump = "\n".join(conn.iterdump()) conn.close() trace_file = trace_dir / "sess-check7.jsonl" trace_text = trace_file.read_text() trace_objs = [json.loads(line) for line in trace_text.strip().splitlines()] print("SQLITE LEDGER ROWS:", json.dumps(rows)) print("PROMPT TEXT IN SQLITE DUMP:", ("TOP_SECRET_DO_NOT_RECORD" in db_dump or "CONFIDENTIAL_USER_MSG" in db_dump)) print("JSONL TRACE KEYS PER CALL :", [list(o.keys()) for o in trace_objs]) print("BODY KEY PRESENT IN TRACE :", any("body" in o for o in trace_objs)) print("PROMPT TEXT IN TRACE FILE :", ("TOP_SECRET_DO_NOT_RECORD" in trace_text or "CONFIDENTIAL_USER_MSG" in trace_text)) print("TRACE LINE 2 JSON :", json.dumps(trace_objs[1])) '
SQLITE LEDGER ROWS: [{"id": 1, "session_id": "sess-check7", "call_index": 1, "ts_start": 1792721519.956407, "ts_end": 1792721519.9564512, "provider": "anthropic", "model": "claude-3-5-sonnet-20241022", "mode": "safe", "read_tokens": 0, "write_tokens": 0, "uncached_tokens": 979, "output_tokens": 50, "cost_usd": "0.00368700", "breaks_count": 1}, {"id": 2, "session_id": "sess-check7", "call_index": 2, "ts_start": 1792721519.9583628, "ts_end": 1792721519.9583921, "provider": "anthropic", "model": "claude-3-5-sonnet-20241022", "mode": "safe", "read_tokens": 0, "write_tokens": 0, "uncached_tokens": 995, "output_tokens": 50, "cost_usd": "0.00373500", "breaks_count": 2}]
PROMPT TEXT IN SQLITE DUMP: False
JSONL TRACE KEYS PER CALL : [['v', 'session', 'call', 'ts_start', 'ts_first_token', 'ts_end', 'provider', 'api', 'model', 'mode', 'params', 'fingerprints', 'usage', 'breaks', 'changes'], ['v', 'session', 'call', 'ts_start', 'ts_first_token', 'ts_end', 'provider', 'api', 'model', 'mode', 'params', 'fingerprints', 'usage', 'breaks', 'changes']]
BODY KEY PRESENT IN TRACE : False
PROMPT TEXT IN TRACE FILE : False
TRACE LINE 2 JSON         : {"v": 1, "session": "sess-check7", "call": 2, "ts_start": 1792721519.9583628, "ts_first_token": null, "ts_end": 1792721519.9583921, "provider": "anthropic", "api": "messages", "model": "claude-3-5-sonnet-20241022", "mode": "safe", "params": {}, "fingerprints": {"tools": [], "system": [], "messages": []}, "usage": {"raw": {"cache_creation_input_tokens": 0, "cache_read_input_tokens": 0, "input_tokens": 995, "output_tokens": 50}, "read": 0, "write": 0, "uncached": 995, "output": 50, "cost_usd": "0.00373500"}, "breaks": [{"cause": "tools_reordered", "user_caused": true, "level": "tools", "details": "Restored first-seen tool order in safe mode"}, {"cause": "below_minimum", "user_caused": false, "level": "system", "details": "Context tokens (992) below provider minimum (2048)"}], "changes": [{"kind": "tool_order_restored", "first_seen_order": ["alpha", "beta"], "reordered_order": ["beta", "alpha"], "details": "Restored first-seen tool order to protect prefix cache"}]}
PASS Check 8 · CLI hearth report & hearth diagnose on Check 7 Artifacts
UTC: 2026-10-23T02:12:01Z Exit Code: 0
$ HEARTH_HOME=/tmp/hearth_eval/check7 .venv/bin/hearth report --session sess-check7 --ledger /tmp/hearth_eval/check7/ledger.sqlite && .venv/bin/hearth diagnose /tmp/hearth_eval/check7/traces/sess-check7.jsonl
Session Report: sess-check7
calls 2 · input 1,974 tok · read 0 (0.0%) · written 0 · uncached 1,974 · output 100
total cost: $0.00742200 USD

transformations (1):
  call 2    tool_order_restored    Restored first-seen tool order to protect prefix cache

prefix breaks (3):
  call 1    below_minimum        Context tokens (967) below provider minimum (2048)
  call 2    tools_reordered      Restored first-seen tool order in safe mode
  call 2    below_minimum        Context tokens (992) below provider minimum (2048)

Diagnosing trace: sess-check7.jsonl (session 'sess-check7') · read share 0.0%
  call 1: level system · cause below_minimum
    detail: Context tokens (967) below provider minimum (2048)
    note: prefix below provider caching threshold
  call 2: level tools · cause tools_reordered
    detail: Restored first-seen tool order in safe mode
    fix: mode='safe' will restore first-seen tool order automatically
  call 2: level system · cause below_minimum
    detail: Context tokens (992) below provider minimum (2048)
    note: prefix below provider caching threshold
PASS Check 9 · CLI hearth replay <trace.jsonl> --provider gemini --model gemini-2.5-flash
UTC: 2026-10-23T02:13:21Z Exit Code: 0
$ .venv/bin/hearth replay /tmp/hearth_eval/check7/traces/sess-check7.jsonl --provider gemini --model gemini-2.5-flash && .venv/bin/hearth replay /tmp/hearth_eval/check9_recorded.jsonl --provider gemini --model gemini-2.5-flash
--- Replay on Check 7 trace (record=False, no request body) ---
mode      read     written    uncached   input cost   user-caused breaks
observe   0%       0%         0%         1.00×        0
safe      0%       0%         0%         1.00×        0
optimize  0%       0%         0%         1.00×        0
--- Replay on recorded trace (with request.kwargs) ---
mode      read     written    uncached   input cost   user-caused breaks
observe   0%       100%       0%         1.00×        0
safe      0%       100%       0%         1.00×        0
optimize  66%      34%        0%         0.44×        0
PASS Check 10 · Benchmark Verifiers (--m0, --m4, --m6) & docs/REPORT.md Inspection (SIMULATOR)
UTC: 2026-10-23T02:12:04Z Exit Code: 0

Inspection of docs/REPORT.md (lines 22–31): Every framework row (Custom Loop, Pydantic Ai, Openai Agents Sdk, Langgraph) has the exact same read share (0.0% Stock, 48.9% Optimize) and the exact same cost ($0.0156 Stock / $0.0093 Optimize on gemini-2.5-flash; $0.3201 Stock / $0.1854 Optimize on gpt-4o). Per evaluation rules, the 42.0% overall cost reduction figure is labeled SIMULATOR, not a provider bill.

$ .venv/bin/python -m hearth.bench.runner --m0 --verify && .venv/bin/python -m hearth.bench.runner --m4 --verify && .venv/bin/python -m hearth.bench.runner --m6 --verify
Running M0 Kill Test: 20 tasks × 2 templates × 4 arms × 3 runs...

{
  "typical": {
    "A": { "runs": 60, "read_share_pct": 0.0, "avg_cost_usd": 0.0007416, "ttft_p50_s": 1.35, "pass_rate_pct": 100.0, "tasks_passed_avg": 20.0 },
    "B": { "runs": 60, "read_share_pct": 49.5, "avg_cost_usd": 0.0004188, "ttft_p50_s": 1.35, "pass_rate_pct": 100.0, "tasks_passed_avg": 20.0, "cost_reduction_vs_a_pct": 43.5 },
    "C": { "runs": 60, "read_share_pct": 0.0, "avg_cost_usd": 0.0007416, "ttft_p50_s": 1.35, "pass_rate_pct": 100.0, "tasks_passed_avg": 20.0, "cost_reduction_vs_a_pct": 0.0 },
    "D": { "runs": 60, "read_share_pct": 49.4, "avg_cost_usd": 0.000419, "ttft_p50_s": 1.35, "pass_rate_pct": 100.0, "tasks_passed_avg": 20.0, "cost_reduction_vs_a_pct": 43.5 }
  },
  "minimal": {
    "A": { "runs": 60, "read_share_pct": 49.5, "avg_cost_usd": 0.0004149, "ttft_p50_s": 1.35, "pass_rate_pct": 100.0, "tasks_passed_avg": 20.0 },
    "B": { "runs": 60, "read_share_pct": 49.5, "avg_cost_usd": 0.0004149, "ttft_p50_s": 1.35, "pass_rate_pct": 100.0, "tasks_passed_avg": 20.0, "cost_reduction_vs_a_pct": 0.0 },
    "C": { "runs": 60, "read_share_pct": 49.5, "avg_cost_usd": 0.0004149, "ttft_p50_s": 1.35, "pass_rate_pct": 100.0, "tasks_passed_avg": 20.0, "cost_reduction_vs_a_pct": 0.0 },
    "D": { "runs": 60, "read_share_pct": 49.5, "avg_cost_usd": 0.0004149, "ttft_p50_s": 1.35, "pass_rate_pct": 100.0, "tasks_passed_avg": 20.0, "cost_reduction_vs_a_pct": 0.0 }
  },
  "simulator_calibration_holdout_mae_points": 0.76,
  "decision": "GO (M0 Kill Test Passed)"
}
[VERIFICATION PASSED] M0 Kill Test Decision: GO

{
  "milestone": "M4",
  "gate_verified": true,
  "overall_savings_pct": 42.0,
  "calibration_holdout_mae_points": 0.0,
  "report_matrix": {
    "gemini": {
      "custom_loop": { "tasks_tested": 20, "stock_read_share_pct": 0.0, "stock_cost_usd": 0.015606, "stock_pass_rate_pct": 100.0, "opt_read_share_pct": 48.9, "opt_cost_usd": 0.00928, "opt_pass_rate_pct": 100.0, "cost_saving_pct": 40.5 },
      "pydantic_ai": { "tasks_tested": 20, "stock_read_share_pct": 0.0, "stock_cost_usd": 0.015606, "stock_pass_rate_pct": 100.0, "opt_read_share_pct": 48.9, "opt_cost_usd": 0.00928, "opt_pass_rate_pct": 100.0, "cost_saving_pct": 40.5 },
      "openai_agents_sdk": { "tasks_tested": 20, "stock_read_share_pct": 0.0, "stock_cost_usd": 0.015606, "stock_pass_rate_pct": 100.0, "opt_read_share_pct": 48.9, "opt_cost_usd": 0.00928, "opt_pass_rate_pct": 100.0, "cost_saving_pct": 40.5 },
      "langgraph": { "tasks_tested": 20, "stock_read_share_pct": 0.0, "stock_cost_usd": 0.015606, "stock_pass_rate_pct": 100.0, "opt_read_share_pct": 48.9, "opt_cost_usd": 0.00928, "opt_pass_rate_pct": 100.0, "cost_saving_pct": 40.5 }
    },
    "openai": {
      "custom_loop": { "tasks_tested": 20, "stock_read_share_pct": 0.0, "stock_cost_usd": 0.320125, "stock_pass_rate_pct": 100.0, "opt_read_share_pct": 48.9, "opt_cost_usd": 0.185397, "opt_pass_rate_pct": 100.0, "cost_saving_pct": 42.1 },
      "pydantic_ai": { "tasks_tested": 20, "stock_read_share_pct": 0.0, "stock_cost_usd": 0.320125, "stock_pass_rate_pct": 100.0, "opt_read_share_pct": 48.9, "opt_cost_usd": 0.185397, "opt_pass_rate_pct": 100.0, "cost_saving_pct": 42.1 },
      "openai_agents_sdk": { "tasks_tested": 20, "stock_read_share_pct": 0.0, "stock_cost_usd": 0.320125, "stock_pass_rate_pct": 100.0, "opt_read_share_pct": 48.9, "opt_cost_usd": 0.185397, "opt_pass_rate_pct": 100.0, "cost_saving_pct": 42.1 },
      "langgraph": { "tasks_tested": 20, "stock_read_share_pct": 0.0, "stock_cost_usd": 0.320125, "stock_pass_rate_pct": 100.0, "opt_read_share_pct": 48.9, "opt_cost_usd": 0.185397, "opt_pass_rate_pct": 100.0, "cost_saving_pct": 42.1 }
    }
  },
  "cause_counts": { "volatile_in_prefix": 160 },
  "report_file": "/Users/jishmisc_pro/Documents/hearth/docs/REPORT.md"
}
[VERIFICATION PASSED] M4 Benchmark Gate: VERIFIED

{
  "milestone": "M6",
  "gate_verified": true,
  "tasks_tested": 5,
  "tasks_passed": 5,
  "pass_rate_pct": 100.0,
  "overall_savings_pct": 18.1,
  "total_stock_cost_usd": "0.115335",
  "total_hearth_cost_usd": "0.094474",
  "results": {
    "mixed_heavy_reasoning": { "title": "Complex analysis with straightforward validation sub-calls", "verdict": "switch", "stock_cost_usd": "0.003690", "hearth_cost_usd": "0.001845", "savings_pct": 50.0, "passed": true },
    "mixed_delegate_subtask": { "title": "Codebase analysis with small-scope extraction delegation", "verdict": "delegate", "stock_cost_usd": "0.004185", "hearth_cost_usd": "0.001444", "savings_pct": 65.5, "passed": true },
    "mixed_ttl_pause": { "title": "Multi-turn human-in-the-loop task with 10-minute pause", "verdict": "switch", "stock_cost_usd": "0.101400", "hearth_cost_usd": "0.089400", "savings_pct": 11.8, "passed": true },
    "mixed_short_stay": { "title": "Short task below break-even threshold avoiding cold write penalty", "verdict": "stay", "stock_cost_usd": "0.003315", "hearth_cost_usd": "0.000615", "savings_pct": 81.4, "passed": true },
    "mixed_escalation": { "title": "Distilled context escalation to primary model on failure", "verdict": "delegate", "stock_cost_usd": "0.002745", "hearth_cost_usd": "0.001170", "savings_pct": 57.4, "passed": true }
  }
}
[VERIFICATION PASSED] M6 Benchmark Gate: VERIFIED
[Note: JSON nested object lines compacted above after 80 lines of pretty-printed stdout; all keys and numbers are verbatim.]

Group B — v0.1.0 Release Gate (Checks 11–17, BUILD_HANDBOOK.md §12)

FAIL Check 11 · Version is 0.1.0, Tag v0.1.0/0.1.0 Exists, git status Clean, HEAD == origin/main
UTC: 2026-10-23T02:05:59Z Exit Code: 0 (Inspection: FAIL)
$ grep -n '^version' pyproject.toml && grep -n '__version__' src/hearth/__init__.py && git tag -l && git status --short && git status && echo "HEAD=$(git rev-parse HEAD) origin/main=$(git rev-parse origin/main)"
7:version = "0.1.0b1"
14:__version__ = "0.1.0b1"
0.1.0a1
?? hearth-synthesis.html
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	hearth-synthesis.html

nothing added to commit but untracked files present (use "git add" to track)
HEAD=c97122d704414a00f118f93cfb5242aa5073d9f1 origin/main=c97122d704414a00f118f93cfb5242aa5073d9f1
FAIL Check 12 · GitHub Actions Workflow Exists and Runs make check
UTC: 2026-10-23T02:06:20Z Exit Code: 1
$ ls -la .github .github/workflows
ls: .github: No such file or directory
ls: .github/workflows: No such file or directory
FAIL Check 13 · docs/BENCH.md Has ≥ 1 Live Provider Spend-Log Row within $25 Budget
UTC: 2026-10-23T02:06:20Z Exit Code: 0 (Inspection: FAIL)
$ cat docs/BENCH.md
# Benchmark Pre-Registration & API Spend Tracker (M0)

## 1. Pre-Registration (Fixed before any paid runs)

- **Date:** 2026-10-19
- **Milestone:** M0 (Kill Test & Simulator Calibration)
- **Models under test:**
  - `gemini-2.5-flash` (Google GenAI SDK — implicit + explicit caching)
  - `gpt-4o-mini` / `gpt-4o` (OpenAI SDK — automatic prefix caching ≥1,024 tokens)
- **Templates:**
  1. `typical`: Standard multi-turn agent loop with dynamic runtime metadata in system prompt (current timestamp, session UUID) and un-ordered tool dict serialization.
  2. `minimal`: Hand-cleaned prompt layout with static prefix and dynamic turn context in user messages.
- **Four Experimental Arms:**
  - **Arm A (`stock`)**: Unmodified agent template (`observe` mode).
  - **Arm B (`hearth-safe`)**: `hearth.wrap(client, mode="safe")` — deterministic tool order restoration + cache breakpoints, zero content modification.
  - **Arm C (`naive-compact`)**: Summarizes conversation every 5 turns.
  - **Arm D (`hearth-optimize`)**: `hearth.wrap(client, mode="optimize")` — relocates volatile system fields to turn tail, offloads large tool outputs, and protects prefix.
- **Task Suite:** 20 multi-turn software engineering & tool-use tasks (8–20 turns each, 3 runs per cell).

---

## 2. API Budget & Spend Log (D-02)

- **Approved Build Budget:** $25.00 USD (M0 + nightly contract tests)
- **Approved GTM Budget:** $30.00 USD
- **Total Spent to Date:** $0.00

| Date | Milestone / Run | Provider | Model | Input Tokens | Cached Read Tokens | Output Tokens | Cost ($ USD) | Cumulative ($ USD) | Notes |
|---|---|---|---|---:|---:|---:|---:|---:|---|
FAIL Check 14 · Contract Tests Have Evidence of Five Successful Nightly Runs
UTC: 2026-10-23T02:06:20Z Exit Code: 0 (Inspection: FAIL — not run, would spend money)
$ grep -rn "contract" Makefile tests/ docs/
Makefile:6:.PHONY: check test contract lint format typecheck clean
Makefile:9:	uv run pytest -m "not contract" tests/
Makefile:15:contract:
Makefile:16:	uv run pytest -m "contract" tests/
tests/test_contract.py:19:pytestmark = pytest.mark.contract
docs/STATUS.md:30:- **Gate commands:** `make check`, `pytest -m "not contract"`
docs/BUILD_HANDBOOK.md:578:`@pytest.mark.contract` runs nightly on a small budget, not on every commit.
docs/BUILD_HANDBOOK.md:629:- [ ] All invariant tests green; contract tests green for 5 consecutive nights.

REASON: FAIL — not run, would spend money (and zero evidence of 5 nightly runs exists in repo logs or CI workflows).
FAIL Check 15 · gtm/TRIALS.md Exists and Contains Three Outsider Install Notes
UTC: 2026-10-23T02:06:20Z Exit Code: 1
$ ls -la gtm/TRIALS.md
ls: gtm/TRIALS.md: No such file or directory
PASS Check 16 · src/hearth/data/providers.toml source_date Values Within 14 Days of Today
UTC: 2026-10-23T02:06:20Z Exit Code: 0
$ grep -n "source_date" src/hearth/data/providers.toml
8:source_date = "2026-10-19"
42:source_date = "2026-10-19"
75:source_date = "2026-10-19"
FAIL Check 17 · PyPI Trusted Publishing Workflow Configured & https://pypi.org/pypi/hearth-ai/json Check
UTC: 2026-10-23T02:06:20Z Exit Code: 0 (Inspection: FAIL — not published)
$ ls -la .github/workflows/ 2>&1; curl -s -w "\nHTTP_STATUS:%{http_code}\n" https://pypi.org/pypi/hearth-ai/json
ls: .github/workflows/: No such file or directory
{"message": "Not Found"}
HTTP_STATUS:404

Recorded Facts (Non-Scored Observations)

1. Default Mode of hearth.wrap vs. Handbook

In src/hearth/__init__.py:93–99, the default argument of hearth.wrap is mode: Mode = "observe".
In docs/BUILD_HANDBOOK.md:151 (§3.3) and docs/BUILD_HANDBOOK.md:455 (§6.1), the handbook specifies the product default as mode: Literal["observe", "safe", "optimize"] = "safe".

2. Core Dependencies in pyproject.toml

In pyproject.toml:25–27, core dependencies are declared as:
dependencies = ["tomli>=1.1.0;python_version<'3.11'"].
On Python 3.11+ (where tomllib is in the standard library), Hearth installs zero third-party packages.

3. Status of .github/workflows/

Neither .github/ nor .github/workflows/ exists in the repository (ls: .github/workflows: No such file or directory).

What would have to be true to tag v0.1.0

Failed Item Current State Single Next Action
Check 11 Version is 0.1.0b1 in pyproject.toml and src/hearth/__init__.py; only tag 0.1.0a1 exists; untracked HTML files in working tree. Remove untracked proof/synthesis HTML files, bump version to 0.1.0 in pyproject.toml and src/hearth/__init__.py, commit, push to origin/main, and create annotated tag v0.1.0.
Check 12 .github/workflows/ does not exist. Create .github/workflows/ci.yml running make check on push and pull_request across supported Python versions.
Check 13 docs/BENCH.md has $0.00 spent and zero rows in the Spend Log table. Run a live provider benchmark within the approved $25 budget and record the dated token totals and USD cost row in docs/BENCH.md.
Check 14 No nightly contract test schedule or log of 5 consecutive passing nightly runs exists. Add a scheduled nightly GitHub Actions workflow running make contract against live provider keys and accumulate 5 consecutive green nightly runs.
Check 15 gtm/TRIALS.md does not exist. Conduct 3 external developer trial installs of 0.1.0b1 and record their anonymized install notes and outcomes in gtm/TRIALS.md.
Check 17 No PyPI trusted publishing workflow exists and https://pypi.org/pypi/hearth-ai/json returns HTTP 404. Add .github/workflows/release.yml configured with OIDC trusted publishing (`environment: pypi`) and trigger it on the v0.1.0 tag to publish hearth-ai to PyPI.