# v0.5 real-machine re-verify

*Date: 2026-04-20 (same-day addendum to [`v0.5.md`](./v0.5.md)).*
*Runner: [`scripts/verify_v0_5.sh`](../../scripts/verify_v0_5.sh). Profile: `verify-gpt-oss` (openai_compat-only, added in this pass).*

This doc is the live-traffic counterpart to the unit-test / httpx-mock coverage
already banked in v0.5-A / v0.5-B / v0.5-C. The v0.5 retrospective's
"Follow-ons" section scheduled it before v0.6 starts, because v0.4's
real-machine day was the single most valuable hour of v0.4 and the bet is
that v0.5 will pay off the same way.

## What's being verified

The `capability-degraded` log line is the public contract of the v0.5 gate:
a greppable message name with a varying `reason` field. Unit tests exercise
the contract ~20 times, but exclusively against httpx-mock. This pass drives
the same contract with real upstream calls, against real providers, and
captures the actual log payloads as evidence.

Four scenarios (three gates; D added in v0.5.1 to close the streaming gap):

| Scenario | Gate | Endpoint | Expected `reason` | Proves |
|---|---|---|---|---|
| A-thinking | v0.5-A | `/v1/messages` with `thinking: {type: enabled, ...}` | `provider-does-not-support` | Request-side strip + log fires when a non-capable openai_compat provider receives a thinking block. HTTP should still succeed (strip → normal call). |
| B-cache-control | v0.5-B | `/v1/messages` with `system[0].cache_control: {type: ephemeral}` | `translation-lossy` | Observability-only gate fires on the engine loop without reordering the chain. HTTP should succeed (cache_control silently dropped in translation). The `system` text is 1200+ tokens so the request is cache-meaningful had anthropic seen it. |
| C-reasoning-strip | v0.5-C | `/v1/chat/completions` against `openai/gpt-oss-120b:free` | `non-standard-field` | Response-side strip removes the non-standard `reasoning` field that gpt-oss:free emits alongside `content`, and logs once. Response body must **not** contain `choices[*].message.reasoning`. |
| D-reasoning-stream | v0.5-C (stream) | `/v1/chat/completions` with `stream: true` against `openai/gpt-oss-120b:free` | `non-standard-field` **exactly once** over the whole stream | Same strip as C but on the SSE path, plus the "log once per stream" dedup (`reasoning_logged` flag in `openai_compat.py`). Asserts (a) HTTP 2xx, (b) capability-degraded line count == 1 for the stream, (c) no `choices[*].delta.reasoning` survives in any chunk. Added v0.5.1 — unit tests covered the dedup, live wasn't. |

Using the `verify-gpt-oss` profile (just `openrouter-gpt-oss-free` — see
[`examples/providers.yaml`](../../examples/providers.yaml)) is deliberate:
it's a profile with zero providers that are capable for any of these three
features, so every gate reliably fires. Ordinary production profiles
(`claude-code-direct` etc.) keep `anthropic-direct` in the chain precisely
so thinking and cache_control are preserved natively; those profiles would
NOT trigger the thinking strip or the cache_control log on the anthropic
provider (which is correct behavior, but not what we're verifying here).

## How to run

Two terminals, two commands. The runner writes artifacts under
`/tmp/coderouter-verify/` by default.

```bash
# Terminal 1 — start the server, stderr (JSON logs) redirected to a file
coderouter serve --config examples/providers.yaml --port 4000 \
  2> /tmp/coderouter-verify.log

# Terminal 2 — fire the 4 scenarios
bash scripts/verify_v0_5.sh
```

Requires `OPENROUTER_API_KEY` in the server's environment (gpt-oss:free is
used for all four scenarios). Exit code 0 iff every scenario passed.

The script also writes `/tmp/coderouter-verify/report.md` — a prefilled
markdown block that can be pasted into the **Evidence** section below
after a successful run.

## Evidence

*Paste the contents of `/tmp/coderouter-verify/report.md` below after running
the script. Until then, this section is intentionally blank.*

<!-- BEGIN: verify report (paste from /tmp/coderouter-verify/report.md) -->

### Run — 2026-04-20 14:33 JST

Runner: `scripts/verify_v0_5.sh`. Profile: `verify-gpt-oss` (openai_compat-only).
Upstream provider: `openrouter-gpt-oss-free` → `openai/gpt-oss-120b:free`.

#### Summary

| Scenario | Gate | Endpoint | Expected `reason` | HTTP | Result |
|---|---|---|---|---|---|
| A-thinking | v0.5-A | `/v1/messages` + `thinking` | `provider-does-not-support` | 200 | PASS |
| B-cache-control | v0.5-B | `/v1/messages` + `cache_control` | `translation-lossy` | 200 | PASS |
| C-reasoning-strip | v0.5-C | `/v1/chat/completions` | `non-standard-field` | 200 | PASS |

Overall: **PASS**. Every gate fired with the expected `reason` + a 2xx from the upstream.

#### A-thinking — PASS

HTTP 200. Observed `capability-degraded` lines:

```json
{"ts": "2026-04-20T14:33:04", "level": "INFO", "logger": "coderouter.routing.fallback", "msg": "capability-degraded", "provider": "openrouter-gpt-oss-free", "dropped": ["thinking"], "reason": "provider-does-not-support"}
{"ts": "2026-04-20T14:33:06", "level": "INFO", "logger": "coderouter.adapters.openai_compat", "msg": "capability-degraded", "provider": "openrouter-gpt-oss-free", "dropped": ["reasoning"], "reason": "non-standard-field"}
```

#### B-cache-control — PASS

HTTP 200. Observed `capability-degraded` lines:

```json
{"ts": "2026-04-20T14:33:06", "level": "INFO", "logger": "coderouter.routing.fallback", "msg": "capability-degraded", "provider": "openrouter-gpt-oss-free", "dropped": ["cache_control"], "reason": "translation-lossy"}
{"ts": "2026-04-20T14:33:17", "level": "INFO", "logger": "coderouter.adapters.openai_compat", "msg": "capability-degraded", "provider": "openrouter-gpt-oss-free", "dropped": ["reasoning"], "reason": "non-standard-field"}
```

#### C-reasoning-strip — PASS

HTTP 200. Observed `capability-degraded` lines:

```json
{"ts": "2026-04-20T14:33:19", "level": "INFO", "logger": "coderouter.adapters.openai_compat", "msg": "capability-degraded", "provider": "openrouter-gpt-oss-free", "dropped": ["reasoning"], "reason": "non-standard-field"}
```

Response body assertion: `choices[*].message.reasoning` **not present** — strip effective.

<!-- END: verify report -->

## What to look for when reading the evidence

- Each scenario's `capability-degraded` line should have a consistent shape:
  `{msg: "capability-degraded", provider: "openrouter-gpt-oss-free",
  dropped: ["<field>"], reason: "<scenario-specific>"}`. Any drift in field
  names between scenarios is a finding (and would argue for promoting the
  payload to a TypedDict, per retro Follow-ons).
- Scenario A and B should fire their log **exactly once per call**. Scenario
  C should fire **once per stream** in the streaming case (not tested in this
  script's non-streaming scenarios, but the unit-test suite covers it; can be
  extended here later with a streaming call if needed).
- HTTP status on all three must be 2xx. Thinking / cache_control being
  stripped means the downstream request is plain Chat Completions, which
  gpt-oss:free accepts normally.
- Scenario C's response must not contain `choices[*].message.reasoning` —
  the script asserts this automatically and flips overall_pass to FAIL if
  the field survives. Seeing it survive would mean `_strip_reasoning_field`
  didn't engage (capability flag accidentally on, or the code path skipped).

## Failure modes worth distinguishing

If the runner reports FAIL, the artifact tree at `/tmp/coderouter-verify/<scenario>/`
has enough to triage without re-running:

- `request.json`: exact body sent. Confirm it's the shape the gate expects.
- `response.json`: upstream body. A 401 means `OPENROUTER_API_KEY` isn't
  reaching the server. A 429 means gpt-oss hit its ~20 req/min rate limit —
  wait a minute and re-run.
- `server-log-slice.jsonl`: every log line during the call. A missing
  `capability-degraded` line with a present `provider-ok` line means the
  gate didn't engage — inspect the gate's predicate to see why (e.g., the
  heuristic table may have promoted gpt-oss to "capable" unintentionally).
- `capability-degraded.jsonl`: filtered view. Empty iff the gate didn't fire.

## What this run proved (beyond the bare PASS)

- **Request-side and response-side gates compose cleanly on the same call.**
  Scenarios A and B each fired **two** `capability-degraded` lines — one from
  `coderouter.routing.fallback` on the outbound path (thinking / cache_control),
  and one from `coderouter.adapters.openai_compat` on the inbound path
  (`reasoning`). This is expected (gpt-oss:free emits `reasoning` on every
  response, so any call through it triggers the v0.5-C strip), but seeing the
  two logger names / two reasons in the same request trace is the first live
  evidence that v0.5-A/B + v0.5-C are independently routable and do not
  interfere. The retro §"gate design" table predicted this (request-side vs
  response-side being orthogonal axes); the evidence now backs it.
- **v0.4-D's log enrichment was re-validated under stress.** The first attempt
  of this run mis-read the env and OpenRouter returned 401 with body
  `{"error":{"message":"No cookie auth credentials found","code":401}}`. The
  `provider-failed` log line surfaced that body verbatim in its `error` field,
  and diagnosis took under a minute. v0.4-D paid off twice: once on the day it
  shipped, once here. Strong argument for continuing to invest in structured
  log shape over feature breadth.
- **`verify-gpt-oss` profile is now a permanent live-sanity fixture.** Any
  future capability-related change can be re-smoked in two terminals and under
  a minute. Worth keeping in `examples/providers.yaml` indefinitely (documented
  as verify-only so it doesn't get used accidentally in production chains).

## Follow-ons (from this run)

- ~~**`provider-failed` → retry-or-surface policy for 401.**~~ **Landed v0.5.1**
  as `chain-uniform-auth-failure`. When every attempt in the chain returned the
  same non-retryable 401 or 403, the engine now emits a warn with
  `hint: probable-misconfig` alongside the existing `provider-failed` trail.
  Auth-only (401/403) by design — a non-retryable 400 ("model not found")
  reflects a different class of config mismatch and stays quiet. The
  `NoProvidersAvailableError` exception shape is unchanged; the hint is a
  pure log addition. See `coderouter/routing/fallback.py` ::
  `_warn_if_uniform_auth_failure` and `tests/test_fallback_misconfig_warn.py`.
- ~~**Streaming verify.**~~ **Landed v0.5.1** as scenario `D-reasoning-stream`.
  Runner now covers both the strip (no `delta.reasoning` in any SSE chunk) and
  the dedup contract (`capability-degraded` line count == 1 per stream). See
  the table at the top of this doc and `scripts/verify_v0_5.sh`.

---

Full narrative of v0.5 itself: [`v0.5.md`](./v0.5.md).
Per-sub-release detail: `CHANGELOG.md` sections `[v0.5-A]` / `[v0.5-B]` / `[v0.5-C]`.
