Service Resilience and Credential Topology

Status:revised after plan review
Date:2026-09-04
Branch:jack_20260904_service_resilience_credential_topology
Type:spec

Goal

After a reboot, an upgrade, or a Claude Code auto-update, the jacked service comes up on its own on macOS, Linux, and Windows: the tray icon appears, the menu-bar pill shows the active account's usage, the statusline names the runtime account, and no macOS password prompt ever names python3.14. Three defects block that today and this spec fixes all three at the source.

Incident evidence

SymptomRoot causeWhere
No tray icon after reboot Cold-disk startup took over the flat 10 s readiness budget; the tray aborted with SystemExit(0); launchd's KeepAlive.SuccessfulExit=false treats exit 0 as final and never retried. jacked/service/tray.py _wait_for_ready, _abort_unready_start
jacked upgrade left nothing running ("old ownership did not exit") macOS quits through NSApp.terminate_, which ends the process without unwinding Python, so the finally in ServiceRunner.run() never removed the instance manifest. The handoff only watches the manifest, saw the dead pid still listed, timed out, and exited 1 without starting the new generation. jacked/service/menubar_mac.py _check_stop; jacked/service/handoff.py _wait_for_handoff_exit
Pill shows "—"; statusline "runtime unknown" The capability registry certifies one exact executable hash (Claude 2.1.259, darwin arm64). Claude 2.1.260 does not match, so every consumer sees UNSUPPORTED. Linux and Windows have never matched at all since 0.99.0. jacked/credentials/runtime.py _SHIPPED_CAPABILITIES, resolve_active_identity
Password prompts naming python3.14 0.99.0 replaced the security CLI with in-process Security.framework calls. The Keychain item is created by Claude Code through security, so security is on its access list and python3.14 is not. Each Python build is a new identity, so the prompt returns after every uv Python upgrade. jacked/credentials/macos_store.py PyObjCSecurityBackend

Design

A. Service lifecycle resilience (all supervisors)

A1 Readiness budget. The cold-start readiness wait becomes a constant COLD_START_READY_TIMEOUT = 90.0 seconds. The wait returns as soon as the port answers, so a warm start is not slowed. The wait also returns False immediately if the uvicorn thread has died, instead of polling a dead server for the full budget. The ready log line reports elapsed seconds so cold-boot timing is visible in the log.

A2 Retryable failure. When the service runs under a native supervisor (launchd, systemd user, Task Scheduler) and never becomes ready, it exits with status 75 (EX_TEMPFAIL) so the supervisor's restart-on-failure policy retries. Under the manual supervisor it keeps the existing human-readable SystemExit message. The deliberate clean exits that yield to another live instance (legacy listener, lost admission) stay exit 0 so supervisors do not fight a working service. The systemd unit gains RestartSec=5 plus StartLimitIntervalSec=300 and StartLimitBurst=5 so retries are bounded rather than throttled off after ten seconds. launchd (SuccessfulExit=false) and Task Scheduler (RestartOnFailure PT1M x3) already retry non-zero exits. launchd has no burst limit, so a small breaker file (start-failures.json under the service directory) counts failed supervised starts; after five inside ten minutes the service logs "giving up" and exits 0, which every supervisor treats as final, and a successful start clears the file. Timing constants live in one place (jacked/service/__init__.py): the cold-start budget (90 s), the handoff exit wait (30 s) and the replacement readiness wait (105 s, derived from the cold-start budget) so no caller can wait less than the service itself needs.

A3 Deterministic ownership teardown. ServiceRunner gets an idempotent release_ownership() that closes the control server and the ownership manifest. run() calls it in its finally as today, and the macOS menu-bar _shutdown() calls it before rumps.quit_application(). Windows and Linux already unwind through run().

A4 Handoff tolerates a dead owner. _wait_for_handoff_exit treats a manifest whose recorded process is dead or identity-mismatched as an exit, exactly like a removed manifest, and lets activation proceed (the new instance clears the stale manifest on acquire, as it already does). One primitive, process_is_stale(process) in jacked/service/instance_storage.py, becomes the only definition of "proven dead" (a probe that times out proves nothing and returns False); manifest_is_proven_stale(path) wraps it, and cli.py, instance_ownership.py and the handoff all use it. The handoff gets two budgets: 30 s for the old owner to exit and 105 s for the replacement to report ready, the latter derived from the cold-start constant. The CLI's restart failure message stops claiming that nothing was signalled and points at jacked service status before recover.

sequenceDiagram
    participant CLI as jacked service restart
    participant Old as old service
    participant Sup as supervisor
    participant New as new service
    CLI->>Old: RESTART_HANDOFF (control socket)
    Old->>Old: shutdown uvicorn, release_ownership()
    Old-->>Sup: exit 0
    CLI->>CLI: manifest gone OR pid proven dead
    CLI->>Sup: install_owned_supervisor (kickstart)
    Sup->>New: start
    New->>New: clear stale manifest, bind, ready (<= 90 s)
    CLI->>New: STATUS until generation matches
  
sequenceDiagram: CLI sends RESTART_HANDOFF; old releases ownership and exits 0; CLI proceeds when manifest is gone or pid is dead; supervisor kickstarts new generation; CLI polls STATUS.
  

B. Credential topology certification (all platforms)

B1 Registry keyed by topology, not bytes. CapabilityRegistry records become CapabilityRecord(platform_system, config_mode, min_build, inspected_through, capability). resolve(identity) matches on platform and config mode, requires build_version >= min_build, and substitutes the observed ExecutableIdentity into the returned capability so the executable hash and version travel as provenance. Evidence strings: build:2.1.260, inspected-through:2.1.260, and build-newer-than-inspected when the observed build is above the inspected version. The mutation kill switch is unchanged. The newer-than-inspected marker is not decorative: on such a build the transaction engine refuses to create a missing authority (a store that moved looks identical to one that was never written) and logs by name any claudeAiOauth keys Claude wrote that jacked's payload would drop.

B2 Shipped records. One record per platform, global config mode, min_build = 2.1.0, inspected_through = 2.1.260:

Scoped config mode (CLAUDE_CONFIG_DIR) has no record and stays unsupported, as today.

B3 Platform-neutral store wiring. runtime.py builds stores from the capability's declarations through one build_stores(capability, home) helper (macos-keychain to MacOSCredentialStore, global-credential-file to FileCredentialStore). The two sys.platform != "darwin" fail-closed branches are removed; the registry is the only platform gate. Three guards travel with the widened certification: FileCredentialStore.write() refuses with CONCURRENT_WRITE when the file changed since that adapter's last read(), including a file that appeared after a MISSING read, checked both before staging and immediately before the atomic replace (Claude Code refreshes tokens in place), and the engine re-reads the authority and records that refusal as FAILED_PRESERVED with the identity now in the store (the macOS post-write readback mismatch becomes ERROR so the two cases stay distinct); a required mirror that refuses is re-read and written once more, since a mirror is a copy of the authority; a payload without the _jackedAccountId stamp resolves to UNUSABLE with the evidence identity:stamp-absent instead of a bare failure; and a scoped launch (CLAUDE_CONFIG_DIR) activates the global authority only where that authority is the Keychain, because on file-authority platforms Claude reads the scoped file and touching the global one would change every other session's account; on the skip path the launch reads the scoped file back through the strict store, refuses the launch if it does not match what was written, and publishes the read-back identity with launch:global-authority:skipped and a fresh launch: revision so the child environment and the statusline agree. session_observer.py uses the same locator constants so the two resolvers share one convention.

B4 Identity detection cache. detect_claude_identity caches the ExecutableIdentity keyed by resolved path, size, and mtime so the 200 MB binary is hashed once per build rather than on every poll from every consumer.

B5 Conflict rendering. When the snapshot state is conflict with the evidence desired-default:conflict (and not an organization conflict) and an observed identity exists, the statusline segment renders observed-email · desired desired-email with email set to the observed account, so the user sees what the runtime is actually using. Other failure states render as today. The menu-bar summary already shows the observed account once the resolver resolves; no change there.

C. Keychain access through the Apple-signed tool

C0 Why the tool and not the framework. Verified on this Mac: /usr/bin/security reads the Claude Code item in 20 ms with no dialog; reading an item whose access list lacks the tool blocks until the child is killed; the tool has no "never prompt" flag; a Keychain lock-status query through SecKeychainGetStatus takes 3 ms, touches no item and never prompts. The design therefore keeps the framework only for that read-only status probe and moves every item access to the tool.

C1 SecurityCliBackend. A new NativeSecurityBackend implementation drives /usr/bin/security: read is find-generic-password -a ACCOUNT -s SERVICE -w on argv (no secret there); update and add run security -i and send add-generic-password -U -a "ACCOUNT" -s "SERVICE" -X HEX on stdin, so the token is never a process argument (hex is Claude Code's own write format, not a confidentiality control). Failures map by exit status first (44 missing, 36 interaction required, 128 canceled) and by stderr text only as a fallback; reasons come from a closed vocabulary and raw stderr goes to the DEBUG log. Output is read as bytes. Non-interactive calls use a 2 s subprocess timeout, strictly below the store's 3 s thread timeout so exactly one mechanism owns the timeout; the child is killed on expiry. Interactive calls get 60 s.

C1b Prompt-free guards in the store. Before any background call the store runs the lock-status probe and returns INTERACTIVE_REQUIRED without spawning when the login keychain is locked. The timed-out latch stays process-wide, keyed by Keychain locator (stores are rebuilt on every resolution, so instance state would never survive a poll), gains a 10 minute cooldown instead of lasting for the process lifetime, covers background writes as well as reads, and is cleared by a successful foreground call, so a one-time "Always Allow" for security heals everything. The lock probe runs on the bounded worker thread and swallows the framework's own error class. Two tool quirks are handled: security -i splits lines longer than 4095 bytes, so an upsert is sent as hex when it fits and as escaped ASCII JSON when that fits; an item too large for either (many MCP tokens) fails closed with a reason naming the size, and only the explicit JACKED_KEYCHAIN_ARGV_FALLBACK=1 opt-in allows a process-argument write, because argv is readable by other local users through ps and by endpoint agents; write-path stderr and timeout exceptions are never logged because they can carry the payload; find-generic-password -w prints hex for any non-ASCII payload, so reads decode an all-hex answer.

C2 Single identity. SecurityCliBackend is the default backend. PyObjCSecurityBackend is deleted so no credential path can ever present python3.14 to the Keychain; the only remaining framework use is the item-free lock-status probe. The NativeSecurityBackend protocol and the store's bounded-read and readback logic are unchanged, so the existing fake-backend tests keep passing.

D. Dashboard follows the chained Claude Code token flow (appended 2026-09-04)

A re-auth is two server flows: the primary sign-in, then the chained Claude Code token flow the server auto-starts when the primary completes. The dashboard's runOAuthFlow polls only the flow it started and refreshes once at that completion, which is exactly when the second window opens, so the account card never refreshes after the token lands even though the server stores it and marks the account valid. The poller now follows cc_flow_id from a completed primary flow, refreshes again when the chained flow completes, and on a chained failure refreshes and says the account itself was re-authenticated.

E. Observation resolves from the authority (appended 2026-09-04 from Task 10 evidence)

The read-side resolver demanded consensus between the Keychain authority and the file mirror before naming an identity. Claude Code refreshes tokens in the Keychain only, and a jacked switch that stalls at the Keychain step can leave the file holding another account, so the two legitimately drift between jacked writes; every drift blanked the menu-bar pill with CONFLICT even after the build was certified (observed live: Keychain account 4, file account 5). Observation now takes the authority's identity and reports each required mirror as ok, divergent, or missing in evidence. Write verification in the transaction engine keeps demanding consensus; the resolver's default stays consensus so no other caller changes.

Acceptance criteria

  1. Given a supervised cold start whose server needs 45 s, when the tray waits for readiness, then it succeeds and logs the elapsed time (A1).
  2. Given the uvicorn thread dies before the port answers, then the readiness wait returns within one poll interval rather than the full budget (A1).
  3. Given a never-ready start under launchd, systemd, or Task Scheduler, then the process exits with status 75; under the manual supervisor it raises the existing message (A2).
  4. 3b. Given five failed supervised starts inside ten minutes, then the next abort logs "giving up" and exits 0; given a successful start on either platform path, then _note_ready runs and the breaker file is removed; future-dated stamps are ignored (A2).
  5. Given the rendered systemd unit, then it contains RestartSec=5, StartLimitIntervalSec=300, StartLimitBurst=5 and the launchd plist is unchanged (A2).
  6. Given the macOS stop path runs, then release_ownership() is called before rumps.quit_application() and a second call is a no-op (A3).
  7. Given a handoff where the old manifest names a dead pid, then _wait_for_handoff_exit returns None and activation is attempted; a live pid still yields "old ownership did not exit" after the timeout (A4).
  8. Given _manifest_is_proven_stale in cli.py, then it delegates to jacked.service.instance.manifest_is_proven_stale, and instance_ownership.py uses the same process_is_stale rule; a probe that times out is not proven stale (A4).
  9. 7b. Given the handoff defaults, then the exit wait is 30 s and the readiness wait is 105 s, both from jacked.service constants, and every CLI readiness wait (start, autostart install, restart, recover) uses the same 105 s (A4).
  10. Given Claude 2.1.260 with any hash on darwin global, then the registry resolves the keychain topology with can_mutate=True and evidence carrying the build and the newer-than-inspected marker (B1, B2).
  11. Given linux global or windows global, then the registry resolves the file topology; given scoped mode or build 2.0.9, then unsupported; given the kill switch, then unsupported until a fresh resolution (B1, B2).
  12. Given a linux platform, then resolve_active_identity builds a resolver over the credential file and returns its observation instead of "certified platform adapter unavailable"; given a linux activation, then the credential file is written end to end with mode 0600 and the account stamp (B3).
  13. 10b. Given a credential file without the _jackedAccountId stamp, then the observation is UNUSABLE with evidence identity:stamp-absent; given a file that changed after the store read it, then write() returns CONCURRENT_WRITE and the file is untouched (B3).
  14. 10c. Given a build newer than inspected and a missing authority, then activation is UNUSABLE and nothing is written; given the authority carries an unknown claudeAiOauth key, then a warning names it; given a scoped launch on a file-authority platform, then the global authority is not activated (B1, B3).
  15. Given two calls to detect_claude_identity with an unchanged binary, then the file is hashed once; given a changed mtime, then it is hashed again (B4).
  16. 11b. Given the version probe times out, then detect_claude_identity raises OSError; tests never see another test's cached identity (B4).
  17. Given a conflict snapshot with observed and desired identities, then the statusline segment names the observed email and the desired email and email equals the observed one (B5).
  18. Given SecurityCliBackend with a fake runner, then read maps exit 44 to MISSING, 36 to INTERACTIVE_REQUIRED, 128 to DENIED, success to OK with the payload bytes; update and add run security -i with the upsert command on stdin and no secret in argv; a hung non-interactive call is reported as ERROR; raw stderr never appears in a reason (C1).
  19. 13b. Given a locked login keychain, then a background read returns INTERACTIVE_REQUIRED without spawning the tool (C1b).
  20. 13c. Given a background read that timed out, then a background write on a new store instance for the same locator is refused without a backend call, the latch expires after the cooldown, and a successful foreground call clears it; given an upsert whose hex form exceeds 4095 bytes, then escaped JSON is sent on stdin; given one that fits neither, then the write is UNUSABLE unless the argv opt-in is set; given hex output from a read, then it is decoded (C1b).
  21. Given the credentials package, then no module calls SecItemCopyMatching, SecItemAdd, SecItemUpdate or imports LocalAuthentication (C2).
  22. 16. Given a completed primary re-auth flow whose status carries cc_flow_id, then the dashboard keeps polling that flow and refreshes the accounts view again when it completes, with a banner naming the Claude Code token; given the chained flow fails, then the view is refreshed and the banner names the failure while stating the account itself was re-authenticated; given a completed flow without cc_flow_id, then behaviour is unchanged (D).
  23. 17. Given the authority reads a stamped payload and a required mirror reads a different payload or is missing, when jacked observes the active identity, then the observation is RESOLVED with the authority's identity and evidence names the mirror as divergent or missing; given the authority is missing, unusable, or unstamped, then the existing results are unchanged; given the resolver's default, then consensus is still required (E).
  24. Given the real Mac after install, then the service resolves the Keychain identity without a prompt, the pill shows the observed account's usage, the statusline names the runtime account, and a kickstart after a forced stop brings the icon back (Phase 6 evidence).

Non-goals

Files and interfaces

FileChange
jacked/service/__init__.py, jacked/service/start_failures.pyShared timing constants; start-failure breaker.
jacked/service/tray.py_wait_for_ready fails fast on a dead thread; _abort_unready_start exits 75 under native supervisors behind the breaker; new release_ownership().
jacked/service/menubar_mac.py_shutdown() calls release_ownership() before quitting.
jacked/service/handoff.pyDead-pid detection in _wait_for_handoff_exit; default timeout 30 s.
jacked/service/instance_storage.py, instance.py, instance_ownership.pyOne process_is_stale rule and its manifest_is_proven_stale wrapper.
jacked/cli.py_manifest_is_proven_stale delegates to the shared helper.
jacked/service/supervisors/__init__.pysystemd unit restart limits.
jacked/credentials/capabilities.pyCapabilityRecord, topology matching, version comparison, evidence.
jacked/credentials/runtime.pyShipped topology records; build_stores; platform gates removed; identity cache; scoped-launch rule.
jacked/credentials/transaction.py, resolver.py, file_store.py, jacked/api/session_observer.py, jacked/launch.pyMissing-authority rule, schema-drift warning, stamp-absent evidence, file compare-and-swap, shared locators, scoped-launch gate.
jacked/credentials/macos_store.pySecurityCliBackend default; lock probe; cooling latch; PyObjC item access removed.
jacked/statusline_account.pyConflict rendering with observed identity.
docs/architecture/oauth-and-credential-flows.md, docs/architecture/auto-swap-system.mdReplace the exact-hash description with topology records.
tests/unit/...New and updated tests per acceptance criterion.

Generated with the jacked HTML artifact template.