Concurrency audit and “disk I/O error” fix

Scope: src/kiss/core, src/kiss/agents/sorcar, src/kiss/server, src/kiss/agents/vscode, src/kiss/tests/agents/third_party_agents. Goal: find and fix race conditions, hangs, deadlocks and redundancies; then (added mid-task) fix the recurring sqlite3.OperationalError: disk I/O error.

1. How the work was organised

Audit and fix (claude-fable-5-1). Eight workers with disjoint file ownership grepped every concurrency primitive in their files, read each implicated region and its call sites, and fixed only defects they could explain with a concrete interleaving. Every fix was preceded by an end-to-end test (real threads, subprocesses, sockets, git repositories; no mocks).
Independent read-only review (gpt-5.6-sol). Four reviewers checked the diff for introduced bugs and incomplete wiring, ran each new test three times, and looked for issues the first pass missed. Reviewer spend was about $25 in total (well under the 50% cap).
Fix the review findings (claude-fable-5-1). Four workers verified each finding before touching code, reproduced it with a test, and applied the minimal fix. Findings that could not be substantiated were left alone and are listed in section 5.
Root-cause the disk I/O error (section 4), then the full test suite, uv run check --full.

2. What was found and fixed

Severity is the reviewer’s or worker’s assessment of production impact. Every row has a regression test (test_concaudit_* files under src/kiss/tests/ or src/kiss/agents/vscode/test/).

2.1 Hangs

Where	Problem	Fix
`server/web_server.py` cloudflared stderr drain	After the 30 s URL wait timed out, the only thread draining cloudflared’s `stderr=PIPE` exited. Both callers keep the process alive, so the 64 KiB pipe filled, cloudflared blocked inside its Go logging mutex, its metrics endpoint stopped answering and the watchdog could never restart it.	The drain thread now runs until EOF (process exit). If the thread cannot be started, the process is killed. The pipe is decoded with `errors="replace"` so one bad byte cannot kill the drain.
`core/models/model.py` `_CLIProcess`	Strict UTF-8 decoding of the claude/codex CLI pipes: one undecodable byte raised inside the drain thread, which treated it as “pipe closed”. stderr then filled and the child blocked; on stdout the rest of the turn was silently dropped.	`errors="replace"` on both pipes.
`agents/sorcar/useful_tools.py` Bash tool	When the shell exited but a background child kept the stdout pipe open, a user Stop changed nothing: the stop monitor saw the shell already gone and returned, and the tool blocked for the full timeout (300 s default; measured 25 s in the test).	`_consume_stream` wakes on the stop event; the tool returns the output collected so far. Helper threads are now started inside the cleanup `try`, so a failed start still kills the process group.
`agents/sorcar/sorcar_agent.py` `run_tasks_parallel`	Children were submitted in a list comprehension outside the guarded region; a stop injected during submission left `futures` empty and `shutdown(wait=True)` waited for untracked children.	Submissions are appended one by one inside the same `try`; the existing abandon path now sees them.
`server/web_server.py` startup lock	The UDS sidecar `flock(LOCK_EX)` was unbounded and uncancellable; a wedged sibling daemon stalled startup forever.	`_flock_with_deadline` polls `LOCK_NB`; on expiry the daemon starts WSS-only like any other UDS bind failure.
`server/web_server.py` UDS send path	`writer.drain()` had no timeout; a client that stopped reading held the endpoint lock and every later broadcast queued another future (199 pending in 0.5 s in the reviewer’s reproduction).	30 s drain timeout; the endpoint is removed and closed on expiry.
`server/web_server.py` `stop_async`	`_stop_tunnel()` blocked the event loop in `Popen.wait(5)` when cloudflared ignored SIGTERM (5.01 s loop stall measured).	Offloaded with `asyncio.to_thread`.
`agents/vscode/src/DependencyInstaller.ts`	Synchronous `Atomics.wait` sleeps and synchronous `lsof`/`systemctl`/`python --version`/`git` spawns froze the extension host (3058 ms measured during a daemon restart); installs ran with no timeout.	Async sleeps and awaited spawns with the same deadlines; 30-minute ceiling on install steps with process-group kill; `sleepSync` removed.
`agents/vscode/src/SorcarSidebarView.ts`	With an empty `kissSorcar.defaultModel` the constructor ran `uv run python …` synchronously (15 s timeout) on activation and on every new panel.	Starts from a provisional value; the model is resolved asynchronously (single-flight) and adopted only if the user has not picked one.
`agents/vscode/src/AgentClient.ts` `dispose()`	`socket.end()` waits for the write buffer to drain; with a non-reading daemon and a large buffered command Node never exited (reproduced with 64 MiB).	`socket.destroy()`.
`agents/vscode/media/main.js`	`createImageBitmap`/`toBlob`/`FileReader` had no deadline; a hung conversion latched `awaitingAttachments` and suppressed every send.	60 s deadline routed to the existing visible error path.
`tests/agents/third_party_agents`	A Gmail OAuth test abandoned a thread blocked in `run_local_server` for the rest of the pytest process; non-daemon threads joined without deadlines; `urlopen`/`git` without timeouts.	The test now completes the OAuth redirect itself; daemon threads with bounded joins and liveness asserts; timeouts added.

2.2 Race conditions

Where	Problem	Fix
`agents/sorcar/persistence.py`	Deleted `-wal`/`-shm` under a live process → every new connection fails with `disk I/O error` (section 4).	Detect by inode identity or `SQLITE_IOERR`; checkpoint; close every connection to that database; reconnect.
`server/web_server.py` tunnel watchdog vs `stop_async`	Cancelling the watchdog task did not cancel the executor running `_start_tunnel`; a fresh live cloudflared was published after shutdown (reproduced: alive 2 s after stop).	`_tunnel_lock` plus a one-way `_tunnel_stopped` flag; a late spawn is killed under the lock.
`server/merge_flow.py`, `server/server.py`	Several worktree mutations set `is_merging` but never published `merge_thread`, so shutdown’s `_await_active_merges` could return while git or worktree deletion was still running.	`threading.current_thread()` is published with every mutating claim and cleared in the same `finally`.
`agents/sorcar/sorcar_agent.py` `reclaim_abandoned_subagents`	Returned `True` from a pre-wait snapshot while a child registered during the wait was still running; callers then deleted a worktree it was writing to.	The result is computed from the live list under the lock.
`agents/sorcar/worktree_sorcar_agent.py`, `server/task_runner.py`	Merge warnings were read under `_warning_lock`, the lock released, and the combination written back: a concurrent flush delivered the old warning twice, a concurrent set was lost. Two call sites, plus the failed-broadcast restore path.	New `WorktreeSorcarAgent.add_warning(text, prepend=)` does read-combine-write under one lock hold; all three sites use it.
`server/web_server.py` `_tab_worktree_dirs`	A stale fan-out broadcast from an old task could restore the old worktree path for a tab that was concurrently rebound.	`_record_tab_worktree_dir` re-checks the subscription under the printer lock before writing.
`core/models/model.py` `_ToolCallFilteredStream.__exit__`	The flush could raise (the production token callback raises `KeyboardInterrupt` on Stop) before the model’s callbacks were restored, leaving a reused adapter bound to a dead filter and treating later turns as tool-bearing.	Restore in a `finally`.
`core/models/openai_compatible_model2.py`	A transport failure mid-reasoning left the printer in thinking mode for the retry.	Thinking bracket closed in `finally`, matching the other transports.
`agents/vscode/src/extension.ts`	Timer callbacks checked module globals only before `await`; deactivation during the await produced an unhandled rejection (`reading 'widenToOneThird'`).	Capture the controller, re-check after every await, catch.
Thread-start failures	`Thread.start()` raising (thread exhaustion) left permanent wedges: a per-tab commit-message claim never released, a task shown as “running” forever, a dead autocomplete worker published, a stop that never armed its watchdog, a stranded SIGTERM latch, a live-usage monitor leaked.	Each site releases its claim, emits the terminal status, or falls back inline; tested with a real `RLIMIT_NPROC=1` child.
Third-party agent tests	bind(0)/close/reuse port selection (TOCTOU) in five “unreachable server” tests.	`refusing_port` fixture keeps a bound, non-listening socket alive for the test.

2.3 Deadlocks

No live deadlock was found. One latent trap was removed: core/vscode_config.py had an unused wrapper that took _config_lock and the api-keys flock; had a locked caller used it, the re-entrant RLock would pass but the per-file-description flock would block forever. Lock orders verified consistent: STATE_LOCK → printer._lock, repo_lock → _reclaim_process_lock, _journal_lock → flock → _rw_lock.write → _caches_lock, _config_lock → store flock → RC flock.

2.4 Redundancies removed

_attachments_to_content_parts duplicated in both OpenAI transports → one implementation in OpenAICompatibleBase.
Anthropic client and connection pool rebuilt on every initialize() → reused unless the key or workspace changes.
Redundant post-os.replace chmod in atomic_write_text.
Dead _edit_api_keys_env_file wrapper; dead _free_port helper; sleepSync.
Tests: _isolated_kiss_home fixture duplicated in five files → one conftest.py fixture; seven copies of server shutdown boilerplate → a context manager; two near-identical raw HTTP helpers merged.

3. Reviewer-caught regression, reverted

The first pass consolidated four copies of “detach the executor, bank its usage” in relentless_agent.py into one finally. The reviewer showed that an asynchronous KeyboardInterrupt (the server’s force-stop injection) landing between the two statements inside the finally would now lose the banking, whereas the original except BaseException handler covered that window. The file was reverted to its original form; the redundancy stays.

4. The “disk I/O error”

4.1 Evidence

The running daemon (PID 2514112) held sixteen descriptors to ~/.kiss/sorcar.db-wal (deleted) and sorcar.db-shm (deleted), and was still writing to the deleted WAL (6.5 MB of frames). No -wal/-shm existed on disk. A trivial run_parallel from this task failed with Unhandled exception: disk I/O error; older log entries show the same error at PRAGMA journal_mode=WAL and BEGIN IMMEDIATE, tasks failing within 100 ms of starting, and events lost across a restart.

4.2 Mechanism

SQLite keeps one -shm mapping per process per database inode, shared by every connection, but opens the -wal by name for each connection. After something unlinks the sidecars, old connections keep working on deleted inodes while any new connection in the same process pairs the old index with a new, empty WAL.

Reproduced deterministically with plain sqlite3 (see test_plain_sqlite_reproduces_the_failure_mode): unlink the sidecars while one connection is open, open a second connection in the same process → OperationalError: disk I/O error, code 522 (SQLITE_IOERR_SHORT_READ). A separate process sees a database without the frames still in the deleted WAL (in the test it does not even see the schema), and creates its own sidecars, so two WALs now diverge over one main file.

4.3 Fix (`src/kiss/agents/sorcar/persistence.py`)

_open_conns: every connection _get_db opens, with its owning thread and database path. Connections of dead threads are closed and dropped explicitly (_prune_dead_thread_conns) instead of being left to the garbage collector.
_attached_shm: per database path, the (st_dev, st_ino) of the -shm the process is mapped to, recorded by the first live connection.
_sidecars_orphaned(): on every _get_db(), compare the -shm on disk with the recorded identity (missing or replaced → orphaned). The database file is re-stat’ed afterwards so a whole-database deletion (the test fixtures’ redirect/delete pattern) is not mistaken for an orphaned mapping.
_recover_orphaned_sidecars(): PRAGMA wal_checkpoint(PASSIVE) on a live connection (folds the frames committed into the deleted WAL into the main file), then interrupt and close every connection to that database so SQLite drops the dead mapping, bump the generation so every thread reconnects.
_open_db_connection(): a fresh connection failing with SQLITE_IOERR while others are open triggers the same recovery and one retry (covers the stat→open race).

Tests: src/kiss/tests/agents/sorcar/test_concaudit_wal_sidecar_orphan_recovery.py (six tests: new thread after unlink, same thread, sidecars recreated by another process, event writer, forced SQLITE_IOERR path, raw SQLite documentation). Before the fix: disk I/O error; after: all commits, including those made against the deleted WAL, are visible in-process and to a separate process.

Two things to know. (1) The process that unlinked the sidecars was not identified. The daemon code does not unlink them (that path was removed after the 2026-08-15 corruption); scripts/sync-task-db.sh only does so on the remote after stopping the web app; no cron or timer does. The fix makes the daemon self-heal whatever the cause. (2) The currently running daemon has the old code and cannot open new connections until it is restarted with the new build; its shutdown should be allowed to complete (not SIGKILLed by the failsafe) so the frames still in its deleted WAL are checkpointed.

5. Reviewer findings not acted on (with reason)

ConsolePrinter.reset() clears _mid_line without a newline — no production caller of reset().
persistence._stop_event_writer two-stopper livelock — its only production caller is the single-shot atexit hook.
commands._cmd_generate_commit_message except BaseException — runs on an executor thread that never receives a non-Exception BaseException.
_free_port() bind/close/reuse in five backend tests — needs the backends (out of scope) to accept port=0.
OpenAI client connect timeout (30 min pre-watchdog window on a provider that never sends headers) — design decision left to the owner.

6. Verification

Full suite, 9029 tests in 30 parallel splits: 9027 passed on the first run. One failure was a regression in my persistence change (generation stamped after the open; reintroduced the known A4 race) — fixed and re-run. The other, test_live_remote_panels_match_extension, is a Playwright page-load timeout under 30-way load and passes in isolation.
Each new test file was run three times by the reviewers or the fix workers; none flaked.
uv run check --full: uv sync, API docs, compileall, ruff, mypy, pyright, VS Code typecheck and lint all pass.
VS Code extension: all 310 test files pass (run in six slices).

Because the running daemon could not spawn sub-agents (the very bug of section 4), the full-suite splits were run as background pytest processes rather than through run_parallel.