docs/research · field study · 2026-09-05

hal0 × ODS Field Study

A source-level comparison of Osmantic's ODS and hal0, a hands-on install of ODS, all 106 open hal0 issues triaged against ODS's code, and a ranked plan for what to copy, what to build, and what to leave alone.

ODS Osmantic/ODS@21f4b3a (v2.6.0 line, main) hal0 hal0ai/hal0@108b366 method 9 domain reports · every claim cited to file:line · reports attached in full install sandbox, CPU tier 0, Docker 29.3.1, rootless install user

Verdict

The premise was half right, and the half that was wrong matters more. ODS is ahead where the product meets the operator: installer structure and failure forensics, the two-file extension model, auth as a product feature, a phone-first agent portal, persona craft, AMD host tuning, consequence-first copy, bootstrap-then-hot-swap, and a fleet-verified model catalog. hal0 is ahead in the engine room: update integrity, agent provisioning rigor, MCP, memory architecture, the container runtime, diagnosis modelling, the multi-slot inference model, the CLI's architecture, and, surprisingly, mDNS.

That split is good news. Almost everything worth taking from ODS is additive: shapes and discipline layered onto mechanisms hal0 already has. Very little of it is a rewrite, and the few rewrites on offer (compose, the monolithic CLI, the git-pull updater) would be regressions.

ODS leadshal0 leadsbar = my read of the balance per domain, from the attached reports
Agents (Hermes)split
ODS: auth, phone portal, persona, OAuth. hal0: provisioning, context floor, sandboxing, plugins, tests.
ExtensionsODS
Two files, fourteen consumers auto-attach. hal0 has the ingredients and none of the wiring.
Install & reliabilitysplit
ODS: structure, forensics, bootstrap. hal0: cosign updater, typed diagnosis, deeper preflight.
Modelssplit
ODS: verified catalog, rollback with re-proof, stable alias. hal0: multi-slot, registry, GGUF parsing, live GTT fit.
Network & mDNSsplit
ODS: one friendly URL, safer default bind, onboarding. hal0: lighter mDNS, more rigorous (opt-in) auth design.
MCP & memoryhal0
ODS ships no MCP. hal0 has two servers, a registry, REST, CLI, a default-deny policy, and Hindsight.
Dashboard & copyODS
ODS explains consequences and next steps. hal0 explains internals; its next-step plumbing exists but is empty.
Container runtimehal0
Quadlets, systemd backoff, digest pins beat compose plus tag pins. ODS wins only on AMD host tuning.
CLIhal0
A modular Typer tree versus one 270 KB Bash file. ODS is more feature-complete in a few verbs.

Three premise corrections

ODS ships no MCP at all.

No mcp_servers block in the Hermes template, no MCP for OpenCode or Open WebUI, no MCP UI or CLI verb. The n8n "RAG pipelines" are sticky-note stubs. There is nothing to port on this axis. Meanwhile hal0 already has most of the "user-added MCP servers" feature you asked for: a per-server TOML registry at /etc/hal0/mcp-servers/, a 945-line REST router, hal0 mcp list|install|uninstall, and an SSRF-guarded manifest resolver. Three gaps remain: no process supervisor (the route literally says "pending ADR-0015"), no join between installed servers and the Hermes config, and no install drawer in the dashboard. Update 2026-10-04: two of the three closed on main — #2253 (ADR-0015) shipped the exposure join, [secrets]/[tools]/[exposure] and the CLI/REST verbs for http/sse servers; the stdio supervisor is deferred and the drawer is #2245. Review also found that the shipped join mirrors the tool policy without enforcing it on Hermes's call path — see report 06 §E.3 and hal0 issue #2303.

hal0 already has mDNS.

src/hal0/services/mdns.py writes avahi service-group XML, is wired to a dashboard card, and is tested. ODS runs its own zeroconf daemon plus a Caddy host-router on :80 to get chat.ods.local. hal0's approach is lighter for what it does. What hal0 lacks is not mDNS, it is the single friendly URL and a safe LAN posture, an address publisher if subdomain URLs are wanted (avahi service files advertise _http._tcp records on the machine's hostname; they do not create chat.<host>.local A-records, which is exactly what ODS's bin/ods-mdns.py exists for), and one dead path: the base hal0.service avahi file the dashboard promises is never written.

ODS's Hermes is packaging; the product around it is the prize.

ODS runs the pinned upstream container with five files of glue and a regex YAML patcher for upgrades. hal0's provisioner is more rigorous (hermes config migrate + config set, converging writes, a backup, an operator overrides file), its context-floor logic reads the floor from the installed Hermes instead of hard-coding 65536, and its systemd sandbox is tighter than the container. What ODS does better is everything an operator touches: magic-link auth with owner cards, a mobile chat portal whose pooled WebSocket bridge keeps the prompt cache warm, agent-driven OAuth setup, and a persona written from live testing. Keep pip-installed Hermes. Copy the product layer.

Two findings that should not wait for a plan

A. Approval endpoints execute gated tools with auth removed.

/api/agent/approvals/{id}/approve runs model_pull, slot_delete and config_write and its module explicitly notes auth was removed (src/hal0/api/routes/approvals.py:24-27), on an API that binds 0.0.0.0:8080 by default. ODS's session_signer.py (171 lines, stdlib only) plus its verify-session/admin-session routes are a near-verbatim fix, and hal0 already has half the machinery in api/agents/_auth.py. About 250 lines. Do this before anything else in this document. Update 2026-10-04: addressed on main for the LAN-bound case by #2246 (946c512) — ADMIN routes, approvals included, now require an admin session/key from any non-loopback caller when the box is LAN-bound, even with require_auth off. The loopback dev-open path remains by design.

B. hal0-Max recommends a model ODS found broken on unified memory.

hal0's top first-run bundle for the 100 GB Strix Halo class sets chat.coder = Qwen3-Coder-Next-80B-A3B (curated.py:260-275). ODS's fleet testing found that exact family produces all-? output on unified-memory backends, reproduced on Strix Halo and on Grace-Blackwell, and now hard-routes every unified-memory host away from it (installers/lib/tier-map.sh:100-138, scripts/select-model.py:326-372). ODS tested against llama.cpp b9014 and its own Lemonade path, so this may not reproduce on hal0's ROCm runner. It costs one afternoon on a real 395 to find out and it decides whether the flagship pick is a correctness bug.

A third, smaller item: the sandbox install surfaced a real ODS bug (the host agent inherits the installer's lock file descriptor in the non-systemd fallback path, so later model operations block forever). Details in the install section. Since permission flows both ways, it is worth sending upstream.

Where hal0 falls short

The honest list, in the order I would fix it. Each item names the evidence; each maps to a row in the port ledger.

  1. Mutation endpoints run unauthenticated on a LAN-open bind. Approvals execute gated tools; auth is off by default and the doctor's own posture check prints "PASS (dev/loopback)" without reading the bind address (cli/doctor_all.py:62-83, issue #1822).
  2. The installer is one 4,002-line script with no persisted log. No step files, no tee'd transcript, no failure report, no summary JSON, no dry run, no per-step contracts. ODS has all six; that is why it can unit-test detection and tier mapping and hal0 cannot.
  3. There is no extension model. The services registry is a hard-coded four-entry tuple, hal0 app supports exactly openwebui, and the UI already lists n8n with an icon and no backend. ODS gets a working service from two files and no code edits.
  4. Remediation is designed but empty. Diagnosis.next_steps and a panel that renders them exist, yet health_report.to_diagnosis() hard-codes next_steps=[] for every check. Issues #1845, #1870, #1994 and #2108 are all this gap wearing different hats.
  5. Long privileged calls are unbounded and silent. ContainerProvider._run defaults timeout=None and the load path passes nothing; the CLI blocks up to ~2,400 s with no output. ODS bounds every hang-prone call and prints while it waits.
  6. Hardware truth goes stale. A missing or old hardware.json books VRAM as RAM; the ROCm lane depends on rocm-smi being installed rather than on /dev/kfd existing; seed slots copy context_size = 65536 onto every box. ODS rebuilds its capability profile every run and derives every number from it.
  7. No AMD host tuning. GTT/TTM sizing, swappiness, and a tuned profile are absent from hal0's installer even though hal0's own upstream toolbox README documents them as required for Strix Halo. Update 2026-10-04: the IOMMU half of that README (amd_iommu=off) is not for hal0 to port — it removes /dev/accel (the XDNA NPU) and hal0's own guidance is iommu=pt amd_iommu=on (drivers.mdx:102, ADR-0003). The gap is GTT/TTM, sysctl and tuned — and on kernels ≥ 6.14 even GTT pinning is an opt-in floor, not a default: amdgpu sizes the pool dynamically and hal0 reads the live value per request (drivers.mdx:86-97), see D7.
  8. Open WebUI is half wired. hal0 sets chat and voice, but not RAG embeddings, web search or image generation, so Open WebUI's own upload and image buttons point at nothing. ODS wires all of it, including a baked ComfyUI workflow.
  9. The copy explains internals, not consequences. Tooltips read like hal0.toml [memory] · requires restart. ODS's status vocabulary and settings hints lead with what happens to you. This is an audience choice, but it should be a choice.
  10. Own docs have drifted. The "15-phase, checkpointed" provisioner is 12 steps and deliberately uncheckpointed; BUNDLED_AGENTS is misquoted; a unit's Documentation= path does not exist; two byte-identical 629-line memory providers have no sync; CLAUDE.md says docs/adr/ holds the decision records while ARCHITECTURE.md says hal0 keeps no ADR tree, and the ADR numbers the code cites (0012, 0015, 0023) live in the gitignored internal tree per CHANGELOG.md, so a contributor reading routes/mcp.py cannot find the decision it defers to. Fix these before anyone ports against them. (ADR bookkeeping reconciled on main by #2244: docs/adr/ now holds 0004, 0012, 0013, 0015, 0020 and 0023.)

How ODS does it, domain by domain

Mechanism first, then what to copy and what to skip. The full reports with every citation are in the appendices; this is the distilled version.

Agents and Hermes

splitreport 01

ODS runs nousresearch/hermes-agent:v2026.6.5 as a container that exposes no host port. A Caddy sidecar (hermes-proxy) fronts it with forward_auth to dashboard-api/api/auth/verify-session, which HMAC-verifies a stateless cookie of the form <id>.<expiry>.<sig>. Cookies come from magic links (single-use guest, revoke-only owner cards, hashed at rest, rate-limited, QR-rendered) or from POST /api/auth/admin-session so the owner is never locked out. Config is a bind-mounted cli-config.yaml template that encodes hard-won settings: a schema-level disabled_toolsets: [terminal, browser] because small models reach for a shell on weather questions; max_tokens: 1024 so a looping model cannot pin the only slot; compression tuned to 0.75 / 0.50 / 40 after a single web-search result kept triggering it. The persona (SOUL.md) gets a generated "About this installation" block built from docker ps and a live model probe. A per-cookie WebSocket pool in hermes_bridge.py keeps one Hermes session per phone so llama-server's KV cache stays warm across messages, with exactly one retry, allowed only before prompt.submit lands.

hal0 runs Hermes as a pip-installed wheel under a sandboxed hal0-agent@.service (Type=notify, watchdog, ProtectSystem=strict), provisions it with converging writes through Hermes's own config migrate and config set, reads the 64K context floor from the installed package instead of hard-coding it, injects a lean STATE.md refreshed on slot swaps, and ships real Hermes plugins for provider discovery and Hindsight memory. Its policy layer is on the call path; ODS's APE policy engine is attached to nothing.

Copy

  • session_signer.py + routers/auth.py → generalize api/agents/_auth.py; gate approvals first.
  • The three non-obvious Caddyfile lines: explicit route {} wrapper, strip Sec-Websocket-*/Upgrade/Connection from the auth sub-request, redir * /auth/required 303.
  • oauth_passthrough.py + oauth-providers.json + the SOUL script that teaches the agent to drive OAuth setup.
  • The pooled-bridge discipline (same socket for create and submit; retry only before submit) if hal0 ever adds a REST/SSE chat surface.
  • Tool-name → spinner caption table with the honest fallback Using `<name>`….
  • The generated-config-writers matrix (docs/INSTALLER-ARCHITECTURE.md:118-135) as a section in ARCHITECTURE.md.
  • The persona voice rules as a persona option, not a replacement for the admin SOUL.

Extensions

ODSreport 02

Your description is accurate. A working ODS extension is extensions/services/<id>/manifest.yaml (20 to 30 lines) plus compose.yaml (30 to 50 lines) and zero edits anywhere else; searxng ships as 22 + 33 lines. From those two files, fourteen consumers pick it up: the bash service registry, the compose resolver, ods list|enable|disable|logs|shell, the generic health check, the dashboard's services and features loaders, sidebar quicklinks, feature cards, setup hooks, CI audits, the catalog generator, presets, and the host agent. Four independent walkers each re-read the directory, all gated on schema_version: ods.services.v1; that redundancy is why it feels magical. GPU and mode overlays (compose.{nvidia,amd,local}.yaml) are picked up by filename probe, not by manifest declaration. Pre-wiring is four conventions: Docker DNS equals the compose service name, the LLM is the gateway alias ods/current, ports bind loopback with an env override, data lives under ./data/<id>/.

Two honest seams: a hand-maintained 21-line ENABLE_* table in phase 03 (the file's own comment says omission is a bug), and enable/disable implemented as renaming compose.yaml to .disabled, which makes the content tree the state store. hal0 has every ingredient (a typed launch plan, a validated quadlet writer, a services registry with the right fields, a portable envelope format, a port authority, a registry-driven Services page) and none of the wiring. The hard constraint is the root-side allow-list in installer/wrappers/hal0-systemctl:389-516, which pins units to the hal0-slot@ prefix by design.

Copy (translated to podman/systemd)

  • Manifest schema → Pydantic manifest.toml with schema_version = "hal0.ext.v1"; keep field names that carry meaning so ODS manifests convert mechanically.
  • The registry loader (service-registry.sh is already a Python program embedded in bash) → hal0/extensions/registry.py.
  • calculate_feature_status and its vocabulary enabled | available | insufficient_vram | services_needed.
  • audit-extensions.py checks → hal0 app doctor.
  • The heavily commented template and the swap-safe doc; they are a large part of why the model is usable.
  • Render: manifest.toml → RuntimeLaunchPlan → _render_quadlet_from_plan → hal0-ext@<id>.container; env written to /etc/hal0/extensions/<id>.env with OPENAI_BASE_URL, OPENAI_MODEL=hal0/brain, and slot:<name> references resolved through the port authority.

Install and reliability

splitreport 03

install-core.sh is a 363-line orchestrator over 14 phase files and 19 pure libraries, every file carrying a Purpose / Expects / Provides / Modder notes header, with a contracts document giving each phase Owns / Inputs / Outputs / Idempotency / Failure modes and a written reinstall contract. The narrator helpers (ai, ai_ok, ai_warn, signal) all tee to a log file; on compose failure a redacted install-report-*.txt captures the failed image, port occupancy with owning PID, and the compose config, and ods doctor mines it later. Image pulls retry with a backoff table, validate after pull, and bail immediately on non-retryable errors. A fixture-driven preflight engine emits JSON ({id, status, message, action}) and six named scenarios pin its verdicts in CI. The signature move is bootstrap mode: a 1.2 GiB pinned model boots the stack in about two minutes while a detached upgrader downloads the real model with script-owned resume, snapshots every file it will touch (recording absence with a .missing sentinel), force-recreates, asserts the container's --model argument actually changed, and rolls back through a proven completion on any of four failure paths, deleting the small model only after HOT_SWAP_VERIFIED=true.

hal0 is ahead on the parts that protect a fleet: a cosign-verified, staged, atomic-symlink updater with venv rollback (ODS does git pull), typed Diagnosis/Evidence/NextStep with a frozen ID taxonomy, and a preflight that classifies LXC privilege and GPU gid problems ODS's threshold checker never sees. hal0 already has a support bundle, a QR code and a summary card; those are not gaps.

Copy

  • The module header convention, first, because it makes the decomposition reviewable.
  • Tee every narrator line to ${VAR_DIR}/logs/install-<ts>.log; write a failure report from the ERR trap; have hal0 doctor bundle glob both.
  • The 25-line progress protocol (HAL0_PROGRESS:<pct>:<step>:<msg>, no-op unless a flag is set).
  • Pull retry: backoff table, post-pull podman image inspect, non-retryable classifier. hal0's only pull today is (podman pull … || true) &.
  • Preflight JSON using Diagnosis field names plus a fixture corpus; reserve HAL0-PREFLIGHT-* IDs.
  • --summary-json, per-step time estimates, and later --dry-run plus a simulate harness.
  • Bootstrap-then-hot-swap as five ideas (status file, snapshot with .missing, verified-before-cleanup, lock, transient systemd-run unit), not as a file.

Models

splitreport 04

config/model-library.json (56 entries) carries per-consumer fleet-test provenance: app_compatibility.{openai_chat, hermes_talk, perplexica, …} with status, reason, evidence path, host scope and product SHA. select-model.py is a deterministic offline selector that charges an estimated KV footprint on top of file size against a memory envelope (55 % of RAM on unified memory), and encodes the unified-memory exclusion above. Activation goes through the host agent's transaction: snapshot every file it touches, swap, then require the previous model to re-prove identity and a completion before a rollback counts as successful. A small model-router service gives every app one stable alias, ods/current, that survives a swap; it exists alongside LiteLLM because LiteLLM's alias map is static YAML.

hal0's shape is ahead: N independent slots, a real registry mid-cutover to SQLite, GGUF header parsing for capabilities, a live amdgpu sysfs GTT-fit check at request time, first-class mmproj two-file pulls, and HF update checks. ODS's local models are text-only.

Copy

  • The unified-memory exclusion as knowledge (verify first; see finding B).
  • The app_compatibility schema idea as an optional field; populating it is a separate effort.
  • Install-time re-verify of an existing file against its pinned SHA, with auto re-pull.
  • The rollback bar: previous model re-proves before rollback is "successful".
  • The stable-alias idea, generalizing hal0/brain to hal0/chat only after a design pass on the dispatcher.

Network, mDNS, onboarding, auth posture

splitreport 05

Two layers. bin/ods-mdns.py is a zeroconf daemon under systemd that publishes <device>.local plus chat., dashboard., auth., api., hermes. and talk. A-records and re-reads .env every 30 s; a static AST test pins that subdomain set to the Caddyfile routes and the magic-link redirect targets so the three cannot drift. ods-proxy is Caddy on :80 doing host-based routing (path-based broke Open WebUI's assets, websockets and OAuth callbacks). It is the one service allowed to bind 0.0.0.0; everything else stays on loopback unless --lan. The cookie is Domain=<device>.local-scoped so one redemption authenticates every subdomain. Onboarding is appliance-grade: a printable 4×6 setup card with a Wi-Fi QR and a setup-URL QR, a terminal QR fallback, and a disabled-by-default AP mode that ODS itself calls destructive.

hal0's control plane and Open WebUI bind 0.0.0.0 by default with auth off; inference slots stay loopback. A full three-tier auth layer exists (api/auth.py, security/exposure.py) but ships off after an earlier auto-enable locked operators out of a dashboard that had no login UI. The doctor's posture check never reads the bind address (issue #1822, pinned to doctor_all.py:75).

Copy

  • The doctor cross-check (bind non-loopback and auth off → WARN with the remedy). About 20 lines. Highest leverage per line in this study.
  • Write the base avahi file the dashboard already claims exists.
  • Optionally a single-port Caddy gateway from the chat.<device>.local block, if the LAN decision goes that way.
  • A hal0 setup-card reusing the existing LAN-IP detection and QR primitive.

MCP and memory

hal0report 06

ODS has no MCP. Its memory is upstream Hermes's own files plus memory-shepherd, a 338-line timer that archives everything below a --- separator in a markdown file every three hours. Qdrant and TEI are deployed and the README says vectors land in Qdrant; nothing writes to it. The one genuinely excellent pattern here is generated-config discipline: patch-hermes-config.py is a block-scoped, comment-preserving patcher that distinguishes "operator wins" keys (fill only if missing) from "ODS converges" keys (re-assert unconditionally), returns changed: bool, and writes only on diff. APE's approval grants are bound to an args fingerprint and are strictly one-shot; that idea is worth stealing even though APE itself is not.

The user-added-MCP design in the report is concrete and reuses hal0's own pieces: extend the existing per-server TOML with [secrets] (indirection into /etc/hal0/api.env), [tools] (the existing ToolPolicy, so a new server starts with zero callable tools), and [exposure] (hermes, brain now; openwebui/opencode return 501 until built); regenerate Hermes's mcp_servers block on every registry mutation, deleting only entries hal0 itself wrote (main's #2253 keeps that ownership list in a side-car manifest rather than sentinel comments, since YAML comments do not survive a re-dump); add hal0 mcp test <id> that probes the server and prints each tool with its verdict; run stdio servers behind a client-attached bridge unit (hal0-mcp@<id> executing a hal0-owned stdio↔streamable-http adapter), never as a free-standing process with nobody on its pipe. Write ADR-0015 first; the code already promises it. Corrected 2026-10-04 (report 06 §E.2–E.3): the builtin loop's admin bearer must not be reused for third-party URLs; the [tools] mirror only describes policy — Hermes talks to the server directly and nothing enforces allow/gated/blocked until hal0 fronts user-installed servers with a policy proxy mount (issue #2303); and a stdio server needs a client-attached bridge, not a free-standing unit.

Copy

  • The block-scoped patcher for one job: removing stale hal0-owned mcp_servers blocks, which hermes config set cannot express.
  • The writers table (see agents).
  • Windowed caps plus a circuit breaker for gated tools, keyed on (server, tool) not on guessed intent.
  • Persisted, args-fingerprinted one-shot approval grants; the queue is in-memory today and a restart drops pending approvals.

Dashboard and self-explanation

ODSreport 07

ODS's nav is seven items and outcome-named. GET /api/features turns manifest data plus live health into first-person sentences ("Your hardware can run Document Q&A. Enable it?"), and GET /api/features/{id}/enable returns numbered steps with deep links that a modal renders with working buttons. Every extension status word has a one-sentence description; every settings save states in advance which containers it will recreate; the phone-first FirstBoot wizard takes four taps; the docs include a "friendly guide" that explains the mesh as an office building and an "if you know Ollama, ODS adds…" table. The installer, when it auto-disables something, prints the exact command to reverse that in the same breath.

hal0 is more rigorous where it counts (one classifier drives CLI and API; RestartApiPanel is a complete confirm → restart → poll → toast loop; ApplyBadge gives per-field restart guidance) and weaker where the user is stuck: next_steps is hard-coded empty, waits are silent, one screen shows two different "free memory" numbers, and the nav teaches the architecture ("Slots", "Runner Images", "Stacks") before it teaches what you can do.

Copy

  • Populate health_report.to_diagnosis() with real NextSteps (the CLI already knows the commands) and make the chips copy or run. Highest leverage in this domain.
  • The enable-steps modal shape for slot-load failures and convergence panels.
  • Status word → one sentence, alongside the precise phase vocabulary.
  • "Auto-derived X; reverse with Y" wherever hal0 picks a default silently.
  • Full Open WebUI env wiring (RAG, search, image), gated on live slot state.
  • The README "at a glance" tables and a post-install checklist page.

Container runtime, CLI, AMD host tuning

hal0report 08

ODS is Docker Compose with a formal layering contract (base + hardware overlay + mode overlay + extension fragments), resource limits and healthchecks on every service, an env schema validated in a five-tier CI matrix, and tag-only image pins with a written ADR accepting :latest for three images. Podman is tolerated only when it wears the Docker CLI mask. The ods-cli is one 6,595-line Bash file with a written but unexecuted decomposition plan and inconsistent JSON flags. Where ODS is clearly ahead is 10-amd-tuning.sh: render/video groups, amdkfd modprobe, swappiness and cache pressure, amdgpu ppfeaturemask, RAM-scaled GTT/TTM sizing (90 % of RAM at ≥ 96 GB → about 115 GB on a 128 GB box), amd_iommu=off on single-GPU APUs, a tuned accelerator-performance profile, and a reboot banner. hal0's own upstream toolbox README documents the same GTT/TTM GRUB parameters as required and hal0's installer applies none of them. (The README's amd_iommu=off is the one line hal0 should keep ignoring: it drops the NPU's /dev/accel and hal0 documents iommu=pt amd_iommu=on — see D7.)

hal0's runtime is the stronger design: Podman Quadlets with systemd's geometric restart backoff, a boot-time GPU device-group converge service, digest pins on every image, and Pydantic validation at load time. Its CLI is a modular Typer tree with issue-annotated mount points. Two source-confirmed defects match your complaints: unbounded silent lifecycle verbs (#1870/#1869) and hal0 update never re-linking /usr/local/bin/hal0 (#1844, which already caused #2092).

Copy

  • The AMD tuning phase as installer/lib/amd-tuning.sh, opt-in only (kernels ≥ 6.14 size GTT dynamically; see D7), GTT/TTM + sysctl + tuned, with the reboot banner.
  • gfx1151 HSA_OVERRIDE_GFX_VERSION auto-detection without clobbering an operator value.
  • Known-bad kernel and firmware checks in hal0 doctor (kernel < 6.18.4, the broken linux-firmware release), from the toolbox README.
  • Optional MemoryMax=/CPUQuota= on slot quadlets, off by default.
  • A CI job that materializes each bundle tier's config and asserts it loads.

Side-by-side matrix

ConcernODShal0LeadWhy
Agent runtimePinned upstream container, internal-only, Caddy forward_authpip wheel under sandboxed systemd unit, loopback, HMAC WS proxyhal0Tighter sandbox; ODS needed HERMES_DASHBOARD_INSECURE=1
Agent config writesTemplate copied on first start + regex patcher on upgradehermes config migrate/set, converging, backup, operator overrideshal0Works with upstream's config surface instead of around it
Agent context floorHard-coded 65536; installer raises box contextRead from installed Hermes; effective = min(model, slot); read-only diagnosishal0Correct by construction
Agent auth & sharingSigned cookie, magic links, owner cards, admin-sessionOne HMAC seam on the chat proxy; approvals were unauthenticated (gated on LAN-bound boxes since #2246)ODSLargest single gap in hal0
Mobile chat surfaceODS Talk: pooled WS bridge, SSE keepalive, tool captions, TTSNoneODSBest engineering in the ODS repo
Tool policyAPE engine, wired to nothing, advisory by defaultDefault-deny server+tool axes, load-time disjointness, approval inboxhal0hal0's is on the call path
Extension modelmanifest + compose fragment; 14 auto-attaching consumersHard-coded 4-entry registry; hal0 app supports one appODShal0 has ingredients, no wiring
Installer structure363-line orchestrator, 14 phases, 19 pure libs, contracts docOne 4,002-line script, 16 steps, no lib/step splitODSEnables unit tests, dry-run, summary JSON
Install forensicsTee'd log, failure report, preflight JSON, summary JSONTerminal only; good ERR trap; support bundle existsODSTranscript lost when the terminal scrolls
Preflight substanceThreshold checker, fixture-testedLXC privilege classes, GPU gid remediation, soft/hard gateshal0More capable, less provable
Updatergit pull + migrations + snapshot/rollbackcosign keyless, sha256, staged tree, atomic symlink, venv rollbackhal0Different class of integrity
First token after install~2 min via bootstrap model, hot-swap later with rollbackBlocking 15 to 31 GB decision; brain model pull never fatalODSBest reliability idea in the repo
Model catalog56 entries with per-app fleet-test provenance41 curated, code-shipped, tool_calling flag + notesODSEvidence vs. assertion
Inference modelOne llama-server, swap the whole modelN slots, capability cards, NPU trio, GPU arbiterhal0Architecturally ahead
Memory fitStatic envelope at installLive amdgpu sysfs GTT verdict per requesthal0Truth over estimate
Swap rollback barPrevious model must re-prove before rollback countsState-machine transition backODSAdopt the bar
Name → IPOwn zeroconf daemon, subdomain A-recordsavahi service-group XML, system daemonhal0Lighter, same outcome for hal0's shape
Port → serviceCaddy :80 host router, one URLEach service its own port; links follow request HostODSUX; hal0 needs no proxy for raw use
Default bindLoopback; proxy is the sole LAN surface; --lan opt-inControl plane and chat UI 0.0.0.0; slots loopbackODSSafer default
Headless onboardingAP mode, setup card, terminal QR, FirstBoot wizardTerminal QR at installODSAppliance framing vs. operator framing
MCPNone2 hosted servers, registry, REST, CLI, resolver, policyhal0Finish it, don't start it
MemoryHermes files + markdown reset timer + orphaned QdrantProvider ABC, Hindsight, closed namespace grammar, fail-closed readshal0Reliability on CPU boxes is hal0's open problem (#1834)
Remediation UXNumbered steps + deep links from the API; status sentencesTyped next_steps model, rendered, but emptyODSPlumbing exists; fill it
Open WebUI wiringChat, RAG, search, STT, TTS, image with baked workflowChat, STT, TTSODSNative buttons point at nothing
Container runtimeCompose, tag pins, :latest ADR, Docker 29.3 workaroundQuadlets, systemd backoff, digest pins, boot GPU convergehal0Cleaner primitive, no daemon
AMD host tuningGTT/TTM, iommu, sysctl, tuned, reboot bannerDiagnosis only; none appliedODSCorroborated by hal0's own toolbox README for GTT/TTM; amd_iommu=off is NPU-hostile and stays out (D7)
CLI6,595-line Bash file; presets; masked config show35-file Typer tree, issue-annotated; silent long waitshal0Architecture; fix the waits
Config validationBash/jq schema validator, CI matrix per tierPydantic at load; no materialized-config CIhal0Add the CI matrix
Doc hygieneDense operator docs; several stale claimsGood taxonomy; several stale claims and an ADR gaptieBoth need a drift pass

The install, observed

You asked for a fresh privileged LXC with the model directories mounted. I could not do that from this session: it runs in a remote sandbox with no GPU, 4 vCPU, 15 GB RAM, about 30 GB of disk and a Docker daemon I started by hand. So the install below is the closest honest equivalent: a rootless install user, CPU tier 0, non-interactive, with voice, workflows, RAG and ComfyUI disabled to fit the disk. Commands to reproduce it properly on the 395 box are at the end of this section. The full journal is appendix 00.

StepWall clockOutcome
Run 1: preflight → detection → config → dev tools → image pulls → model download07:25:36 → 07:31Rootless posture chosen up front; CPU fallback; tier 0 forced; 7.7 GB of images pulled largest-first; 1.2 GB GGUF SHA-verified; Hermes template patched with the served model id
Run 1: local image builds07:31 → 07:33:59All builds failed on TLS inside BuildKit (sandbox CA). Three attempts each, per-service build log, then the installer refused to start stale images and aborted. Correct behaviour.
Run 2 blocked on the model-lifecycle lock07:36:23 → 07:37Run 1's host agent had inherited the installer's flock descriptor. Killed it; the lock released. A real ODS bug (below).
Run 2: builds → compose validate → up → health → summary→ 07:41:074 min 44 s end to end with caches warm. 13 of 15 containers healthy; readiness summary Ready now: 5/6; summary JSON written
Post-install fixes07:42 → 07:58ods-dashboard and ods-searxng crash-looped because this kernel has no IPv6 (listen [::]:3001, GRANIAN_HOST=::). Patched the install copy; both healthy afterwards.

What the installer taught me about ours

  • Privilege posture is decided once, early. With no passwordless sudo the installer printed "continuing rootless", and every later privileged step was skipped and logged rather than failing under set -e. hal0's installer is sudo bash; the pattern still applies to its optional steps.
  • Six user-facing phases over thirteen internal ones, each with an estimated time. The internal structure is compressed into a storyline the operator can follow.
  • Hardware is probed once and serialized (/tmp/ods-capabilities.json); every later phase and the doctor read the file. The preflight verdict is also JSON, with an action string per check.
  • The model selector explains itself in one sentence: "Qwen 3.5 2B needs about 3GB including context/KV, fits 5.2GB system RAM on cpu, and gives 64K context." hal0's picker has the data to say this and does not.
  • Every disabled feature says why ("Qdrant compose disabled (RAG not enabled or unsupported on this host)"), and every sub-step of the config phase is named, so a failure names the step.
  • Image tags are validated before the pull, largest image first, with a counter and a human label per image. The GGUF is SHA-verified after download and the Hermes template is patched with the model id llama-server will actually advertise, then the substitution is checked.
  • Compose config is validated before up, and manifest depends_on is checked, so a broken fragment fails before a container starts.
  • It found a real ODS bug. In the non-systemd fallback the host agent is started as a background child while the installer holds its model-lifecycle flock; the child inherits the descriptor (/proc/<pid>/fd/10 → …/ods-model-lifecycle-….lock) and holds the lock after the installer exits, so the next run blocks on "Waiting for another model lifecycle operation". ODS guards the bootstrap upgrader against exactly this with _phase11_close_inherited_fds_for_daemon but not the host agent. Under systemd the unit starts fresh and nobody notices. Worth an upstream issue, and a reminder for hal0's own systemd-run plan: never inherit a lock into a daemon.

Two sandbox-only adjustments, disclosed

The sandbox re-terminates outbound TLS, so image builds inside BuildKit could not verify certificates; I injected the sandbox CA into the eight locally built Dockerfiles in my install-user copy only (never the reference clone the reports were written against). And because the installer requires data/hermes to be mode 700 when sudo is unavailable, I pre-created it. Neither changes ODS behaviour.

The surfaces, as installed

Screenshots from the sandbox stack at 1440×900, dark scheme. The dashboard shots were taken after the IPv6 patch; no data beyond the fresh install exists on these pages.

ODS dashboard · overview on a fresh CPU-only install
ODS dashboard · overview on a fresh CPU-only install
ODS dashboard · Extensions page (33 catalog entries, 24 marked incompatible on CPU)
ODS dashboard · Extensions page (33 catalog entries, 24 marked incompatible on CPU)
ODS dashboard · Models page
ODS dashboard · Models page
ODS dashboard · Settings (schema-rendered .env editor)
ODS dashboard · Settings (schema-rendered .env editor)
ODS dashboard · Setup / Owner cards and guest invites (the list endpoint returned 500 on this fresh CPU install; the empty states still explain what to do)
ODS dashboard · Setup / Owner cards and guest invites (the list endpoint returned 500 on this fresh CPU install; the empty states still explain what to do)
Hermes behind hermes-proxy: forward_auth denies with "Owner card required"
Hermes behind hermes-proxy: forward_auth denies with "Owner card required"
Open WebUI as pre-wired by ODS
Open WebUI as pre-wired by ODS
Token Spy usage monitor
Token Spy usage monitor
Perplexica deep-research UI
Perplexica deep-research UI

Reproduce it properly on the Strix Halo box

Corrected 2026-10-04: the first draft used lxc launch / lxc config, which is LXD's client, not Proxmox's. On a Proxmox VE host the container CLI is pct and device passthrough lives in /etc/pve/lxc/<CTID>.conf; the recipe below mirrors hal0's own docs/getting-started/proxmox.mdx. The LXD form follows for anyone on a plain LXD host.

# ── Proxmox VE host ──────────────────────────────────────────────────────────
# pick a template; ODS's tested list includes Ubuntu 24.04. Resolve the exact file name the way
# hal0's scripts/proxmox-ve/hal0.sh:278 does rather than hard-coding a build number
pveam update
TPL=$(pveam available | awk '$1=="system" && $2 ~ /^ubuntu-24.04-standard.*amd64.tar.zst$/ {print $2; exit}')
[ -n "$TPL" ] || { echo "no ubuntu-24.04 template offered by pveam"; exit 1; }
pveam download local "$TPL"
CTID=9105
pct create $CTID local:vztmpl/$TPL \
  --hostname ods-lab --cores 8 --memory 32768 --swap 0 --rootfs local-lvm:64 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --unprivileged 0 --features nesting=1,keyctl=1,fuse=1,mknod=1
# first boot WITHOUT GPU lines: the gid= value must be the render group's id INSIDE the guest,
# which cannot be known before the guest exists (docs/getting-started/proxmox.mdx §3 does the same dance)
pct start $CTID
RG=$(pct exec $CTID -- sh -c 'getent group render >/dev/null || groupadd -r render; getent group render' | cut -d: -f3)
echo "guest render gid: $RG"
pct stop $CTID
# GPU nodes + model stores go in the CT config (PVE 8.2+ dev passthrough). BOTH nodes carry gid=$RG:
# hal0's installer realigns /dev/kfd to the render gid itself (install.sh:487-498; hal0.sh:253,411-446
# does the same on the host side) but ODS's does not, so a kfd left at the host's gid is unreadable by ods.
# A privileged CT with GPU nodes also needs an unconfined AppArmor profile, or the container
# runtime inside the guest cannot launch (docs/getting-started/proxmox.mdx §AppArmor; hal0.sh:387-391).
# hal0's store is mounted read-only; ODS gets its own writable model dir on host storage (mp1) so a
# 15–31 GB download does not land on the 64 GB rootfs.
mkdir -p /var/lib/ods-lab/models
# derive the device majors the way hal0.sh:199-204 does (stat prints hex; cgroup rules want decimal) —
# never assume 226/234
DRI_MAJ=$(printf '%d' 0x$(stat -c '%t' /dev/dri/renderD128)); KFD_MAJ=$(printf '%d' 0x$(stat -c '%t' /dev/kfd))
cat >> /etc/pve/lxc/$CTID.conf <<EOF
dev0: /dev/dri/renderD128,gid=$RG
dev1: /dev/kfd,gid=$RG
mp0: /var/lib/hal0/models,mp=/mnt/hal0-models,ro=1
mp1: /var/lib/ods-lab/models,mp=/mnt/ods-models
lxc.cgroup2.devices.allow: c $DRI_MAJ:* rwm
lxc.cgroup2.devices.allow: c $KFD_MAJ:* rwm
lxc.prlimit.memlock: unlimited
lxc.apparmor.profile: unconfined
EOF
pct start $CTID
pct exec $CTID -- stat -c '%g %n' /dev/dri/renderD128 /dev/kfd    # BOTH must show $RG before installing (docs/getting-started/drivers.mdx)
pct exec $CTID -- bash -lc 'apt-get update && apt-get install -y curl jq git rsync docker.io docker-compose-v2 && \
  useradd -m -s /bin/bash ods && usermod -aG docker,render,video ods && \
  echo "ods ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ods'
# 21f4b3a is the ODS revision every citation and result in this study is pinned to
pct exec $CTID -- su - ods -c 'git clone https://github.com/Osmantic/ODS.git && cd ODS && git checkout 21f4b3a'
# ODS reads ./data/models/*.gguf relative to ~/ODS/ods (docker-compose.base.yml:29; 11-services.sh:668,697) and
# writes its bootstrap download there (bootstrap-model.sh:44), so point that path at the writable host dir
# before install.sh runs. Seed it with any hal0 GGUF you want ODS to reuse — the file name must match ODS's
# catalog entry or the installer downloads its own copy regardless.
pct exec $CTID -- bash -lc 'chown ods:ods /mnt/ods-models && mkdir -p /home/ods/ODS/ods/data && ln -sfn /mnt/ods-models /home/ods/ODS/ods/data/models'
pct exec $CTID -- su - ods -c 'cp /mnt/hal0-models/<path>/<file>.gguf /mnt/ods-models/'   # optional, once per model
pct exec $CTID -- su - ods -c 'cd ~/ODS/ods && ./install.sh --all --summary-json ~/ods-summary.json'
# ODS's phase 10 edits grub/modprobe INSIDE the CT; those belong to the Proxmox host kernel, and the
# GTT/TTM numbers it printed are WRONG for the host: phase 10 sizes them from the CT's own MemTotal
# (10-amd-tuning.sh:140-157) — 32 GB here, not the host's 128 GB. If you want fixed sizing at all (it is
# opt-in; kernels ≥ 6.14 size GTT dynamically, drivers.mdx:86-97), recompute ON THE HOST from its own
# /proc/meminfo — e.g. hal0's documented 128 GB line, drivers.mdx:100-104:
#   amdgpu.gttsize=118784 ttm.pages_limit=30408704 ttm.page_pool_size=30408704
# then reboot the host (report 08 §D.1). Do NOT copy ODS's amd_iommu=off: it removes /dev/accel (the
# XDNA NPU) and contradicts hal0's own iommu=pt amd_iommu=on (drivers.mdx:102; ADR-0003:96-99).
# afterwards: ods status; ods list; ods doctor --json; cat /tmp/ods-preflight-report.json

# ── plain LXD host (not Proxmox) ────────────────────────────────────────────────
lxc launch images:ubuntu/24.04 ods-lab -c security.privileged=true -c security.nesting=true
RG=$(lxc exec ods-lab -- sh -c 'getent group render >/dev/null || groupadd -r render; getent group render' | cut -d: -f3)
lxc config device add ods-lab kfd unix-char path=/dev/kfd gid=$RG
lxc config device add ods-lab dri unix-char path=/dev/dri/renderD128 gid=$RG
mkdir -p /var/lib/ods-lab/models
lxc config device add ods-lab hal0models disk source=/var/lib/hal0/models path=/mnt/hal0-models readonly=true
lxc config device add ods-lab odsmodels disk source=/var/lib/ods-lab/models path=/mnt/ods-models
lxc exec ods-lab -- bash -lc 'apt-get update && apt-get install -y curl jq git rsync docker.io docker-compose-v2 && \
  useradd -m -s /bin/bash ods && usermod -aG docker,render,video ods && \
  echo "ods ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ods'
lxc exec ods-lab -- su - ods -c 'git clone https://github.com/Osmantic/ODS.git && cd ODS && git checkout 21f4b3a'
lxc exec ods-lab -- bash -lc 'chown ods:ods /mnt/ods-models && mkdir -p /home/ods/ODS/ods/data && ln -sfn /mnt/ods-models /home/ods/ODS/ods/data/models'
lxc exec ods-lab -- su - ods -c 'cd ~/ODS/ods && ./install.sh --all --summary-json ~/ods-summary.json'
# same host-side tuning caveat as the Proxmox branch: recompute on the LXD host from host RAM; never amd_iommu=off

Adoption plan

Principles

  • Copy shapes, not files. The ODS artifacts that are literally droppable are small (the session signer, the patcher, the progress protocol, the AMD tuning math). Everything else is a design decision hal0 re-implements in Python on podman and systemd.
  • Extensions are companion services, never slots. A separate unit family, hal0-ext@<id>, rendered from a typed manifest, never from an author-written unit body. Explicitly rootful in v1 — it goes through the same system-scope seam as slots — with compensating controls (digest-pinned images, volumes confined to /var/lib/hal0/ext/<id>/, no devices, capabilities dropped) and third-party import disabled until a user-scope seam makes rootless real (report 02, Decision 3, corrected 2026-10-04).
  • Keep what is better. Quadlets, digest pins, the cosign updater, pip Hermes, the Typer CLI, Hindsight, the shared health classifier. None of these are on the table.
  • Every installed thing stays pre-wired to the slots. Pre-wiring in hal0 is rendered env, computed at unit-render time from live truth: OPENAI_BASE_URL to hal0-api, OPENAI_MODEL=hal0/brain, and slot:<name> references resolved to hal0-api's gateway route for that modality — slots are loopback-only on the host, so a bridge-network extension cannot reach a slot port directly, and a value rendered once does not follow a later port change, whereas the gateway's port is stable and the dispatcher follows the slot. Direct 127.0.0.1:<slot port> wiring is reserved for host-network extensions and needs a slot-change convergence hook that does not exist yet (report 02 §D.5, corrected 2026-10-04). Still stronger than ODS's static DNS names, for a narrower claim.
  • Finish before starting. Wave 0 closes completely before Wave 1 opens. The backlog already has enough 80 % threads.

Port ledger

Ranked within waves. Size: S under a day, M under a week, L multi-week. Risk is integration risk in hal0. "Issues" are the open hal0 issues the row closes or de-risks.

#PortFrom (ODS)To (hal0)SizeRiskImpactIssues
Wave 0 · stop the bleeding
0.1Signed session cookie + verify-session + admin-session; gate approvals and other mutation routes (approvals gated on LAN-bound boxes by #2246)dashboard-api/session_signer.py, routers/auth.pysrc/hal0/api/auth/, generalize api/agents/_auth.pySlowhigh—
0.2Reproduce Qwen3-Coder-Next on a real 395; demote from hal0-Max and add a unified-memory exclusion if it reproducestier-map.sh:100-138, select-model.py:326-372registry/curated.py:260-275, model_fit.py, bundle manifestSlowhigh—
0.3Doctor cross-check: bind non-loopback and auth off → WARN with remedy; same line in the install summaryods-doctor.sh:1074-1089, 13-summary.sh:419-427cli/doctor_all.py:62-83, installer/install.sh:3934Slowhigh#1822
0.4Bound every seam subprocess call; typed timeout errorods-uninstall.sh:321, bootstrap-upgrade.sh:939providers/container.py:2549, :2697, :2708, :2900Slowhigh#1869, #1870
0.5Re-assert /usr/local/bin/hal0 in the updater next to refresh_privileged_wrappers; have deploy.sh call the same refresh13-summary.sh:270-300updater/updater.py:4159, scripts/deploy.shSlowhigh#1844, #2019
0.6Server-built next_steps for convergence; the panel prints exactly those linesods-doctor.sh:1110-1163/api/updates/convergence, cli/update_commands.py:466-492Slowhigh#1845
0.7Deploy verifies the served build identity before printing successods-update.sh:378-406, :647-742scripts/deploy.sh:213-231, expose version on /api/statusSlowhigh#1550
0.8Doc drift pass: provisioner description, BUNDLED_AGENTS, Documentation= path, duplicate memory provider, ADR bookkeeping—ARCHITECTURE.md, CLAUDE.md, docs/adr/, installer/agents/hermes/pluginsSnonemed—
Wave 1 · cheap shapes with big returns
1.1Module header convention on every installer lib; tee'd install log; failure report from the ERR trap; --summary-json; per-step time estimatesui.sh:97-103, compose-failure-report.sh, 13-summary.sh:435-502installer/lib/ui.sh, new installer/lib/failure-report.sh, hal0 doctor bundleMlowhigh—
1.2Pull retry discipline: backoff table, post-pull inspect, non-retryable classifierui.sh:269-370installer/install.sh:2177, components/runner_images_arm.pySlowhigh—
1.3Preflight JSON report using Diagnosis fields + fixture corpus with named scenariosscripts/preflight-engine.sh, tests/contracts/test-preflight-fixtures.shinstaller/lib/preflight.sh, tests/installer/Mlowhigh—
1.4Populate next_steps for the seven doctor checks; chips copy or run; enable-steps modal shaperouters/features.py:130-144, :188-250, FeatureDiscovery.jsxhealth_report.py:246-283, DiagnosisPanel.jsxMlowhigh#1845, #1994
1.5Hardware re-detect cluster: live-probe fallback for stale hardware.json; kfd_present() and render_node_present() sufficient for ROCm; one apply_cpu_fallback(); clamp seed context_size to the envelopedetection.sh:26-46, :187-257, select-model.py:134-182slots/capacity.py:499-585, install/profile_derive.py:110, install/static_seeds.pyMmedhigh#1862, #2216, #1936, #1966, #1868
1.6Manifest-driven services health with gather(return_exceptions=True); first row of the extension registrydashboard-api/config.py:297-431, helpers.py:624-700api/routes/services_health.py:198-307, services/registry.pyMlowhigh#2028, #2192, #1859
1.7AMD host tuning lib (opt-in only, no default-on: kernels ≥ 6.14 size GTT dynamically and hal0 reads the live pool, drivers.mdx:86-97; GTT/TTM + sysctl + tuned only — never amd_iommu=off, which drops the NPU's /dev/accel and contradicts drivers.mdx:102; container-guarded — prints the host-side recipe inside an LXC, never claims applied); gfx1151 HSA auto; kernel/firmware known-bad checks in doctor10-amd-tuning.sh:100-302, .env.example:477new installer/lib/amd-tuning.sh, install/profile_derive.py, cli/doctor_commands.pyMmedhigh—
1.8Full Open WebUI wiring (RAG embeddings, SearXNG-style search if present, image gen with baked workflow), gated on bound slotsdocker-compose.base.yml:100-168openwebui/env_writer.py:106-131Mmedmed—
1.9Update recreates slots on image roll by default (opt-out flag); stack import verifies checksum on commit; uninstall gates its final line on per-step outcomes; TTL on negative FLM probe; retention prune on a timerods-cli:2049, ods-restore.sh:273-310, ods-uninstall.sh:362-402, helpers.py:26-56cli/update_commands.py:516, stacks/portable.py, agent uninstall, capabilities/catalog.py:178, activity pruneMlowmed#2096, #1512, #1519, #1974, #1521
1.10Docs: generated-config-writers table; area → required-validation table; reinstall contract; known-good versions; README "at a glance"; post-install checklistINSTALLER-ARCHITECTURE.md:118-135, HIGH_RISK_CHANGE_MAP.md, KNOWN-GOOD-VERSIONS.mdARCHITECTURE.md, CONTRIBUTING.md, docs/reference/, README.mdSnonemed—
Wave 2 · structural
2.1Decompose install.sh into installer/steps/, one step per PR, UI_STEP_TOTAL test green throughout; then --dry-run and a simulate harnessinstall-core.sh, installers/phases/, simulate-installers.shinstaller/install.sh → orchestrator + stepsLmedhigh—
2.2Finish user-added MCP — partly shipped on main (#2253, ADR-0015): schema, exposure join, hal0 mcp test|allow|gate|block|expose, REST. Remaining: policy proxy mount so [tools] is enforced on Hermes's call path (#2303); client-attached stdio bridge (not a free-standing unit); install drawer (#2245)patch-hermes-config.py:17-59,114-115,215-218mcp/installed.py, routes/mcp.py, agents/hermes_provision.py:2012, ui/src/dash/connections.jsxLmedhigh#1930 first
2.3Extension model v1: manifest.toml → hal0-ext@<id> quadlet; registry walker; capability status vocabulary; hal0 app verbs; allow-list +2 keys; convert OpenWebUI as proof; n8n as the first new extensionmanifest schema, service-registry.sh, features.py:19-110, audit-extensions.py, templatesnew src/hal0/extensions/, system/seam.py:283, installer/wrappers/hal0-systemctl:389-516, cli/app_commands.pyLhighhigh#1947, #1319
2.4Progress/timeout/evidence seam: stream slot phases over SSE into the CLI; failure report per slot; classified retry for netavark/name-conflict12-health.sh:66-106, background-tasks.sh:119-148, ods-cli:960-1140cli/slot_commands.py:311-362, slots/, system/netavark.py + seam.prune_dnatMmedhigh#1870, #2221, #1820, #2180
2.5Persist the approval queue; args-fingerprinted one-shot grants; windowed caps + circuit breaker keyed on (server, tool)ape/main.py:482-612, :954-1010mcp/approval_queue.py, new mcp/limits.pyMlowmed—
2.6Schema-driven settings renderer; one ChangeSet diff shared by preview and applysettings.py:194-241, ods-cli:6261-6324settings pages, slot_configMlowmed#2108, #1967, #2195, #2203, #1511
Wave 3 · product surfaces (each gated on a decision below)
3.1Single-URL LAN gateway: Caddy forward_auth on :80, <host>.local subdomains, owner card + guest inviteods-proxy/Caddyfile, hermes-proxy/Caddyfile:54-129, routers/magic_link.pynew packaging/hal0-proxy/, api/routes/services.py:38-73 (X-Forwarded-Host); prerequisites: take the direct ports off the LAN first (install.sh:157,1567 seed 0.0.0.0 today) and keep that converged on upgrade, or forward_auth is bypassable on :8080/:3001 — but not by binding hal0-api to loopback alone: Open WebUI and every bridge-network extension reach it via host.docker.internal, the podman bridge gateway, not 127.0.0.1 (openwebui/env_writer.py:102-128), so hal0-api needs a second listener on the bridge gateway address or an nftables rule that drops LAN ingress to :8080/:3001 while exempting the bridge subnet; plus an mDNS address publisher for the subdomains (services/mdns.py writes _http._tcp service records only)Lmedmed—
3.2Bootstrap-then-hot-swap for first-run tiers via systemd-run, status JSON, snapshot/rollback, verified-before-cleanupbootstrap-model.sh, 11-services.sh:673-690, bootstrap-upgrade.sh:305-452install/agent_model.py, new install/background_pull.py, hal0-model-pull@.serviceLmedhigh—
3.3Agent-driven OAuth passthrough + provider registry + persona addendumrouters/oauth_passthrough.py, hermes/oauth-providers.jsonnew api/routes/oauth.py, installer/agents/hermes/Mlowmed—
3.4Phone chat portal on the pooled-bridge pattern (only if decided)hermes_bridge.py:162-528, routers/talk.pynew REST/SSE surface beside api/agents/chat_proxy.pyLmedmed—
3.5Setup card + consequence-first copy pass on status words and tooltipsgenerate-setup-card.py, Extensions.jsx:60-84new hal0 setup-card, slot-status.js, settings tooltipsMlowmed—
Wave 4 · optional, design-pass first
4.1Stable alias generalization (hal0/chat) through the dispatchermodel-router/app/main.pysrc/hal0/dispatcher/Lhighmed—
4.2Fleet-verified compatibility ledger on curated entries, populated by hal0's own release-validation kitmodel-library.json:9-63registry/curated.py, tests/release-validation/Lmedmed#1823
4.3Optional resource limits on slot quadlets; CI job materializing each bundle tier's configdeploy.resources, validate-env.ymlproviders/container.py:803, .github/workflows/Slowlow—

Waves

Numbered because the order carries information: each wave makes the next one reviewable.

0~1 week

Stop the bleeding 8 items, all S

Close the auth hole, settle the flagship model question, land the five one-afternoon fixes the issue map ranks highest, and fix the documentation the later waves will be built against.

  • Done when: approvals require a session; the 395 verdict on Qwen3-Coder-Next is recorded in an issue; hal0 doctor all warns on a LAN-open, auth-off box; no seam call is unbounded; hal0 update re-links the CLI; #1845's panel prints a command that works; deploy.sh cannot lie; ARCHITECTURE.md matches the code.
12–3 weeks

Cheap shapes, big returns 10 items, S–M

Instrument the installer, make hardware truth fresh, give the doctor real next steps, wire Open WebUI fully, and offer the AMD host tuning hal0's own upstream documents (opt-in; see D7).

  • Done when: a failed install leaves a log and a report; preflight verdicts are pinned by fixtures; every doctor row has a runnable step; the ROCm lane follows /dev/kfd and a renderD* node together (kfd_present() and render_node_present()); an image roll restarts slots; a 128 GB box's live GTT verdict is in the doctor and fixed sizing is offered, not applied; Open WebUI's upload and image buttons work.
21–2 months

Structural 6 items, M–L

Decompose the installer, finish MCP, ship the extension model with OpenWebUI converted and n8n added, and build the progress seam the CLI needs.

  • Done when: installer/steps/ exists and --dry-run runs in CI; hal0 mcp add reaches Hermes with default-deny tools; hal0 app install n8n renders a unit, claims a port, wires env, probes health and shows a card with no code edit; hal0 slot load prints phases.
3decide first

Product surfaces 5 items, gated on decisions D1, D2, D8

Only after the LAN-posture and phone-surface decisions: the single-URL gateway with owner cards, bootstrap-then-hot-swap, OAuth passthrough, an optional phone portal, and the copy pass.

4optional

Design-pass items 3 items

The stable alias, the fleet ledger, resource limits. Each needs its own design note before code.

Issue clusters: one change, many issues

Of 106 open issues, 4 are direct ports, 34 are pattern adoptions, 29 are informed by ODS, and 39 are unrelated (polish sweeps, test hygiene, upstream pins, Hindsight internals). Eighteen of the twenty an operator actually hits on a real box fall into five clusters. Full table in appendix 09.

Clusterhal0 changeCloses or de-risksLedger
A. Progress / timeout / evidenceA bounded_call() every seam invocation goes through; phase events on the existing SSE bus; a classified retry for known-recoverable failures; a per-slot failure report#1870 #1869 #1873 #2221 #1550 #1820 #21800.4, 0.7, 2.4
B. Manifest-driven service and capability registryOne schema-validated table read by the health route, Omni eligibility, the device picker and the context resolver#2028 #2192 #1966 #1859 #2190 #2184 #1990 #1823 #2234 #1947 #13191.6, 2.3
C. Hardware re-detect and honest CPU fallbackLive-probe fallback for a stale cache; device nodes decide the lane; one apply_cpu_fallback(); one memory-envelope function read by seeds, ruler, extraction sizing and the anchor floor#1862 #2216 #1936 #1966 #1868 #1834 #1867 #2201 #1928 #19291.5
D. Update and deploy delivery integrityRe-assert PATH links; verify the served build; verify digests on commit; gate success on per-step outcomes; recreate on image roll#1844 #2019 #1550 #2096 #1845 #1519 #1512 #1530 #21010.5, 0.6, 0.7, 1.9
E. Schema-driven settings and previewsRender a settings field for every schema key; one diff function shared by preview and apply#2108 #1967 #2195 #2203 #1511 #14452.6

Decisions only you can make

Ordered by conviction: what I would do, the safe version, and the cheapest experiment. Reversible ones can move fast; the two that touch boot and trust deserve a day of thought.

D1. LAN exposure model

What I'd doDefault HAL0_BIND_HOST to loopback (the library fallback already is) together with a bridge-reachable second listener or a LAN-ingress firewall cut — loopback alone cuts off Open WebUI and every bridge extension, which reach hal0-api via host.docker.internal (report 02 §D.5 (iii)) — add an explicit --lan like ODS, and gate mutation routes with the signed session regardless. Revisit the KB-1 objection: it no longer holds now that a login page and a Security settings page exist.
Safe versionKeep 0.0.0.0, ship 0.1 (session on mutation routes) and 0.3 (doctor warning). Nobody gets locked out; the worst hole closes.
Cheap testShip 0.3 alone and count how many support threads mention the warning in a month.

Why it matters: this decision gates the whole of Wave 3.1. Reversible: a bind default is one env var.

D2. Do you want a phone surface?

What I'd doNot this quarter. ODS Talk is three ledger rows of work for a household-appliance framing hal0 has not adopted. Take the bridge discipline as a blueprint and move on.
Safe versionBuild 3.3 (OAuth passthrough) only; it is small, self-contained, and useful to the desktop chat too.
Cheap testOpen Hermes's own dashboard through a magic link (3.1) and see whether anyone uses it from a phone before building a portal.

Why it matters: 3.1, 3.3 and 3.4 stand or fall together. Reversible.

D3. Container Hermes or pip Hermes

What I'd doKeep pip. The systemd unit's ProtectSystem=strict set is a stronger boundary than ODS's container, which needed HERMES_DASHBOARD_INSECURE=1 punched through it, and hal0's plugins depend on the venv.
Safe versionSame, plus a written note in ARCHITECTURE.md so the question stops being reopened.
Cheap testNone needed.

D4. What an extension is allowed to be

What I'd doCompanion services only, never slots; explicitly rootful in v1 with the compensating controls in report 02 Decision 3 (a user-scope seam is the gate for rootless and for third-party import); a small first-party set (OpenWebUI converted as proof, then n8n); third-party import behind an explicit flag with envelope checksums; allow-list widened by exactly two literal-pinned keys; may consume a specific slot via slot:<name>.
Safe versionAPI-only consumption (no slot: refs). The eligibility half of the design collapses and v1 gets meaningfully cheaper.
Cheap testWrite OpenWebUI's manifest.toml by hand before writing any loader. If it cannot be expressed, the schema is wrong.

Why it matters: the seam allow-list is a root-boundary change; decide its width once. Partly irreversible: a published manifest schema is a contract.

D5. MCP supervisor shape (ADR-0015)

What I'd doSequence it: ship the exposure join and hal0 mcp test for http/sse servers first (done on main, #2253 / ADR-0015), then the policy proxy mount so allow/gated/blocked actually bind on Hermes's call path (#2303), then a client-attached stdio bridge behind that same mount. Not a free-standing hal0-mcp@<id>.service: a stdio server with nobody on its pipe is dead.
Safe versionhttp/sse only, forever. Halves the work; stdio users run their own bridge.
Cheap testAdd one hosted MCP server through the existing registry and time how long the manual Hermes wiring takes today. That number is the case for 2.2.

Also decide: credentials in /etc/hal0/api.env (reuses the secrets API) versus a separate file. I would reuse.

D6. Decompose the installer

What I'd doYes, sequenced 1.1 → 2.1 so headers and the log land first and every step move is reviewable; one step per PR with make harness-install between moves. Re-affirm "the installer is the whole setup" before starting, because decomposition is exactly when a post-install wizard gets proposed.
Safe versionStop after 1.1 (headers, log, report, summary JSON). Most of the forensic value, none of the refactor risk.
Cheap testExtract one step (preflight) and see whether the harness stays green.

D7. AMD host tuning: default-on or opt-in

What I'd doOpt-in only, and stay there: on kernels ≥ 6.14 amdgpu sizes GTT dynamically and hal0 already reads the live pool per request (drivers.mdx:86-97), so a fixed 65–90 % reservation is an operator's floor, not a product default — and on a shared Proxmox host it takes that RAM from every other guest. Ship it as an explicit --apply with a reboot banner; the doctor reports the live GTT and offers the pinned recipe only when the kernel is older than 6.14 or the operator asks. It edits GRUB, so keep it reversible. Scope it to GTT/TTM, sysctl and tuned: never write ODS's amd_iommu=off, which removes /dev/accel (XDNA NPU) — hal0 documents iommu=pt amd_iommu=on (drivers.mdx:102, ADR-0003:96-99) and the port must leave that line as hal0 documents it.
Safe versionDoctor-only: detect the missing GRUB parameters and print the exact lines, never write them.
Cheap testApply the toolbox README's GTT/TTM values (with hal0's iommu=pt amd_iommu=on, not the README's amd_iommu=off) on one box and confirm the fit: a GGUF that previously failed the live GTT check now loads. Do not chase the README's 5 to 12 % or ODS's ~6 %; both are the IOMMU-off delta hal0 forgoes to keep the NPU.

Slow down here: this is the one item in the plan that can leave a box unbootable if the initramfs step fails.

D8. Who is the copy for?

What I'd doKeep the technical register for reference pages; switch status words, remediation, and install output to consequence-first. That is where ODS's advantage actually lives and it is cheap.
Safe versionStatus sentences only (one small map beside slot-status.js).
Cheap testRewrite the five most-seen tooltips and watch the Discord questions.

D9. Bootstrap-then-hot-swap

What I'd doYes, but after the progress seam (2.4) exists, via systemd-run with journald logging, a status file for the dashboard, and the snapshot / verified-before-cleanup rules. It strengthens "the installer is the whole setup" by removing the one question big enough to make people defer.
Safe versionKeep the brain model as the de facto bootstrap and only add the status file so the dashboard shows the big pull's progress.
Cheap testMeasure time-to-first-token on a fresh install today versus with the brain slot answering while the tier model pulls.

Do not copy · do not do

Do not copy from ODS

  • Docker Compose, or any layering engine. Quadlets plus rendered units are the better primitive.
  • ods-cli, or any single-file dispatcher.
  • ods-update.sh's git pull mechanism, or the :latest ADR.
  • APE, the agent templates, memory-shepherd, the agents metrics router, the regex config patcher as a primary path.
  • The zeroconf daemon, the cookie-domain trick alone, AP mode, the mobile installer stub.
  • The CRT theme, the lore copy, the typing animation, the Tauri GUI, per-platform phase copies.
  • A hard-coded Hermes context floor, HERMES_DASHBOARD_INSECURE, HTML token scraping.
  • Rename-the-file as enable state, the hand-maintained ENABLE_* table, a committed generated catalog, ports.json.
  • A vector database with no consumer.

Do not do next

  • Do not fork ODS or vendor its tree. Take shapes; the files are wrong-shaped for podman/systemd.
  • Do not start Wave 1 while any Wave 0 item is open.
  • Do not begin the extension loader before the allow-list width (D4) is written down.
  • Do not build the Caddy gateway before D1 is decided; it changes the trust model — and never enable forward_auth on :80 while hal0-api (:8080) and Open WebUI (:3001) still answer the LAN directly, or the gateway is decorative — and closing them must keep the container path open (Open WebUI reaches hal0-api through host.docker.internal, not loopback).
  • Do not add a first-run wizard. Strengthen the end of the installer instead.
  • Do not ship hal0-Max's coder pick until 0.2 has a recorded verdict.
  • Do not port ODS's memory pieces to fix #1834 or #2016; those are Hindsight problems.
  • Do not widen the MCP exposure targets past hermes and brain in v1.

Appendices

The full reports, unedited. Every claim in the sections above traces to a file:line in one of these. Reports 01, 02, 03, 06 and 09 were written by Opus agents; 04, 05, 07 and 08 by Sonnet agents; 00 is my own install journal.

00-install-journal.md Install journal (sandbox run)

ODS install journal (sandbox run, 2026-09-05)

Environment: Claude Code remote sandbox (Ubuntu 24.04, 4 vCPU, 15 GB RAM, ~30 GB writable disk, no GPU, Docker 29.3.1 daemon started by hand with overlay2, egress only via an HTTPS proxy on the host). This is NOT the requested privileged LXC with the AI-model directories mounted — that must be done on the operator's Strix Halo box. Commands to reproduce there are at the end.

0. Pre-install decisions forced by the sandbox

Constraint ODS behaviour observed Notes for hal0
Installer refuses root (installers/phases/01-preflight.sh:22) Hard error with a one-line reason hal0's installer is sudo bash; ODS is rootless-first and escalates only for "extras"
No passwordless sudo for the install user ods_prepare_sudo (installers/lib/sudo.sh) detects it up front, prints "continuing rootless", every later ods_sudo call is skipped and logged instead of hanging or failing under set -e Clean pattern: decide privilege posture once, early, then make every privileged step optional and visibly skipped
data/hermes must be uid:gid:700 on a rootful runtime or the phase errors (phase 06) Pre-created it with mode 700 before running Upstream Hermes image demands 0700 on HERMES_HOME
Containers have no direct internet Image pulls are done by the daemon (which has the proxy), model download is done by the host (curl/huggingface_hub), so this mostly works Same split hal0 already has (host downloads, container serves)

Flags used: --non-interactive --skip-docker --tier 0 --no-voice --no-workflows --no-rag --no-comfyui --summary-json …

1. Phase-by-phase observations

Boot / arg parsing (install.sh → installers/dispatch.sh → install-core.sh)
  • install.sh is a 40-line dispatcher: resolves OS → Linux/macOS/Windows target and execs it.
  • install-core.sh parses ~40 flags; defaults live in one block (voice/workflows/RAG/recommended/Hermes on; OpenClaw/Langfuse/proxy/Tailscale off) with a comment on every non-obvious default.
  • Distro detection prints Detected distro: ubuntu (like: debian, pkg: apt); PyYAML is auto-provisioned for the system python because the service registry reads YAML manifests.
Phase 1/6 "Pre-flight Checks" (internal phases 01)
  • A themed banner + narrator voice: Signal acquired. I will guide the installation. Stay with me.; every phase header carries an estimated time (EST. TIME: ~30 seconds).
  • The UI shows 6 user-facing phases while the code has 13 internal phases — a deliberate compression of internal structure into a human storyline.
  • Checks: OS, curl, jq (auto-installs), optional tools (rsync → prints the exact apt/dnf/pacman command), active UFW/firewalld warning, filesystem POSIX-permission check (exFAT/NTFS/9p refuse chmod 600 .env), related-install detection (scans $HOME for other compose stacks with open-webui + dashboard-api + llama-server/litellm, and running compose projects) so two stacks never fight over ports silently.
"System detection" (internal phase 02 + 02b)
  • Writes a capability profile JSON (/tmp/ods-capabilities.json: platform, gpu vendor/name/memory_type/count/vram, runtime backend + health URL + port, compose overlays, recommended tier, hardware class). This one file is what later phases and the dashboard read — hardware is probed once and serialised.
  • No GPU → apply_cpu_gpu_fallback → GPU_BACKEND=cpu, backend contract loaded from config/backends/cpu.json.
  • Prints the full compose selection (base + cpu overlay + every enabled extension's compose fragment + docker-compose.tier0.yml memory-limit overlay), then prints one line per disabled feature saying why it is disabled ("Qdrant compose disabled (RAG not enabled or unsupported on this host)").
  • Catalog model selector output is a sentence a human can act on: Qwen 3.5 2B needs about 3GB including context/KV, fits 5.2GB system RAM on cpu, and gives 64K context. Throughput requires a local benchmark after first launch. Bounded by --tier 0's model size ceiling (1221MB); use ODS_DISABLE_CATALOG_MODEL_SELECTOR=true to bypass.
"Requirements check" (internal phase 04)
  • Delegates to a fixture-driven preflight engine that emits /tmp/ods-preflight-report.json: {summary:{checks,blockers,warnings,can_proceed}, checks:[{id,status,message,action}]}. Each check has an action string — the remediation is data, not prose in a log.
  • Disk requirement is computed from the selected model size + 15 GB images, not a static table.
"Directories / config" (internal phase 06)
  • Every sub-step is named and logged (Phase 06 step: prune-retired-services, copy-extensions-library, configure-legacy-openclaw, prepare-service-permissions, generate-env, render-model-router-config, validate-env, generate-searxng-config). A failure names the step, not just the phase.
  • Docker rootless state is detected once (lib/rootless-ownership.sh) and drives every ownership decision; ODS_UID/ODS_GID are persisted into .env so re-runs and containers agree on who owns data/.
  • A hard-coded chown … 1000:1000 for token-spy failed under uid 1001 and was reported as non-fatal with the consequence spelled out ("container may crash if installer ran as a different uid").
  • .env is generated with secrets (chmod 600), then validated against .env.schema.json before anything starts. The extension library (optional catalog) is copied into data/extensions-library/ so the dashboard can browse it offline.
"Dev tools" (internal phase 07)
  • Installs Claude Code, Codex CLI and OpenCode into ~/.npm-global (no sudo needed); OpenCode failed and the installer printed the exact one-liner to retry later. Non-fatal by design.
  • Starts the ODS host agent (bin/ods-host-agent.py): a host-side helper for model downloads and mDNS. No systemd in the sandbox → "starting the host agent without a system service … Run ods agent start after reboot". Graceful degradation, with the recovery command printed.
  • Installs zeroconf via pip --user when the distro package cannot be installed: the mDNS announcer is a Python script (bin/ods-mdns.py), not an avahi service file. It publishes <device>.local plus chat/dashboard/auth/api/hermes/talk.<device>.local A-records and re-reads .env every 30 s so renames need no restart. A static AST test (tests/test_mdns_subdomains.py) pins the subdomain set to the Caddyfile routes and the magic-link redirect targets.
Phase 4/6 "Downloading Modules" (internal phase 08)
  • Validates that each pinned image tag exists in the registry before pulling (llama-server, Hermes), so a bad pin fails in seconds, not after a 4 GB download.
  • Pull order is largest first, with a spinner, elapsed time, [n/N] counter and a human label per image ("LLAMA-SERVER — downloading the brain (CPU)", "HERMES PROXY — magic-link auth gate (Caddy)"). Rotating one-line taglines fill the wait. Images pulled: llama.cpp server-b8248 (113 MB), open-webui v0.7.2 (4.39 GB), perplexica, hermes-agent v2026.6.5 (3.13 GB), caddy 2.11.3-alpine (63 MB).
Phase 5/6 "Starting Services" (internal phase 11)
  • Compose flags are recomputed from the install directory and saved to .compose-flags (the CLI reuses them; no re-detection at every ods start).
  • GGUF download (host-side, huggingface_hub), then SHA-256 verification against the pinned digest in installers/lib/bootstrap-model.sh.
  • Generates models.ini for llama-server, then patches the Hermes template with the actually-served model id (model.default=Qwen3.5-2B-Q4_K_M.gguf, context=65536) and verifies the substitution landed.
  • Validates service dependencies (manifest depends_on) and runs docker compose config before up — a broken fragment fails before any container starts.
  • Local builds are skipped per disabled service ("Skipping local image build for disabled service: brave-search").
Run 1 outcome: local image builds failed, install aborted (by design)
  • Every locally built service (dashboard, dashboard-api, model-router, ape, privacy-shield, token-spy, …) failed with certificate verify failed inside BuildKit: the sandbox re-terminates outbound TLS and build containers do not trust its CA. Not an ODS defect.
  • ODS behaviour worth copying: each build gets three attempts with a 5 s delay, a per-service build log (/tmp/ods-install.log.<svc>.build.log), a cross-check that the build actually produced the tagged image, and then a refusal to start an image left by an earlier install ("Fix the build error and rerun the installer"). It never silently ran stale code.
  • Sandbox fix: the sandbox CA was appended to the eight locally built Dockerfiles in the install user's private copy only.
A real ODS bug found while re-running: the host agent inherits the installer's lock
  • Run 2 blocked at "Waiting for another model lifecycle operation before Linux installer model configuration…".
  • Cause: in the no-systemd fallback, phase 07 starts bin/ods-host-agent.py as a background child after the installer has taken its model-lifecycle flock. The child inherits the descriptor (/proc/<agent-pid>/fd/10 → /tmp/ods-model-lifecycle-<uid>/ods-model-lifecycle-<cksum>.lock) and keeps the lock for its whole lifetime, so any later installer run or ods model swap waits forever. ODS closes inherited FDs for the bootstrap upgrader (_phase11_close_inherited_fds_for_daemon) but not for the host agent. Under systemd the unit starts fresh and nobody notices. Killing the agent released the lock; run 2 then started its own agent, which inherited the lock again.
  • Lesson for hal0's planned systemd-run background pulls: never let a daemon inherit a lock descriptor; take locks inside the daemon.
Run 2 outcome: stack up in 4 min 44 s (images and model already cached)
Phase Wall clock Notes
Run 1 start → killed at builds 07:25:36 → 07:33:59 (8 m 23 s) ~7.7 GB of images pulled in ~3 min; 1.2 GB model downloaded and SHA-verified
Run 2 start → "YOUR ODS IS LIVE" 07:36:23 → 07:41:07 (4 m 44 s) includes 6 local image builds; ~1 min was the inherited-lock wait
Containers healthy at the end of run 2 13 of 15 ods-dashboard and ods-searxng restart-looping
  • Readiness summary: Ready now: 5/6 with a per-URL table, a "Needs attention" block, and a "Next:" block naming the exact compose logs command. This is the shape hal0's install summary should have.
  • --summary-json wrote a versioned payload: installer version, tier id and name, runtime.{gpu_backend, backend_service, llm_model, compose_flags, dry_run}, hardware class, preflight report path.
  • Extension manifest validation ran against the installed ODS version: 27 manifests, 0 incompatible, then an extension runtime check probing each enabled non-core service's health URL from the manifest.
  • The generated .env has ~150 keys; .compose-flags caches the resolved stack; config/llama-server/models.ini is four lines; data/ has one directory per service.
Two IPv6 findings (matter for Proxmox LXCs with IPv6 disabled)
  • ods-dashboard: nginx config has listen [::]:3001; unconditionally ("IPv6 support for Docker healthcheck"); on a kernel without IPv6, nginx fails with socket() [::]:3001 failed (97: Address family not supported by protocol) and the container crash-loops. Sandbox fix: comment out the IPv6 listen line in the image.
  • ods-searxng: the upstream image defaults GRANIAN_HOST=::; same failure. Sandbox fix: GRANIAN_HOST=0.0.0.0 in the service env.
  • Both are one-line hardenings ODS could ship (listen [::]:3001 ipv6only=on; guarded, or IPv4 default plus an env toggle). hal0 already binds IPv4 explicitly in its unit templates; keep it that way.
Live surfaces after the fixes
  • llama-server /v1/models → Qwen3.5-2B-Q4_K_M.gguf; a direct chat completion answers in ~20 tokens on 4 vCPU.
  • LiteLLM exposes only the aliases default and *; a completion to ods/current is answered by the router and reports the real model id in the response.
  • hermes-proxy / → 303 → /auth/required ("Owner card required") with no cookie: the forward_auth gate is real.
  • dashboard-api /api/status returns per-service status/state/severity/countsAsIssue plus the manifest's llm contract (Hermes: route: gateway, min_context: 65536, swap_safe: true, with a one-sentence swap_safe_reason).
  • /api/features → 4 feature cards (hermes-agent: services_needed, hermes-sso: services_needed, coding: insufficient_vram, agent-governance: enabled) and two plain-English recommendations ("Limited GPU memory — chat will work with small models", "Consider cloud hybrid mode for better quality"). /api/features/hermes-agent/enable returns numbered steps plus an "Open Hermes" link.
  • /api/extensions/catalog → 33 catalog entries; 24 incompatible on the CPU backend, 9 not_installed. /api/external-links lists 7 sidebar quicklinks derived from manifests.
  • ods list prints 27 services with category (core/recommended/optional) and state (always-on/enabled/disabled); ods status prints the container table, then per-service health checks.
  • The rendered persona (data/persona/SOUL.md) opens with "About this ODS install — read this BEFORE answering questions about your environment": host, backend, serving model and context window, the dashboard URL, the running-services list, and a "you can call these / you can only point the operator at these" split. Two inaccuracies in the generated block: it advertises http://vm.local and chat.vm.local although ods-proxy is disabled in this install, and it expands APE as "agentic prompt engineering surface" (it is the Agent Policy Engine). A generated context is only as good as its description table.

2. What hal0 should take from the install experience (short list)

  1. One capability-profile JSON written once per run, read by everything after it.
  2. Preflight and doctor verdicts as JSON rows with an action string; fixture-tested.
  3. A tee'd install log plus a redacted failure report written from the ERR trap.
  4. Per-step time estimates, named sub-steps, and one line per disabled feature saying why.
  5. Validate image tags before pulling; pull largest first; retry with backoff; validate after pull.
  6. A readiness summary with Ready / Needs attention / Next, and --summary-json.
  7. Bind IPv4 explicitly (hal0 already does); never inherit a lock into a daemon (hal0 must, when it adds background pulls).
01-agents.md Agents and Hermes

Agents end-to-end: ODS vs hal0

Source-level comparison. Every claim below is cited to file:line in the two read-only checkouts (/home/user/ods, /home/user/hal0). Where a doc asserts something the code does not do, it is called out as drift rather than repeated.


A. How ODS does it

A.1 Hermes is a pinned upstream container, not a fork

ODS runs nousresearch/hermes-agent:v2026.6.5 (ods/extensions/services/hermes/compose.yaml:8), version-pinned with an HERMES_AGENT_IMAGE escape hatch and a documented bump ritual (ods/docs/HERMES.md:142-173 + a bump-history table at :284-290). No upstream code is forked; the whole integration is a packaging layer of five files.

The container runs gateway run (compose.yaml:153-155) rather than dashboard, because the cron scheduler tick lives in the gateway main loop — an earlier draft ran only the dashboard and cron jobs silently never fired (compose.yaml:139-152). HERMES_DASHBOARD=1 embeds the React SPA in the same process (compose.yaml:50-52), and HERMES_DASHBOARD_TUI=1 (:87) turns on the PTY-backed chat tab that upstream defaults off.

Network posture is the load-bearing decision: Hermes declares expose: "9119" and no ports: (compose.yaml:137-138), and the manifest pins external_port_default: 0 specifically so dashboard-api cannot advertise a non-existent direct URL (manifest.yaml:19-23). Health polls /api/status because /api/health does not exist in the pinned image and everything else is session-gated (manifest.yaml:24-29, mirrored in the container healthcheck at compose.yaml:156-164). Resource caps, no-new-privileges, log rotation and UID remapping to the host user round it out (compose.yaml:18-19,167-181).

A.2 Config: a template that gets surgically patched

cli-config.yaml.template is bind-mounted over the in-image cli-config.yaml.example (compose.yaml:97), so Hermes's own entrypoint copies ODS's defaults into /opt/data/config.yaml on first start. The template is unusually well-reasoned:

  • model.base_url wins over the OPENAI_BASE_URL env var — empirically verified on macOS where the env var was ignored (cli-config.yaml.template:34-42).
  • model.context_length: 131072 and auxiliary.compression.context_length both set, because Hermes checks the compression model's window against the 64K floor independently (:74-84).
  • model.max_tokens: 1024 to stop a looping local model monopolising the only inference slot (:51-57).
  • providers.custom.request_timeout_seconds: 180 for slow local prefill (:59-70).
  • agent.disabled_toolsets: [terminal, browser] — a schema-level subtraction, chosen over prompt-only guidance because removing tools from the schema is the only reliable fix for wrong-tool selection on small models (:86-104).
  • Compression tuned to threshold 0.75 / target_ratio 0.50 / protect_last_n 40 after live testing showed a single 10-15 kB web-search result spiking past a 0.50 threshold and compressing away the conversation (:153-186).
  • WhatsApp pre-seeded disabled with bridge_port: 3010 purely to dodge upstream's port-3000 bridge colliding with Open WebUI (:121-151).

Because Hermes only copies the template on first start, upgrades need a second writer: scripts/patch-hermes-config.py is a line-oriented YAML patcher that converges only ODS-owned keys and leaves operator sections untouched (patch-hermes-config.py:1-8). It has real nuance — _ensure_provider_timeout only overwrites ODS's own shipped 180s default, never an operator's value (:120-156); _ensure_whatsapp_bridge never flips an existing enabled state (:220-262); _ensure_compression does converge every install to the tuned values so existing boxes migrate on the next bootstrap-upgrade (:186-215).

Installer phase 11 drives it: it picks the model name per mode (switchboard alias ods/current, Lemonade, external, or extra.$GGUF_FILE), picks base_url (litellm on AMD/switchboard, llama-server otherwise), passes --api-key when routing through LiteLLM, then verifies the substitution took with a grep -Fqx and warns loudly if it did not (installers/phases/11-services.sh:992-1094).

A.3 The 64K context floor

HERMES_MIN_CONTEXT = 65536 is a shared constant (extensions/services/dashboard-api/context_policy.py:3), declared per-service in the manifest (hermes/manifest.yaml:37-45 — llm.min_context: 65536, route: gateway, plus a custom probe against /api/talk/message/stream with auth: cookie:dream-session), consumed by the dashboard readiness surfaces (dashboard-api/main.py:537-587, performance_oracle.py:1499), and — crucially — enforced by raising the whole box's context at install: installers/phases/03-features.sh:93-102 bumps MAX_CONTEXT to 65536 when Hermes is enabled and appends the reason to the model recommendation. macOS has the same logic (installers/macos/install-macos.sh:1534-1539).

A.4 SOUL.md and the generated "About this installation" block

SOUL.md.template is a consumer-grade persona written for text-to-speech delivery — length budgets, "speak in sentences not lists", literal-output honouring, and hard tool-selection rules that exist because operators watched models stall chat for minutes reaching for a shell on weather questions (SOUL.md.template:103-120). It contains an eight-step script teaching the agent to drive OAuth skill setup end-to-end (:26-60).

The single line <!-- INSTALLATION_CONTEXT --> (:3) is replaced at build time by scripts/build-installation-context.py. That script builds a {container_name: service_id} index by regexing every extension manifest (:86-127), runs docker ps --filter name=ods- and collapses compound containers (ods-langfuse-clickhouse → langfuse) to parent service ids (:130-168), best-effort asks llama-server/Lemonade what model is actually loaded (:171-188), and renders a bounded Markdown block that explicitly tells the model not to run tool calls to second-guess it (:277-297). It also distinguishes the ODS dashboard from Hermes's own :9119 because the model has a strong prior to name the wrong one (:270-287). A local-lemonade profile emits a much shorter variant for prompt-constrained backends (:328-377), and build_soul self-heals the pathological case where Docker auto-created data/persona/SOUL.md as a directory (:406-422).

Refresh path: ods-cli's _ods_cli_refresh_soul regenerates the file and then docker exec ods-hermes cp /opt/hermes/docker/SOUL.md /opt/data/SOUL.md (ods-cli:1691-1719) — a container-side copy chosen because macOS Docker Desktop's virtiofs refuses nested bind mounts (compose.yaml:98-127). It runs both before compose (to repair a directory-shaped SOUL.md that would fail the mount) and after (so docker ps sees real state) — ods-cli:1582-1612, mirrored at installers/phases/11-services.sh:1108-1123 and :1293-1300.

hermes-proxy is a 50 MB Caddy sidecar (hermes-proxy/compose.yaml:6). Its Caddyfile keeps /health, /healthz, /favicon.ico and /auth/required* public inside an explicit route {} block (order matters — global directive order would otherwise run forward_auth first), then forward_auths everything else to dashboard-api/api/auth/verify-session (hermes-proxy/Caddyfile:54-129). Two hard-won details: an earlier draft only checked that the cookie header existed, trivially bypassable (:8-12); and the WebSocket upgrade headers must be stripped from the auth sub-request or FastAPI 403s a valid session (:104-119). Denials 303-redirect (not 302) so a no-cookie POST does not replay its body (:96), and the redirect needs an explicit * matcher or Caddy misparses it (:123-127).

The credential itself is a stateless HMAC cookie: <random-id>.<expiry-epoch>.<base64url-HMAC-SHA256> (dashboard-api/session_signer.py:1-45), verified with hmac.compare_digest and an independent server-side expiry check (:125-171). An unset ODS_SESSION_SECRET disables issuing entirely rather than silently signing with an empty key (:60-66,109-114). verify-session is deliberately not API-key-gated (the cookie is the credential) and returns a byte-identical 401 for every failure mode (routers/auth.py:65-107). POST /api/auth/admin-session trades DASHBOARD_API_KEY for an identical cookie so the box owner is not locked out of their own services (routers/auth.py:110-180).

Cookies are minted by magic-link redemption (routers/magic_link.py:1-46): token_urlsafe(32), only the SHA-256 hash persisted, single-use by default, 60-minute guest expiry, per-IP rate limit on redemption, and identical 404s for invalid/expired/redeemed so a holder cannot fingerprint state. The dashboard's /invites page exposes owner cards and scoped guest invites (chat vs hermes) with QR printing (dashboard/src/pages/Invites.jsx:23-31).

A.6 ODS Talk: the server-side bridge that is the best code in the repo

Phones never see Hermes. dashboard-api/hermes_bridge.py scrapes the dashboard token out of Hermes's HTML (TOKEN_RE, :43, _fetch_hermes_token:99-113), opens the JSON-RPC WebSocket on the Docker network, and returns only simplified results. Two architectural findings are baked in as comments:

  • Hermes scopes streaming events to the WS that owns the session, so session.create and prompt.submit must happen on the same socket (hermes_bridge.py:9-15).
  • Therefore a per-cookie connection pool keeps one long-lived WS + session per phone, so llama-server's KV cache stays warm for the ~16 k-token agent system prompt — the second message costs ~1 k tokens of prefill, not 17 k (:17-25, _HermesConnection:162-176).

The pool is genuinely careful: _POOL_GUARD is held only for dict bookkeeping while the network open happens under a per-key lock, so one cold open cannot head-of-line-block every other phone (:236-270); an idle sweeper evicts after ODS_TALK_IDLE_EXPIRY (300 s) but skips connections whose lock is held (:279-311); and exactly one failure mode retries transparently — HermesConnectionStale, raised only when send_str fails before Hermes accepted the prompt, so a retry can never duplicate tool calls (:63-73, :471-528).

routers/talk.py wraps that in SSE with a 5 s keepalive comment frame (iOS Safari closes idle streams during 30-60 s cold prefill) and client-disconnect cancellation so an abandoned tab does not pin an inference slot (:444-460). A _TOOL_LABELS table maps Hermes tool names to spinner captions with an honest Using `<name>`… fallback (:404-442). /api/talk/status composes Hermes, whisper and TTS health plus model compatibility into a capabilities dict including live_mic_requires_secure_context (:560-590).

A.7 OAuth passthrough

routers/oauth_passthrough.py closes the "copy the code from your URL bar" gap. POST /api/oauth/init mints a nonce bound to {skill_id, return_url}; the provider redirect lands on the unauthenticated /api/oauth/callback, which validates and consumes the nonce before writing {code, state, ts} to data/persona/oauth_callback.json for the agent to pick up (:1-66). The return_url is bound at init, eliminating the open-redirect surface, and skill_id is resolved server-side so an attacker cannot steer which skill the code is exchanged against (:52-62). oauth-providers.json is the operator- facing registry of flows, credential filenames and acceptable redirect URIs, with an explicit "prefer PKCE / device flow so local appliances need no shared secret" stance (oauth-providers.json:22-51).

A.8 The rest of the agent surface
  • dashboard-api/routers/agents.py is a red herring — 77 lines of GPU/ session/throughput metrics (:14-77) backed by agent_monitor.py, which itself admits tokens_per_second and queue_depth have no data source (agent_monitor.py:27-30). Real Hermes control lives in talk.py, auth.py, magic_link.py and the Hermes SPA itself.
  • APE (extensions/services/ape/main.py) is a competent standalone policy gateway: intent classification, allowlists, path guards, deny regexes, sliding windows (5 m/1 h/1 d), a circuit breaker, warmup grace, persisted state under a file lock, and require_approval escalation that deliberately does not raise so a framework can route to a human and retry via /approve (main.py:1-56,187-200,793-1072). It is not wired to Hermes (see E).
  • ods/agents/templates/ is five OpenClaw-era YAML agent templates with a README dated 2026-02-11 (agents/templates/README.md:1-24); nothing outside the files themselves references them.
  • memory-shepherd/ is a systemd-timer baseline-reset tool with a genuinely good idea (operator-owned baseline above a --- separator, scratch below, archived and reset every 3 h — README.md:14-51) but its shipped config still points at config/openclaw/workspace/*.md (memory-shepherd.conf:19-35). It is not wired to Hermes.
  • OpenCode runs as a host user unit on loopback 3003, deliberately UnsetEnvironment=OPENCODE_SERVER_PASSWORD (opencode/opencode-web.service:16-19), configured by phase 07 which writes opencode.json pointing at LiteLLM (with LITELLM_KEY) on Lemonade or llama-server direct (no-key) otherwise, with a jq-rewrite path and a deterministic fresh-write fallback (installers/phases/07-devtools.sh:140-219). The generated-config writer matrix in docs/INSTALLER-ARCHITECTURE.md:118-135 lists every place each surface is written across Linux/macOS/Windows/upgrade — the single most transferable process artifact in the repo.
A.9 ODS doc drift found
  1. docs/HERMES.md:110 states compression threshold 0.50 / target_ratio 0.20. The shipped template is 0.75 / 0.50 / 40 (cli-config.yaml.template:171-186) and the patcher converges to the same (patch-hermes-config.py:207-212).
  2. docs/HERMES.md:115 claims "4 CPUs / 4GB RAM hard limit". The compose default is cpus: ${HERMES_CPU_LIMIT:-1.0} (compose.yaml:170).
  3. docs/MIGRATION-OPENCLAW-TO-HERMES.md:28 claims an "APE policy plugin (pre_tool_call hook) routes every tool call through ODS's policy engine". pre_tool_call appears nowhere else in the repository — that doc line is its only occurrence — and docs/HERMES.md:140 says the opposite ("No APE policy enforcement yet"). The migration table is marketing, not code.
  4. Same table, :27, claims voice is "wired through ODS's whisper + kokoro out of the box". True at the ODS Talk portal layer (routers/talk.py:279-368), not inside Hermes; docs/HERMES.md:182 still lists it as unverified roadmap.
  5. cli-config.yaml.template:20-27 admits the model-name substitution is an "open follow-up" requiring hand-edits on AMD/Apple — but installers/phases/11-services.sh:1068-1094 has shipped it. The comment is stale.

B. How hal0 does it today

hal0 runs Hermes as a pip-installed wheel in a hal0-owned venv under a sandboxed systemd template unit, not a container. BUNDLED_AGENTS is ("hermes", "pi") with single-pick enforced in the manager, not the API or CLI (src/hal0/agents/manager.py:117, :28-31).

Unit. installer/systemd/hal0-agent@.service is Type=notify + WatchdogSec=60 (:21-26,64), User=hal0, NoNewPrivileges, ProtectSystem=strict, ProtectHome, PrivateTmp, the full Protect*/ Restrict* set (:69-77), RuntimeDirectory/LogsDirectory created before the namespace is built (:79-87), and StartLimitBurst=5/120s (:16-19). The ReadWritePaths= comment is a model of honesty: it spells out exactly what the boundary does not buy — /etc/hal0 is hal0-owned 2775, so the agent can rewrite api.env and upstreams.toml, and %i.env was left 0644 on every upgraded box by #1876 (:89-130). The hermes drop-in sets HERMES_HOME, HERMES_DASHBOARD_TUI=1, HAL0_INFERENCE_BASE, and an ExecStartPre=-/usr/local/bin/hal0-agent %i render-context so every restart refreshes live context (override.conf:11-49). HERMES_WEB_DIST is resolved at runtime from the venv's real site-packages after a hardcoded python3.12 path crash-looped the unit on 3.14 hosts (override.conf:26-35).

Shim. src/hal0/cli/agent_shim.py is pure-stdlib by design so it cannot be broken by hal0 wheel import drift, translates serve into hermes dashboard --tui --skip-build --no-open --host 127.0.0.1, and owns the sd_notify handshake (:1-33).

Provisioner. hermes_provision.py is 6 769 lines. It is explicitly not checkpointed any more — idempotency comes from every write being a converging write, and provision.json is a last-run snapshot (:1-21). The pipeline is 12 steps (_INSTALL_STEPS:6583-6595). config_write is the standout: it runs hermes config migrate (Hermes owns and schema-migrates its own file), then layers hal0's keys via hermes config set, then deep-merges only the two irreducible list keys plus an operator overrides.yaml — never a wholesale rewrite (:2290-2308). It snapshots to config.yaml.bak first (:2318-2323) and derives the terminal-tool posture from pre-migrate content so a fresh box cannot be mistaken for an existing opt-in, failing closed on an unrecordable fresh opt-in (:2324-2371).

Context files. _phase_context_link renders SOUL.md, /etc/hal0/AGENTS.md, MCP-CLIENTS.md from Jinja templates with observable fallbacks (:3247-3330). Live state is separated from stable identity: STATE.md.j2 is a deliberately lean volatile snapshot with an _as_of: stamp (STATE.md.j2:1-29), injected every session by inject-system-state.sh, which cats the pre-rendered file and only kicks a detached background refresh when it is older than 300 s — never blocking the session (inject-system-state.sh:20-30). hermes_refresh.spawn_context_refresh fires the same refresh from the daemon on slot swap / capability apply without blocking the event loop (hermes_refresh.py:1-31).

Context floor. anchor_window.py reads Hermes's own MINIMUM_CONTEXT_LENGTH out of the installed venv rather than hard-coding it, falling back to HERMES_MINIMUM_CONTEXT_LENGTH = 64_000 with a drift test (:64-84). It computes effective = min(model window, slot ceiling), asks the by-id route because hal0/agent is a virtual name that may resolve elsewhere, treats "unknown" as not a pass, and is deliberately read-only — it names the slot, its ceiling and the repair command instead of silently rewriting operator config (:1-50).

Discovery. hal0 ships two real Hermes plugins rather than pointing a base URL: hal0-provider is a ProviderProfile advertising hal0-api with live /v1/models discovery (X-hal0-Model-Filter: hal0, alias-stripping, no cache) and default_aux_model = "hal0/agent" so retargeting a role hot-swaps with no gateway restart (installer/agents/hermes/plugins/hal0-provider/profile.py:1-27); hal0-memory is an exclusive memory provider wiring Hermes durable memory to Hindsight through /api/memory/* (plugins/hal0-memory/plugin.yaml).

Chat surface. chat_proxy.py bridges the browser to loopback Hermes over JSON-RPC WS; WS upgrades are gated by an Origin allowlist + HMAC session cookie, the outbound hop carries the runtime.json embed token as a Bearer header (never a query string), tool.progress is coalesced at 100 ms while message.delta passes through untouched, and reconnect is the browser's job (:1-40). _auth.py documents why: an unauthenticated 0.0.0.0:8080 JSON-RPC bridge into Hermes's tool surface is LAN-RCE (:1-33); the cookie is HMAC-SHA256 over a JSON payload with an 8 h TTL and a 0600 secret at /var/lib/hal0/agents/secret.bin (:60-80).

Policy. mcp_client.py is a transport-agnostic policy layer with a three-tier verdict (allow/gated/blocked, plus unknown_server/ unknown_tool) that the wire client consults (:1-45). Gated calls land in ApprovalQueue, surfaced by /api/agent/approvals with an SSE stream that replays the pending set on subscribe (api/routes/approvals.py:1-27). personas.py owns a TOML store with active.txt, conservative default auto-approve/require-approval globs, and a distinct brain-profile memory identity (:1-55).

Operator surfaces. api/routes/agents.py gives list / persona-enums / skills / install / uninstall / activity (:68-347); restart.py, and memory_stats.py back the sidebar chips; api/plugins/manifest_proxy.py reverse-proxies Hermes plugin manifests while stripping Authorization/Cookie inbound, injecting X-hal0-Agent outbound, enforcing SRI hashes and rejecting traversal (:1-36). The v3 dashboard renders agents as trading-card style agent-cards with live health, throughput and restart wired to real endpoints (ui/src/dash/agents/agents-overview.jsx:1-20). Test coverage is substantial: 30 files under tests/agents/, 56 agent-related test modules overall.

B.1 hal0 doc drift found
  1. ARCHITECTURE.md:322-329 describes a "15-phase" pipeline including persona_seed, namespace_register, model_automap and self_report. _INSTALL_STEPS has 12 entries; model_automap is gone and the other three relocated to the hal0-api lifespan (hermes_provision.py:6570-6595).
  2. ARCHITECTURE.md:325 says "Idempotent + checkpointed via …provision.json". The module says the opposite — checkpoints are gone, the file is a snapshot (hermes_provision.py:9-21).
  3. ARCHITECTURE.md:272-274 says BUNDLED_AGENTS = ("pi-coder", "hermes"); code says ("hermes", "pi") (manager.py:117).
  4. The unit and its drop-in both point Documentation= at docs/agents/hermes/SERVICE.md (hal0-agent@.service:8, override.conf:9); that path does not exist in the tree.
  5. installer/agents/hermes/plugins/hal0-memory/provider.py and src/hal0/agents/hermes/plugins/memory_hindsight/provider.py are byte-identical 629-line copies with no visible sync mechanism. hermes_provision.py:976-979 names the installer tree as canonical.

C. Honest scorecard

ODS is genuinely better at:

  1. Auth as a product feature. Signed stateless cookie + forward_auth + magic links + owner cards + admin-session bootstrap is a complete, proxy-agnostic story (session_signer.py, routers/auth.py, routers/magic_link.py, hermes-proxy/Caddyfile). hal0 has one HMAC seam on the chat proxy and explicitly unauthenticated approval endpoints (approvals.py:24-27,46-48) that execute model_pull / slot_delete / config_write. That is the single largest gap. (Update 2026-10-04: #2246 on main, 946c512, closes the LAN case — ADMIN routes, approvals included, now require an admin session/key from any non-loopback caller when the box is LAN-bound, even with require_auth off; the loopback dev-open path remains by design. The product-feature gap — magic links, owner cards, admin-session — stands.)
  2. A mobile chat portal that never exposes the agent. hermes_bridge.py + talk.py is better engineering than anything on either side: per-cookie WS pooling for prompt-cache warmth, one narrowly-scoped transparent retry, SSE keepalive, disconnect cancellation, tool-name spinner captions.
  3. Persona craft. SOUL.md.template encodes hard-won behavioural rules (TTS-shaped output, literal-output honouring, "web_search first, never curl") that hal0's admin-flavoured SOUL.md.j2 does not attempt.
  4. Agent-driven OAuth. oauth_passthrough.py + oauth-providers.json + the SOUL.md script is a complete capability hal0 lacks entirely.
  5. Cross-platform config-writer discipline. The generated-config-writers matrix (docs/INSTALLER-ARCHITECTURE.md:118-135) is a process artifact hal0 would benefit from copying verbatim.
  6. Doc density for operators — bump history, troubleshooting per symptom, explicit "known limitations" tables (HERMES-SSO.md:129-146).

hal0 is genuinely better at:

  1. Provisioning rigour. Converging writes over checkpoints, config.yaml.bak, hermes config migrate + config set (working with upstream's own config surface instead of regexing YAML), operator overrides.yaml, and repair-mode ownership reconciliation. patch-hermes-config.py is a clever hack; hal0's config_write is the right answer.
  2. Context-floor correctness. Reading the floor from the installed Hermes, resolving the effective window through the by-id route, and refusing to treat unknown as a pass (anchor_window.py:1-50) is strictly better than ODS's hard-coded 65536 and auto-bump of MAX_CONTEXT.
  3. Live-state injection. A lean STATE.md refreshed on restart, on slot swap, and lazily on a 300 s TTL beats regenerating a large SOUL.md and docker exec cp-ing it.
  4. Sandboxing. The systemd unit's Protect* set plus the brutally honest ReadWritePaths comment beats a container with no-new-privileges and full bridge-network egress (docs/HERMES.md:139).
  5. Deep integration. Real Hermes plugins (provider + memory) with restart-free role aliases and live model discovery, versus ODS's static model.default string that needs an installer patcher.
  6. Policy that is actually wired. The MCP allow-list → approval-queue path is real code; APE is a fine engine attached to nothing.
  7. Test coverage. 30 test modules under tests/agents/, including idempotency, upgrade, ownership and security-deliverable suites.

Equivalent: image/wheel pinning with human-gated bumps (HERMES.md:142-173 vs [tool.hal0.upstream-hermes] + the weekly drift action); dashboard agent cards; per-agent restart; upstream-owns-runtime posture.


D. Port candidates

D1. Signed-cookie session + verify-session + admin-session (highest value). Source: ods/extensions/services/dashboard-api/session_signer.py (171 lines), routers/auth.py (180 lines). Target: src/hal0/api/auth/session_signer.py + src/hal0/api/routes/auth.py. hal0 already has HMAC cookie machinery in api/agents/_auth.py:60-120 — the port is generalising it off the agent namespace and adding the two endpoints so any reverse proxy can gate on it. Shape worth copying verbatim:

def verify(cookie_value: str) -> Tuple[bool, str]:
    if not _SECRET:                      return False, "no-secret"
    parts = cookie_value.split(".")
    if len(parts) != 3:                  return False, "malformed"
    random_id, expiry_str, claimed_sig = parts
    if not hmac.compare_digest(_sign(f"{random_id}.{expiry_str}").encode(),
                               claimed_sig.encode()):
        return False, "bad-signature"
    if int(expiry_str) <= int(time.time()):  return False, "expired"
    return True, "ok"

Size ~250 lines + tests. Dependencies: none (stdlib). Risk: low. Immediate win — put /api/agent/approvals/* behind it.

D2. Caddy forward_auth sidecar. Source: hermes-proxy/Caddyfile:54-129, compose.yaml, auth-required/index.html. hal0 has no container layer, so port the Caddyfile and run Caddy as another systemd unit in front of hal0-api on the LAN port. The three non-obvious lines to copy exactly: the explicit route {} wrapper, the six header_up -Sec-Websocket-*/-Upgrade/-Connection strips, and redir * /auth/required 303. Size ~150 lines of config. Depends on D1. Risk: medium (changes the LAN trust model — see F).

D3. OAuth passthrough + provider registry. Source: routers/oauth_passthrough.py (570 lines), hermes/oauth-providers.json, SOUL.md.template:26-60. Target: src/hal0/api/routes/oauth.py + installer/agents/hermes/oauth-providers.json + an addendum to hermes_templates/SOUL.md.j2. Size ~600 lines + a persona section. Dependencies: a shared operator-owned directory both hal0-api and the agent can read — hal0 already has /var/lib/hal0 and both run as hal0, which is simpler than ODS's cross-container arrangement. Risk: low-medium; keep the nonce-consumed-before-write ordering and the init-bound return_url.

D4. The pooled WS bridge pattern. Source: hermes_bridge.py:162-528. Even though hal0's chat_proxy.py proxies the browser's own WS rather than pooling server-side, the findings transfer directly: same-WS session create+submit, warm-prompt-cache pooling, per-key opening locks, and — most importantly — the HermesConnectionStale discipline of retrying only before prompt.submit lands. If hal0 ever adds a REST/SSE chat surface (a phone portal), this is the blueprint. Size ~400 lines. Risk: medium.

D5. Tool-name → spinner caption table. Source: routers/talk.py:404-442. Target: ui/src/dash/agents/ or the chat proxy. ~40 lines, zero dependencies, risk none. Copy the honest fallback:

return f"Using `{tool_name}`…"

D6. Generated-config-writers matrix. Source: docs/INSTALLER-ARCHITECTURE.md:118-135. Target: a section in hal0's ARCHITECTURE.md or CONTRIBUTING.md listing every writer of config.yaml, hermes.env, the gateway drop-in, personas and runtime.json across install / reprovision / repair / upgrade. Documentation only; risk none; high value given hal0 has at least four writers of Hermes state.

D7. ods doctor / ods repair hermes-workers pattern. Source: scripts/ods-doctor.sh:479-489, scripts/prune-hermes-slash-workers.sh, policy env knobs HERMES_SLASH_WORKER_MAX_COUNT/MAX_AGE_SECONDS. hal0's watchdog restarts a hung agent but nothing prunes leaked children. Size ~120 lines. Risk: low. Explicitly manual, never automatic — copy that stance too.

D8. Persona voice guidance. Source: SOUL.md.template:18-24,77-83,103-120. Target: a voice/concise persona TOML in hal0's persona store rather than overwriting the admin SOUL.md.j2. ~80 lines of prose. Risk none.


E. Do not copy

  • ods/extensions/services/ape/. The engine is decent, but it is wired to nothing — pre_tool_call appears only in a doc, and docs/HERMES.md:140 concedes there is no enforcement. Its default WriteFile.allowed_paths still names /home/node/.openclaw/workspace (ape/main.py:127-140). hal0's mcp_client.py three-tier classifier plus ApprovalQueue already occupies this slot and is actually on the call path. Cherry-pick only the ideas — sliding-window tiers, circuit breaker, require_approval as advisory non-raising escalation (main.py:36-52).
  • ods/agents/templates/*.yaml. OpenClaw-era, unreferenced by any code, validated once in Feb 2026 against a model neither project ships.
  • memory-shepherd/ as shipped. The baseline/---/scratch pattern is worth stealing conceptually, but the shipped config targets OpenClaw workspace paths (memory-shepherd.conf:19-35) and the whole thing is orthogonal to Hermes's own memory. hal0's Hindsight namespaces already solve durable memory; bolting on a periodic reset would fight it.
  • patch-hermes-config.py's regex YAML surgery. hal0's hermes config set + targeted deep-merge is strictly better. Do not regress.
  • dashboard-api/routers/agents.py + agent_monitor.py. Metrics with no data source (agent_monitor.py:27-30) rendering an HTML fragment from an f-string. hal0's /api/agents + memory_stats + activity are better.
  • Hard-coding HERMES_MIN_CONTEXT = 65536 (context_policy.py:3) and auto-raising MAX_CONTEXT at install (03-features.sh:93-102). hal0's read-it-from-the-installed-Hermes + refuse-to-mutate posture is the correct one; adopting ODS's would be a regression.
  • HERMES_DASHBOARD_INSECURE=1 (compose.yaml:59). Only defensible because 9119 is unbound. hal0's Hermes is already loopback-bound and does not need it.
  • Scraping window.__HERMES_SESSION_TOKEN__ out of HTML (hermes_bridge.py:43). hal0 has a real runtime.json embed token; regex-ing the SPA is a fragile fallback, not a pattern.

F. Open questions for the owner

  1. Container Hermes vs pip Hermes. ODS gets a hermetic multi-GB image with playwright/ffmpeg/chromium and one-line rollback; hal0 gets systemd sandboxing, no Docker dependency, and direct loopback access, at the cost of a git clone + venv build and an OS toolchain preflight (installer/agents/hermes-prereqs.sh:1-40). Recommendation: keep pip. The container's isolation is weaker than the unit's ProtectSystem=strict, and ODS itself had to punch HERMES_DASHBOARD_INSECURE=1 through it.
  2. Trusted-LAN vs authenticated. hal0's standing decision is "LAN trust plus an upstream reverse proxy own authentication" (ARCHITECTURE.md:405-408), yet /api/agent/approvals/{id}/approve executes gated tools unauthenticated (approvals.py:24-27). Either the standing decision needs a carve-out for mutation endpoints (D1 alone), or hal0 adopts the full ODS posture (D1+D2). This is the decision that gates D2, D3 and D4. (#2246 answered the first half on main with a posture-coupled gate for LAN-bound boxes rather than a blanket carve-out; the D2–D4 question is untouched.)
  3. Do you want a phone surface? ODS Talk (magic link → mobile portal → pooled bridge → TTS) is roughly three of the port candidates and the single biggest feature difference. If the answer is no, D2/D3/D4 drop to nice-to-have and D1 alone is sufficient.
  4. Whose persona wins? ODS's SOUL is a consumer voice assistant; hal0's is a homelab admin. Porting D8 as a persona option rather than a replacement keeps both, but someone must decide the default.
  5. Policy engine: build up mcp_client or adopt an APE-shaped sidecar? Recommendation: extend mcp_client.py with APE's window tiers and circuit breaker rather than running a second service.
  6. Fix hal0's own drift first. The five items in B.1 — especially the duplicated 629-line memory-provider tree and the stale 15-phase description — will mislead anyone doing this porting work. They are cheap to fix and should land before any port.
02-extensions.md Extension system

ODS extension system → hal0: a source-level comparative study

Scope: the ODS extension mechanism (/home/user/ods/ods/), verified against source, and a concrete proposal for expressing the same shape in hal0's systemd/podman world (/home/user/hal0/). All line numbers are from the trees as read on 2026-09-05.


A. How ODS does it (mechanism-level)

A.1 The unit of extension is a directory with a manifest

Every ODS service — bundled or user-installed — is one directory whose only mandatory contents are manifest.yaml and (for non-core services) compose.yaml. /home/user/ods/ods/extensions/README.md:51 states the contract plainly:

The core contract is simple: a service manifest describes what the service is, and a compose fragment describes how it runs. The registry, CLI, dashboard, health checks, and compose resolver discover the service from those files.

That is literally true in code. There are four independent walkers over extensions/services/*/ (plus data/user-extensions/*/), each doing its own manifest read, none of them requiring registration anywhere else:

Walker File:line What it produces
Bash service registry /home/user/ods/ods/lib/service-registry.sh:5,50 14 associative arrays: aliases, container names, compose paths, categories, deps, health paths/timeouts, ports/port-envs, host-network flag, display names, setup hooks, GPU backends
Compose resolver /home/user/ods/ods/scripts/resolve-compose-stack.sh:517 the -f a.yml -f b.yml … flag string for docker compose
Dashboard API /home/user/ods/ods/extensions/services/dashboard-api/config.py:297 SERVICES dict (health checks, quicklinks) + FEATURES list
Catalog generator /home/user/ods/ods/scripts/generate-extensions-catalog.py:123 config/extensions-catalog.json for the dashboard Extensions page

All four gate on the same sentinel, schema_version: ods.services.v1 (service-registry.sh:136, resolve-compose-stack.sh:540, config.py:336), and all four accept manifest.yaml | manifest.yml | manifest.json (service-registry.sh:123-127, resolve-compose-stack.sh:524-528, config.py:319).

A.2 The manifest contract (extensions/schema/service-manifest.v1.json)

Root: schema_version (const), optional compatibility{ods_min,ods_max}, service (required), features[], tags[]. The schema is additionalProperties: true throughout, so forks add x_-prefixed fields without breaking validation (docs/SERVICE_MANIFEST_V2_PLAN.md:79-82). Its library copy (extensions/library/schema/service-manifest.v1.json) is byte-identical — verified by diff — and kept so by scripts/sync-manifest-schema.py.

Per-field consumers (the load-bearing ones):

  • id — registry key, compose-service name, CLI arg, container-name default ods-<id> (service-registry.sh:158), directory name (enforced by scripts/audit-extensions.py:463 service-id-directory-mismatch).
  • aliases[] — CLI shorthand; sr_resolve() also strips a leading ods- so a name copied out of docker ps resolves (service-registry.sh:263-269).
  • port / external_port_env / external_port_default — SERVICE_PORTS seeded from the manifest, then overridden from .env by sr_resolve_ports() (service-registry.sh:239-247); the same resolution in Python at config.py:371-377.
  • health / health_timeout — the generic probe URL is built as http://127.0.0.1:${port}${health} in scripts/health-check.sh:159-188.
  • category (core|recommended|optional) — core means "defined in docker-compose.base.yml, no compose fragment, cannot be disabled" (ods-cli:2762, ods-cli:2879).
  • compose_file — relative path; the resolver rejects one that escapes the extension dir (resolve-compose-stack.sh:418-426) and treats a .disabled suffix as "off".
  • depends_on[] — drives the enable-cascade prompt (ods-cli:2805-2831), the reverse-dependent warning on disable (ods-cli:2881-2902), and installer-time validate_service_dependencies (installers/phases/11-services.sh:1133).
  • gpu_backends[] — the resolver skips a manifest whose backends don't include the detected one, with all/none as wildcards (resolve-compose-stack.sh:544-547); ods enable warns-and-prompts instead of refusing (ods-cli:2768-2803).
  • hooks.{pre,post}_{install,start,uninstall} / legacy setup_hook — SERVICE_SETUP_HOOKS prefers hooks.post_install (service-registry.sh:196-203); installer phase 11 runs every non-empty hook with $1=INSTALL_DIR $2=GPU_BACKEND (installers/phases/11-services.sh:1374-1386). Live example: extensions/services/langfuse/hooks/post_install.sh chowns bind mounts for the uid-70 postgres / uid-101 clickhouse images.
  • startup_check / startup_timeout — one-shot CLI extensions whose container exits cleanly (service-registry.sh:186, dashboard _is_one_shot_extension, routers/extensions.py:317-325).
  • host_network — services with network_mode: host have no mapped port and no HTTP health; the flag switches off both checks (service-registry.sh:38-42, audit-extensions.py:489-503).
  • ui_path / external_link / public_url_env — the dashboard sidebar quicklink, built entirely from manifests at extensions/services/dashboard-api/main.py:1567-1583.
  • llm{consumes,route,pinning,min_context,probe} — the swap-safety contract (docs/SWAP-SAFE-EXTENSIONS.md:38-72). route: gateway means http://litellm:4000/v1 + model ods/current; the schema conditionally requires route, pinning and probe once consumes: true.
  • features[] — dashboard feature cards. Requirements split into runtime health (requirements.services / services_any / vram_gb) and enablement (enabled_services_all / enabled_services_any), plus launch{type,service,path} and priority.

Supporting docs: docs/EXTENSIONS.md (authoring + runtime lifecycle, 514 lines), docs/COMPOSE_RESOLVER_CONTRACTS.md:38-58 ("Adding A Service" = exactly two files), docs/SWAP-SAFE-EXTENSIONS.md (LLM tiers), docs/EXTENSION-PR-BRANCHING.md:9-14 (core vs. library branch targets), docs/BUILD-ON-ODS-SERVER.md:37-53 (fork guidance), docs/SERVICE_MANIFEST_V2_PLAN.md (planning note only; v1 is the shipped schema).

A.3 The compose layer model

scripts/resolve-compose-stack.sh is 772 lines, of which ~700 are one embedded Python program. Order of assembly:

  1. Base + hardware overlay (lines 54-122): a cascade selecting docker-compose.base.yml plus one of nvidia|amd|apple|arc|intel|cpu|cloud| lemonade-external, keyed on --tier, --gpu-backend, --ods-mode, and env (LEMONADE_EXTERNAL, EXTERNAL_LLM_URL).
  2. Multi-GPU overlay (143-146): docker-compose.multigpu-<backend>.yml when --gpu-count > 1.
  3. Bundled extensions (517-586): for each extensions/services/<id>/ with a v1 manifest and a compatible gpu_backends, append compose.yaml, then — by filesystem probe, not by manifest — compose.<backend>.yaml (563), compose.local.yaml (578, only in local/hybrid mode, non-Apple, non-external-LLM), compose.multigpu-<backend>.yaml (584).
  4. User extensions (607-725): same walk over data/user-extensions/*/, but every fragment is content-scanned first by _scan_user_compose_content (resolve-compose-stack.sh:270-406) — rejecting privileged, build:, user: root, host net/pid/ipc/userns, dangerous caps, seccomp:unconfined, devices:, GPU reservations, docker-socket mounts, absolute bind mounts, extra_hosts, sysctls, reserved com.docker.compose.* labels, non-loopback port bindings, and any service name colliding with config/core-service-ids.json.
  5. docker-compose.external-llm.yml (730-735) and docker-compose.override.yml (742-748, also content-scanned).

Extension composes carry no networks: block for the bundled set (see extensions/services/searxng/compose.yaml) — they land on the compose project's default network and are reachable by their compose service name. The library template does declare networks: [ods-network] with external: true (extensions/library/templates/compose-template.yaml:64-92).

ods-cli caches the resolved string in $INSTALL_DIR/.compose-flags and revalidates every referenced path before using it (ods-cli:852+); _regenerate_compose_flags (ods-cli:755) re-runs the resolver whenever an extension is enabled or disabled.

A.4 Enable/disable is a file rename

ods enable <id> renames compose.yaml.disabled → compose.yaml; ods disable stops the container then renames back (ods-cli:2732-2920, dispatched at ods-cli:6567-6568). There is no state database. ods list derives status by asking whether the compose path exists and appears in the currently active flag string (ods-cli:3004-3065). The dashboard API does the same rename under flock (docs/EXTENSIONS.md:437-444), and config.py:329-333 skips any extension directory containing compose.yaml.disabled so a disabled service vanishes from health checks and feature recommendations without touching its manifest.

.env feature flags (ENABLE_VOICE, ENABLE_WORKFLOWS, ENABLE_COMFYUI, …) are a second, installer-only gate that works by performing the same rename. installers/phases/03-features.sh:104-131 is explicit about why:

### Without this sync, an extension's compose.yaml is ALWAYS picked up by
### resolve-compose-stack.sh regardless of the ENABLE_* flag — the flag then
### only gates cosmetic things (image pre-pull, health checks, summary URLs)
### and the service still starts. Every optional service must be listed here
### or the user can't opt out of it.

The 21 _sync_extension_compose calls at lines 179-204 are hand-maintained. This is the single biggest seam in an otherwise auto-discovering system.

A.5 Ports

config/ports.json is not a runtime input. Runtime ports come from the manifest plus .env. ports.json exists only for tests/contracts/test-port-contracts.py, which cross-checks each declared entry against .env.schema.json defaults, .env.example defaults, the manifest's external_port_env/external_port_default, and the actual compose port mapping (lines 106-195). Because the test iterates ports.json entries — not manifests — a new extension with a new port env var passes CI without being added. Adding it is opt-in rigour for bundled services.

A.6 Health checks

scripts/health-check.sh is fully registry-driven: test_service() (line 159) reads SERVICE_PORT_ENVS, SERVICE_PORTS, SERVICE_HEALTH, SERVICE_HEALTH_TIMEOUTS, checks the container state via docker inspect first, then curls http://127.0.0.1:<port><health>. It loops all core services (line 304) and all enabled extension services (369, gated on -f "$ext_compose") in parallel; a core failure is critical (exit 2), an extension failure is degraded (exit 1).

Installer phase 12 (installers/phases/12-health.sh) is the exception: it does sr_load + sr_resolve_ports (lines 23-30) and pulls ports/paths/container names from the registry, but each probe is a hand-written line gated on an ENABLE_* flag (e.g. line 616 for n8n). A new extension gets no installer-phase probe for free; it does get scripts/health-check.sh and the dashboard's /api/services for free.

A.7 Dashboard surface
  • GET /api/features (routers/features.py:113) computes per-feature status from manifest features[] + live service health + GPU VRAM. Vocabulary (features.py:70-77): enabled (all enabled_services_all healthy) → insufficient_vram (total VRAM < requirements.vram_gb) → services_needed (required services not healthy) → available. It also emits suggestions[] ("Your hardware can run X. Enable it?" / "X needs Y to be running.", lines 130-144) and coarse recommendations[] bucketed by VRAM (162-177).
  • GET /api/extensions/catalog (routers/extensions.py:1090) serves the static config/extensions-catalog.json enriched by _compute_extension_status (extensions.py:328-402): installing / setting_up / error / cli_installed / enabled / unhealthy / stopped / disabled / incompatible / not_installed. The React page renders exactly that vocabulary (extensions/services/dashboard/src/pages/Extensions.jsx:60-80,336-337).
  • GET /api/external-links (main.py:1567) turns every manifest with an external port into a sidebar quicklink. A new extension appears in the sidebar with no UI change.
  • The dashboard plugin registry (dashboard/src/plugins/registry.js) is the only place needing a code edit — and only for an extension that wants its own internal dashboard route, not for a quicklink.

Install flow for catalog extensions: POST /api/extensions/{id}/install copies from $ODS_DATA_DIR/extensions-library/<id>/ to $ODS_DATA_DIR/user-extensions/<id>/ after a 50 MB size check and the compose security scan, then asks the host agent (bin/ods-host-agent.py, POST /v1/extension/start) to bring it up (docs/EXTENSIONS.md:425-455).

A.8 Pre-wiring: what an extension author must actually know

Four conventions, and only four:

  1. Docker DNS = compose service name. open-webui reaches search as http://searxng:8080/search?q=<query>&format=json (docker-compose.base.yml:117), embeddings as http://embeddings:80/v1 (127), STT as http://whisper:8000/v1 (159), TTS as http://tts:8880/v1 (164).
  2. LLM = the gateway alias. OPENAI_BASE_URL=http://litellm:4000/v1, model ods/current, key ${LITELLM_KEY} (docs/SWAP-SAFE-EXTENSIONS.md:17-33). Direct routes are permitted but must declare pinning: dynamic + a probe. In practice several bundled services take the direct route with a documented fallback chain — extensions/services/perplexica/compose.yaml:13: OPENAI_BASE_URL=${HERMES_LLM_BASE_URL:-${LLM_API_URL:-http://llama-server:8080}/v1}.
  3. Ports bind loopback with an env override: "${BIND_ADDRESS:-127.0.0.1}:${MY_PORT:-1234}:1234" — enforced for user extensions by the resolver scan (resolve-compose-stack.sh:315-333).
  4. Data lives at ./data/<id>/, config at ./config/<id>/ — see extensions/services/n8n/compose.yaml:24-25.

Templates encode all of this: extensions/library/templates/service-template.yaml (151 commented lines), compose-template.yaml, compose-gpu-swap.yaml, compose-gpu-only.yaml, dashboard-plugin-template.js.

A.9 Three worked examples

searxng — 3 files, 55 lines of contract. manifest.yaml (22 lines: id, alias search, 8080→SEARXNG_PORT:8888, health: /healthz, gpu_backends: [all], category: recommended, no features), compose.yaml (33 lines: pinned image, SEARXNG_SECRET with a :? guard, one ./config/searxng bind mount, loopback port, healthcheck, cpu/mem limits), README.md. Touched by: registry, resolver, ods enable search, generic health check, /api/external-links, ENABLE_SEARXNG (03-features.sh:180), and — as a consumer — docker-compose.base.yml:117 and perplexica/compose.yaml:11. Note the coupling inversion: SearXNG knows nothing about its consumers; they name it.

n8n — 5 files, one feature card. manifest.yaml (54 lines) adds health_timeout: 15, four documented env_vars (two required, one secret), and a features[] block (workflows, icon Workflow, launch.type: service, priority 4). compose.yaml (46 lines) runs as ${ODS_UID}:${ODS_GID} with a custom entrypoint, bind-mounting two helper scripts from the extension dir itself (./extensions/services/n8n/n8n-entrypoint.sh:/opt/ods/…:ro,z). Extra surfaces: the /api/features card, a hard-coded enable-instruction block (routers/features.py:236), an installer probe (12-health.sh:616), an ENABLE_WORKFLOWS sync (03-features.sh:184), a workflow catalog under extensions/library/workflows/n8n/, and a config/n8n mount.

comfyui — the GPU-only pattern. compose.yaml is a 14-line stub services: {} whose only job is to exist so the registry counts ComfyUI as installed; the real definitions live in compose.nvidia.yaml (local build: + GPU reservations), compose.amd.yaml (ROCm image, /dev/dri + /dev/kfd, group_add: [$VIDEO_GID, $RENDER_GID], MIOpen tuning) and two compose.multigpu-* overlays. The manifest's gpu_backends: [amd, nvidia] makes the resolver drop it entirely on CPU/Apple. Non-manifest coupling: 03-features.sh:82-91 force-disables it on Tier 0/1 (shm_size: 8g), 11-services.sh:1167 adds it to the local-build list. It is the most-coupled extension in the bundled set — and still needs no registry edit.


B. Minimal-extension proof

The claim holds. For a service that runs from a pre-built image on the current GPU backend, the true minimal set is:

extensions/services/<id>/manifest.yaml    ~20-32 lines
extensions/services/<id>/compose.yaml     ~30-50 lines

Empirically, the smallest complete shipped examples:

Extension manifest compose other
extensions/library/services/chromadb/ 33 31 README only
extensions/library/services/gitea/ 32 49 README only
extensions/services/searxng/ 22 33 README only

Zero edits to any other file. No registry list, no CLI case statement, no compose include, no UI import, no ports.json row, no .env.schema.json property.

Everything that auto-picks it up from those two files:

  1. lib/service-registry.sh:50 — all 14 lookup tables, on next sr_load.
  2. scripts/resolve-compose-stack.sh:517 — compose -f flag, plus any compose.{nvidia,amd,apple,cpu}.yaml, compose.local.yaml, compose.multigpu-*.yaml siblings, purely by filename.
  3. ods list (ods-cli:3004) — appears with category + enabled/disabled status.
  4. ods enable|disable|purge (ods-cli:2732,2859,2922) — including alias resolution, dependency cascade, GPU-compat prompt, and Dockerfile-build hint.
  5. ods start|stop|restart|logs|shell <id-or-alias> — via sr_resolve/sr_container.
  6. scripts/health-check.sh:369 — parallel extension probe, JSON output, exit code.
  7. extensions/services/dashboard-api/config.py:297 — SERVICES entry ⇒ /api/services, /api/status, per-service health polling.
  8. GET /api/external-links (main.py:1567) ⇒ dashboard sidebar quicklink with icon, port, ui_path, health needle.
  9. GET /api/features (routers/features.py:113) ⇒ a feature card, if features[] present, with status/suggestion/launch.
  10. installers/phases/11-services.sh:1374 ⇒ setup hook execution, if declared.
  11. scripts/audit-extensions.py + scripts/validate-manifests.sh ⇒ CI validation.
  12. scripts/generate-extensions-catalog.py ⇒ catalog row (library dir only).
  13. ods preset save/apply ⇒ captured in extensions.list.
  14. bin/ods-host-agent.py /v1/extension/{start,stop,logs} ⇒ dashboard lifecycle.

What still needs a hand-edit, and why:

Want Extra edit File:line
An ENABLE_X installer opt-out one _sync_extension_compose line installers/phases/03-features.sh:179-204
Blocking installer health gate one _check_health line installers/phases/12-health.sh:~616
Port-contract CI parity (bundled only) config/ports.json + .env.schema.json + .env.example tests/contracts/test-port-contracts.py:106
A local build: for a user extension not possible — the scan rejects it resolve-compose-stack.sh:251-252
An internal dashboard route (not a quicklink) plugin registration dashboard/src/plugins/registry.js:14-20
Feature "how to enable" prose dict entry routers/features.py:224-248
Catalog entry for a bundled service generator only reads extensions/library/services scripts/generate-extensions-catalog.py:33

So: one to two files for a working extension; three to five if you want it to be a first-class bundled service with an installer toggle and CI port parity. The owner's description is accurate.


C. hal0 today: what exists, what's missing

hal0's runtime model is one podman container per inference slot under hal0-slot@<name>.service, rendered as a Podman Quadlet .container file (src/hal0/providers/container.py:490,803), plus a small set of companion units. Mapping ODS concepts onto what exists:

ODS concept hal0 nearest analogue Verdict
extensions/services/<id>/manifest.yaml src/hal0/services/registry.py:65 SERVICES: tuple[ServiceDef, ...] Hardcoded Python, 4 entries (openwebui, comfyui, hermes, hindsight). Same fields as an ODS manifest (id, name, unit, port, probe, public_url_env, actions, mdns, loopback_port, hints) — but no filesystem discovery.
compose.yaml fragment packaging/systemd/hal0-openwebui.service (static unit) or RuntimeLaunchPlan → Quadlet (providers/base.py:163, container.py:803) Two unrelated mechanisms. Slots get a typed, rendered plan; companions get a hand-written .service with a literal podman run.
resolve-compose-stack.sh (none) No layering engine. Hardware selection lives inside ContainerProvider + profiles.toml, not in a merged file set.
ods enable/disable/list hal0 app install/list/uninstall (src/hal0/cli/app_commands.py:33-36) _SUPPORTED = frozenset({"openwebui"}), _APP_UNITS a 1-entry dict. The verb shape is right; the registry behind it is a literal.
Feature/extension registry src/hal0/install/extensions.py:26 EXTENSIONS: list[Extension] Literally named "the first-run Extensions registry (spec §6.4)" — three hardcoded entries (openwebui, comfyui, hermes), with install_extension() an if/elif chain (lines 145-165).
/api/features status vocabulary GET /api/capabilities + GET /api/services Capability cards exist (src/hal0/capabilities/catalog.py) but they group models per capability, not "which optional services can this box run". No insufficient_vram / services_needed / available vocabulary for services.
config/ports.json src/hal0/ports/authority.py hal0 is better here. No stored table; claims are recomputed from slot TOMLs, live runtime snapshots, reserved ports and actual LISTEN sockets, so deletion releases a port atomically.
docker-compose.override.yml Quadlet drop-ins hal0-slot@<t>.container.d/ Exists; explicitly the recommended replacement for the deprecated extra_args (container.py:955).
extensions/library/ optional catalog docs/reference/profile-addons.mdx (.hal0profile.json envelope + index.json) and stacks/portable.py (.hal0stack.json) The distribution pattern already exists — signed-ish JSON envelopes with content checksums, dashboard Import, an addons index served from a site. It carries profiles and stacks; it does not carry services.
Setup hooks hal0.agents.hermes_provision (15 phases), install/extensions.py Bespoke per component.
Dashboard Extensions page ui/src/dash/services.jsx (431 lines), services-card.jsx Registry-driven from GET /api/services and already fail-soft.

Two findings worth calling out:

  1. hal0's UI already wants this. ui/src/dash/dashboard-redesign.jsx:858 and ui/src/dash/services.jsx:4,46 both list n8n in SERVICE_ORDER and ship an n8n icon — but there is no ServiceDef, no unit, and no install path anywhere in src/. The extension-shaped hole is already visible in the product.
  2. The privileged seam is the real constraint. hal0-api runs as User=hal0; every unit write goes through sudo -n hal0-systemctl write-quadlet <token> and the root side allow-lists the body: sections [Unit]/[Container]/[Service]/[Install] in strictly increasing order, exactly 3 [Unit] keys, ~14 [Container] keys, 9 [Service] keys, and [Install] WantedBy=hal0.target only (installer/wrappers/hal0-systemctl:389-516). Unit-file paths are pinned to the hal0-slot@ prefix (src/hal0/system/seam.py:283-312). No Exec*= on the host side, PodmanArgs= restricted to --group-add/--security-opt/--ipc/--ulimit. "Drop in an arbitrary third-party unit fragment" is architecturally refused today, deliberately (#1740, #1759).

Net: hal0 has all the ingredients — a typed launch plan, a validated unit writer, a services registry with the right fields, a portable-envelope distribution pattern, a port authority, a registry-driven Services page, and an app install verb — but they are wired together by hand for four fixed services. There is no manifest, no discovery walk, and no way for an operator or a third party to add a fifth.


D. Proposed hal0 extension model

Design goal, restated in hal0 terms: an extension author writes one manifest; hal0 renders the unit, claims the port, wires the env, registers the service, probes its health, and shows it in the dashboard — with no code change anywhere.

D.1 File layout
/usr/lib/hal0/extensions/<id>/        # shipped with the release (read-only)
/var/lib/hal0/extensions/<id>/        # operator- or catalog-installed
    manifest.toml                     # REQUIRED  (~25-40 lines)
    unit.toml                         # OPTIONAL  quadlet overrides (rare)
    hooks/post_install.sh             # OPTIONAL
    README.md                         # OPTIONAL
/etc/hal0/extensions.toml             # enable/disable state + operator overrides
/etc/containers/systemd/hal0-ext@<id>.container   # GENERATED, never authored (rootful system Quadlet — see Decision 3)

Discovery order mirrors hal0.config.paths conventions (/var/lib wins over /usr/lib, HAL0_EXTENSIONS_DIR overrides both for tests) — exactly the resolution shape already used by hal0.bundles.tiers._candidate_roots (src/hal0/bundles/tiers.py:44-56).

D.2 Manifest schema (manifest.toml, schema_version = "hal0.ext.v1")
schema_version = "hal0.ext.v1"

[extension]
id = "n8n"                  # [a-z0-9][a-z0-9-]*, == directory name
name = "n8n (Workflows)"
category = "optional"       # core | recommended | optional
hal0_min = "0.4.0"          # ≙ compatibility.ods_min

[runtime]
image = "docker.io/n8nio/n8n@sha256:…"   # digest-pinned, like RUNNER_IMAGES
port = 5678                 # in-container;  publish_port defaults to it
network = "bridge"          # bridge | host   (slots use host)
volumes = [ { source = "$DATA/n8n", target = "/home/node/.n8n" } ]
user = "$HAL0_UID:$HAL0_GID"
### devices/group_add gated — see D.6;  [runtime.rocm]/[runtime.vulkan]/[runtime.cpu]
### sub-tables replace ODS's compose.<backend>.yaml overlays.

[health]
kind = "http"               # http | tcp | container | none
path = "/healthz"
timeout_s = 15

[wiring]                                # <- "prewired to existing slots"
openai_base_url_env = "OPENAI_BASE_URL" # ⇒ http://host.docker.internal:8080/v1
openai_api_key_env  = "OPENAI_API_KEY"
model_env           = "OPENAI_MODEL"    # ⇒ "hal0/brain" — never a concrete id
slot_env = { WHISPER_URL = "slot:stt", TTS_URL = "slot:tts", COMFY_URL = "slot:img" }

[[env]]
key = "N8N_PASS"; required = true; secret = true

[[capability]]                          # ≙ ODS features[]
id = "workflows"; name = "Workflow Automation"; icon = "Workflow"
requires_services = ["n8n"]; requires_vram_gb = 0; priority = 40
launch = { kind = "service", service = "n8n" }

[hooks]
post_install = "hooks/post_install.sh"

Near-verbatim ports from ODS (semantics unchanged, TOML instead of YAML, Pydantic instead of JSON Schema): id/name/category/aliases, port/external_port, health+health_timeout, depends_on, env_vars[], hooks{}, compatibility.ods_min → hal0_min, and the whole features[] → [[capability]] block including the enabled_* vs requires_* split and launch{}.

Deliberately dropped: compose_file (no compose), gpu_backends (hal0 derives device class from hardware.json; use [runtime] requires_device = "gpu-rocm" instead), container_uid (hal0 owns data-dir ownership through hal0.install.perms), host_network (a policy decision, not an author's).

D.3 Discovery points (the "everything auto-attaches" contract)
Consumer Implementation ODS counterpart
hal0.extensions.registry.load() walk the two roots, parse+validate manifests, memoise on mtime — a direct Python port of sr_load (lib/service-registry.sh:50) minus the bash-emitting subprocess service-registry.sh
hal0.services.registry.SERVICES becomes builtin_services() + extension_services(); ServiceDef gains source: "builtin"\|"extension" and manifest_path config.py:load_extension_manifests
GET /api/services + Services page zero changes — it already renders whatever the registry returns (src/hal0/api/routes/services.py:140) /api/services
Unit rendering ExtensionProvider.launch_plan(manifest) -> RuntimeLaunchPlan, then the existing _render_quadlet_from_plan (container.py:803) — parameterised for the artifact family (correction 2026-10-04): as written it names every container through slot_container_name() (slots/naming.py:70; ContainerName=hal0-slot-<token>, container.py:826,880), so an extension id equal to a slot token would collide with that slot's podman container; extensions need their own namer (hal0-ext-<id>) threaded through the renderer compose fragment
Unit write SystemCtlSeam.write_quadlet extended to accept the hal0-ext@ prefix (system/seam.py:283), wrapper validate_quadlet_body reused unchanged resolver -f flag
Port claim hal0.ports.authority.collect_claims gains an "extension-config" source config/ports.json (better)
Health the probe field already dispatches http/systemd/comfyui in services_health.py; add tcp/container health-check.sh:159
Capability cards GET /api/capabilities merges [[capability]] rows and computes enabled\|available\|insufficient_vram\|services_needed routers/features.py:70-77
mDNS services/mdns.py already writes hal0-addon-<id>.service per advertised id — extensions inherit it free dashboard quicklinks
Journald logs GET /api/logs?unit=hal0-ext@<id>.service — already generic ods logs <id>
Catalog hal0.extensions.portable — a .hal0ext.json envelope + index.json, modelled byte-for-byte on stacks/portable.py:41 and docs/reference/profile-addons.mdx extensions/library/ + extensions-catalog.json
D.4 CLI verbs

Promote hal0 app from a 1-entry frozenset to the extension registry, keeping the existing verb names (src/hal0/cli/app_commands.py):

hal0 app list                     # id, category, state (enabled|disabled|not-installed|incompatible)
hal0 app install <id>             # render unit → seam write → daemon-reload → enable --now → post_install hook (bundled only; see below)
hal0 app enable|disable <id>      # flips /etc/hal0/extensions.toml, converges the unit
hal0 app uninstall <id>           # disable + remove unit; --purge also drops /var/lib/hal0/ext/<id>
hal0 app import <file|url>        # .hal0ext.json envelope, checksum-verified
hal0 app doctor [<id>]            # manifest validation ≙ scripts/audit-extensions.py

Correction (2026-10-04) — hooks and the trust boundary. post_install as drawn runs on the host, after the unit is enabled, as whoever invoked the verb (hal0 from the API, root from a shell). For an imported manifest that is arbitrary third-party code outside every rootless, capability-drop, read-only-root and mount restriction §E describes. Rule: host hooks are honoured only for bundled extensions that ship in the hal0 repository (already inside the trust boundary); an imported manifest's hooks/ are either refused at hal0 app import or run as a one-shot inside the extension's own container under the same confinement (ExecStartPost= → podman exec), never on the host.

hal0 doctor gains an extensions arm; the ODS audit-extensions.py check list (schema version, id↔directory match, category/type enum, positive port, health starts with /, alias collisions, dangling depends_on, duplicate capability ids) ports almost line-for-line into hal0/extensions/audit.py.

D.5 Pre-wiring, translated

ODS pre-wiring is docker-network DNS + compose interpolation. hal0 has neither: slot containers run Network=host with a loopback fence (container.py:_loopback_fence_command), so there is no service-name DNS. The translation is rendered env, computed at unit-render time from live truth:

  • OPENAI_BASE_URL → http://host.docker.internal:8080/v1 for bridge-network extensions (podman honours --add-host=host.docker.internal:host-gateway, already used at packaging/systemd/hal0-openwebui.service ExecStart), or http://127.0.0.1:8080/v1 for host-network ones. This is exactly what hal0.openwebui.env_writer already does for OpenWebUI (src/hal0/openwebui/env_writer.py:11-30) — generalise that writer.
  • OPENAI_MODEL → hal0/brain (the virtual model that already resolves through the dispatcher), never a concrete model id. This is the direct analogue of ODS's ods/current swap-safety rule and should be documented the same way (docs/SWAP-SAFE-EXTENSIONS.md:17-33 ports nearly verbatim).
  • slot:<name> refs in [wiring].slot_env resolve at render time through hal0.ports.authority. Correction (2026-10-04): the first draft resolved them to http://127.0.0.1:<slot port> unconditionally and claimed the URL "follows the slot if the operator moves its port". Both halves were wrong. (i) Inside a bridge-network extension 127.0.0.1 is the extension's own loopback, and host.docker.internal does not help either: slots are deliberately loopback-only on the host — _loopback_fence_command flips every 0.0.0.0 bind to 127.0.0.1 under Network=host (src/hal0/providers/container.py:619-647; providers/base.py:206 "host port the container listens on (127.0.0.1 only)") — so the host-gateway address cannot reach a slot port at all. A bridge extension must therefore go through hal0-api, which does listen on the LAN interface: slot:stt → http://host.docker.internal:8080 plus the gateway's own route for that modality (POST /v1/audio/transcriptions → the stt slot, routes/v1.py:1398; /v1/images/generations → the img slot). This is exactly how openwebui/env_writer.py:112-128 already wires AUDIO_STT_OPENAI_API_BASE_URL and AUDIO_TTS_OPENAI_API_BASE_URL — at hal0-api :8080, never at a slot. Only a network = "host" extension may be handed http://127.0.0.1:<slot port> directly. (ii) A render-time value does not follow anything: the generated env file keeps the old port, and nothing in the convergence list re-renders dependants when a slot TOML changes. Gateway URLs sidestep this (hal0-api's port is stable and the dispatcher follows the slot); for the host-network direct form, record depends_on_slots in the extension registry and have the slot-config write path (SlotConfigStore) trigger extension convergence — a hook that does not exist today and must ship with the feature. ODS's static DNS names and .env ports have the same staleness problem; the honest claim is "gateway-routed by default, so a slot move is invisible", not "the URL follows the port". (iii) Correction (2026-10-04): host.docker.internal resolves to the host's address on the podman bridge (--add-host=host.docker.internal:host-gateway, env_writer.py:102-105), not to 127.0.0.1. Report 05's gateway work wants the direct ports off the LAN; if that is done by binding hal0-api to loopback alone, every bridge extension — and hal0's own Open WebUI — loses inference, STT, TTS and image routes. The two must be designed together: hal0-api keeps a listener the bridge can reach (a second bind on the podman gateway address, or an authenticated container-facing listener on a shared internal network) while the LAN-facing :8080/:3001 go away, or the LAN cut is an nftables rule that exempts the bridge subnet. HAL0_BIND_HOST is a single value today (config/network.py:37-45), so the two-listener form is new work, not configuration.
  • Env lands in /etc/hal0/extensions/<id>.env, written atomically by hal0.config.env.write_env_atomic (the same primitive slot env files use) and referenced from the generated unit — keeping secrets out of the unit file and out of the podman run argv, which is the lesson of #1466 already recorded in env_writer.py:44-52.
D.6 Near-verbatim ports vs. things needing translation

Near-verbatim (port the logic, change the language): - lib/service-registry.sh:50-235 → hal0/extensions/registry.py. The loader is already a Python program embedded in bash; lifting it out and dropping the _esc() bash-quoting layer is a simplification, not a rewrite. - extensions/schema/service-manifest.v1.json → Pydantic models in hal0/extensions/schema.py. Keep field names that carry meaning (category, health, depends_on, env_vars, hooks, compatibility) so ODS-authored manifests convert mechanically. - routers/features.py:19-110 calculate_feature_status → capability-status computation, including the four-value vocabulary and the requires/enabled split. - scripts/audit-extensions.py:417-747 → hal0/extensions/audit.py. - docs/EXTENSIONS.md §"30-Minute Path" + service-template.yaml → a hal0 write-an-extension.mdx and a heavily-commented manifest.toml template. The commented template is a large part of why ODS's model is usable; do not skip it. - docs/SWAP-SAFE-EXTENSIONS.md tiers 1-3 — hal0's slots load/unload constantly, so this doc is more necessary here, not less.

Needs real translation: - Compose merge → one rendered unit. ODS gets layering free from docker compose -f a -f b. hal0 must merge itself: manifest.toml → RuntimeLaunchPlan → optional operator override via a hal0-ext@<id>.container.d/ drop-in. That mechanism already exists as the documented replacement for extra_args (container.py:947-957), so the story is base manifest + typed drop-in, not N merged files. - GPU overlays. ODS picks compose.<backend>.yaml by filename; hal0 already resolves device class from hardware.json and image from RUNNER_IMAGES. Use [runtime.rocm] / [runtime.vulkan] / [runtime.cpu] sub-tables — one file, not four. - The seam allow-list. installer/wrappers/hal0-systemctl:389-465 must gain the hal0-ext@ prefix and two new [Container] keys (EnvironmentFile=, AddHost=; Environment=/PublishPort=/Volume= are already allowed). Every widening is a root-boundary change needing the scrutiny #1740 got. Highest-risk piece here. - ENABLE_* .env gating → /etc/hal0/extensions.toml. ODS's rename-the-file trick has no analogue and no merit here; use a state file with the atomic write/rollback shape SlotConfigStore already implements (src/hal0/slot_config/__init__.py). - Devices / GPU passthrough. ODS refuses devices: for user extensions outright (resolve-compose-stack.sh:275-276). hal0 should default to the same refusal, with an operator-only escape for locally-authored manifests.


E. Things NOT to copy

  1. Enable/disable by renaming compose.yaml ↔ compose.yaml.disabled. It makes the content tree the state store, so a git pull, a backup restore, or an rsync silently changes which services run. ODS pays for this with the "stale disabled marker" reconciliation at ods-cli:2833-2842. hal0 has SlotConfigStore with atomic commit/revert; use a state file.
  2. The hand-maintained _sync_extension_compose table. 21 lines at installers/phases/03-features.sh:179-204 that must be edited for every new optional service — the exact seam the rest of the design avoids, and the reason the file carries a 10-line comment explaining why omission is a bug. Derive the toggle from category + a state file instead.
  3. Two parallel status vocabularies. /api/features returns enabled|available|insufficient_vram|services_needed; /api/extensions/catalog returns ten different values. The dashboard has to teach users both (Extensions.jsx:73-80 is a legend). Pick one lifecycle vocabulary and one eligibility vocabulary and derive card copy from them.
  4. A generated static catalog JSON checked into the repo. config/extensions-catalog.json is produced by scripts/generate-extensions-catalog.py from manifests and then committed, so it can drift (hence the reminder in docs/EXTENSION-PR-BRANCHING.md:12). Serve it from the live manifest walk.
  5. config/ports.json as a fourth place ports are written. hal0's ports/authority.py docstring already argues the case against stored allocation tables; do not regress.
  6. Bash + embedded-Python + associative arrays. service-registry.sh needs a Bash 4 guard and re-execs under Homebrew bash on macOS (health-check.sh:11-19); the compose resolver hard-fails without PyYAML. hal0 is a Python service — the whole registry is ~150 lines of ordinary Python.
  7. Fine-grained llm.probe in v1. ODS's post-swap probe descriptor (schema:…llm.probe) is genuinely useful but only pays off once you have a swap harness driving it. Ship [wiring] first; add probes when there is a consumer.
  8. Allowing arbitrary unit/quadlet content. ODS's content scan is a deny-list over compose keys (resolve-compose-stack.sh:249-334) — a shape that has to be kept in sync with Docker's evolving schema, and which is duplicated in three places (resolver, dashboard _scan_compose_content, host agent). hal0's allow-list over a rendered unit is strictly stronger. Keep rendering from a typed manifest; never accept a unit body an author wrote.

F. Risks and decisions for the owner

Decision 1 — Are extensions services, or also slots? ODS conflates them: a manifest can describe a core LLM server or a git host. hal0's slot layer is a specialised, heavily-invariant subsystem (GpuArbiter, NPU trio exclusivity, model registry binding). Recommendation: extensions are companion services only (hal0-ext@<id>.service), a strictly separate unit family from hal0-slot@. An extension that wants inference asks hal0 over HTTP like every other client.

Decision 2 — How wide does the seam allow-list open? Every [Container] key added to installer/wrappers/hal0-systemctl:407-438 is a root-boundary change. The minimum new set is EnvironmentFile= (pinned to /etc/hal0/extensions/<id>.env) and AddHost=host.docker.internal:host-gateway (pinned to that literal). Recommendation: pin both to literals rather than charset regexes, and require that Volume= sources for extensions live under /var/lib/hal0/ext/<id>/ — the path constraint ODS cannot make because compose mounts are relative to a project dir the operator owns.

Decision 3 — Rootful vs. rootless for extension containers. Slots run rootful podman today, and the wrapper's own HONEST BOUNDARY note (hal0-systemctl:52-70) concedes that anyone who can author a container spec can mount host paths. Third-party extensions raise the stakes materially. Correction (2026-10-04): the first draft recommended "rootless under the hal0 user from day one" while the implementation above (D.1, D.6) writes /etc/containers/systemd/hal0-ext@<id>.container through the system-scope SystemCtlSeam — that is the rootful Quadlet path (hal0-systemctl:111 pins QUADLET_DIR="/etc/containers/systemd"; the wrapper has no --user mode), and a User= inside the container changes the process uid, not the Podman boundary. The two options, stated honestly:

  • Explicitly rootful v1 (recommended for first-party extensions). Keep the system-scope seam; compensate with what the manifest can enforce: digest-pinned images, Volume= sources confined to /var/lib/hal0/ext/<id>/, no devices:, NoNewPrivileges=yes, DropCapability=all, ReadOnly=yes where the image allows, and User= as defence in depth. Say in the docs that an extension runs in the same boundary as a slot. Do not enable hal0 app import of third-party manifests under this mode.
  • Real rootless path (prerequisite for third-party import). Quadlets under ~hal0/.config/containers/systemd/ (or /etc/containers/systemd/users/<uid>/), loginctl enable-linger hal0, a user-scope variant of the seam (systemctl --user, a second allow-list), the subuid/subgid ranges the installer already allocates (installer/install.sh:760), GPU access via the render group membership gpu_perms already manages, and a lab check that host-gateway resolves under the rootless network backend (pasta vs. slirp4netns) before relying on AddHost=host.docker.internal:host-gateway.

Recommendation: ship v1 explicitly rootful with the compensating controls and the import verb disabled; make the rootless seam the gate for Decision 4's third-party import rather than claiming rootless while rendering rootful.

Decision 4 — Who may publish? ODS ships 33 library extensions it does not test on every hardware class and lets the dashboard install them at a click. hal0's culture (digest pins, reproducible runner recipes, packaging/runner/*/manifest.toml provenance notes) is much stricter. Recommendation: ship a small first-party extension set (n8n is already half-designed in the UI), require digest-pinned images in the manifest schema, and gate third-party import behind an explicit hal0 app import --i-understand with the same envelope-checksum machinery stacks/portable.py already has.

Decision 5 — Migration of the existing four. OpenWebUI, ComfyUI, Hermes and Hindsight should not all become extensions. ComfyUI is a slot (hal0-slot@img.service) and its start verb is GPU-arbiter-special-cased (services/registry.py:88-96); Hermes has a 15-phase provisioner. Recommendation: convert OpenWebUI only as the proof — it is a plain podman container with a prewired env file, i.e. exactly the extension shape — and leave ServiceDef as a union of builtin and extension-derived rows. If OpenWebUI cannot be expressed in manifest.toml, the schema is wrong.

Risk — the "everything auto-attaches" promise is only as good as the weakest walker. ODS delivers it because four independent consumers each re-walk the directory. hal0 must fund all of them in one go: registry, unit render, port claim, health probe, capability card, mDNS, logs, CLI. Shipping the manifest without the capability card or without hal0 app doctor produces exactly the half-wired feeling the owner wants to avoid.

Risk — id collisions across sources. ODS mirrors a core-service-id deny list in three places (resolve-compose-stack.sh:178-188, routers/extensions.py, helpers.py:CORE_SERVICE_IDS_FALLBACK) and silently drops duplicates (service-registry.sh:146-148). hal0 should make that one function with one test, and reserve the hal0- id prefix outright.

Open question to settle before code: may an extension consume a specific slot ("n8n needs stt"), or only the API? [wiring].slot_env assumes the former, which is what brings ODS's depends_on + services_needed status into play and is the main reason to port calculate_feature_status. If the answer is "API only", the whole capability/eligibility half of the design collapses and the first release gets meaningfully cheaper.

03-installer-reliability.md Installer and reliability

03 — Installation, Upgrade, and Reliability Engineering: ODS vs hal0

Source-level comparative study. All paths are absolute. ODS product runtime is /home/user/ods/ods/; hal0 is /home/user/hal0/.

The owner's premise — "ODS has me beat on installation and reliability" — is half right. ODS beats hal0 decisively on installer structure, install-time observability, and failure forensics. hal0 beats ODS decisively on update integrity, diagnosis modelling, and preflight depth. The gaps are asymmetric and mostly complementary, which is good news: almost everything worth porting is additive.


A. How ODS does it

A.1 The install pipeline is a sourced-phase state machine

/home/user/ods/ods/install-core.sh is a 363-line orchestrator and nothing else. It does five things in order:

  1. Sets INSTALL_PHASE and installs an ERR trap (install-core.sh:23-38) that reports which phase died, where the log is, and what the partial state is: bash export INSTALL_PHASE="init" cleanup_on_error() { local exit_code=$? echo -e "\033[0;31m[ERROR] Installation failed during phase: ${INSTALL_PHASE}\033[0m" echo -e "\033[0;33m Log file: ${LOG_FILE:-/tmp/ods-install.log}\033[0m" ... echo "To retry, run the installer again. It will resume safely." } trap cleanup_on_error ERR
  2. Installs a double-tap SIGINT handler (:45-64) — one Ctrl+C warns, a second within 3s aborts and calls cancel_active_download; SIGTSTP is trapped to nothing so Ctrl+Z can't background a live install.
  3. Sources 15 pure libraries in dependency order (:74-93).
  4. Parses ~40 flags (:220-273): --dry-run, --non-interactive, --summary-json PATH, --no-bootstrap, --offline, --lan, --tier N, …
  5. Sources the 14 phase files, assigning INSTALL_PHASE before each (:338-363). Phase 13 runs under set +e because it is cosmetic and "must never fail the install".

The lib/phase split is a hard architectural rule (docs/INSTALLER-ARCHITECTURE.md): libraries define functions only and are safe to source; phases execute on source. Everything shares one bash namespace. docs/INSTALLER_PHASE_CONTRACTS.md then gives each phase a contract row — Owns / Inputs / Outputs / Idempotency expectation / Common failure modes — plus a "Required Validation By Phase" table and a written Reinstall Contract ("keep user secrets; keep downloaded models when valid; keep runtime data under data/; exit non-zero only when the installed product is not recoverable").

A.2 The standardized module header

Every file under installers/lib/ and installers/phases/ carries the same header. This is the highest-leverage, lowest-cost thing in the whole repo:

#!/bin/bash
### ============================================================================
### ODS Installer — <Module Name>
### ============================================================================
### Part of: installers/lib/   (or installers/phases/)
### Purpose: <one-line description>
### ### Expects: <globals/functions this file reads>
### Provides: <globals/functions this file defines>
### ### Modder notes:
###   <when and why you'd edit this file>
### ============================================================================

The Expects/Provides chain is how a reader traces dataflow through a global namespace without reading every line (docs/INSTALLER-ARCHITECTURE.md, "File Header Convention"). Real examples: installers/lib/detection.sh:1-22, installers/phases/02-detection.sh:1-22, installers/lib/ui.sh:1-24.

The same doc carries a Mod Recipes table ("Add a hardware tier → edit lib/tier-map.sh + lib/detection.sh") and a Generated Config Writers table naming, per config surface, the Linux/macOS/Windows/upgrade writers that must stay in sync — because "only one platform writer was updated" is their most common regression class.

A.3 Bootstrap: get-ods.sh

/home/user/ods/ods/get-ods.sh (507 lines) is the curl | bash entry point:

  • CWD anchoring (:14-21): re-runs from a deleted directory (post-uninstall) otherwise fail with a misleading "check your internet connection".
  • Error classification (:52-65): format_git_clone_error maps git stderr to a specific human cause (getcwd, DNS, permissions).
  • ODS_REF pinning (:90-108, 414-438): a 40-hex SHA gets --depth 1 + git fetch origin <sha> + git checkout --detach; a branch/tag gets --branch. Sparse checkout of only ods/, with a full-clone fallback.
  • Coexistence refusal (:110-215): before cloning it scans sibling directories for an install signature (.env + a compose file naming open-webui, dashboard-api, and llama-server|litellm) and queries Docker for compose projects with the same service tuple, refusing with an explicit ODS_ALLOW_LEGACY_PARALLEL=1 escape hatch.
  • Runtime/dev split (:443-468): rsync excludes tests/, docs/, *.md, .github/ — the installed tree is smaller than the repo tree.
  • docs/INSTALLER_TRUST.md documents the provenance story honestly, including what is not yet true ("ODS does not yet publish a complete signed-release or checksum/SBOM chain"), with a numbered roadmap.
A.4 The pure libraries

Highlights, all under /home/user/ods/ods/installers/lib/:

  • detection.sh (842 lines). ods_in_container() (:150-163) checks /.dockerenv, /run/.containerenv, systemd-detect-virt --container, /proc/1/cgroup regex, and /proc/1/environ; ods_container_label() (:165-185) names it (lxc, docker, …). This feeds show_amd_gpu_device_guidance() (:219-233), which prints LXD-specific remedies (lxc config device add <container> kfd unix-char path=/dev/kfd — LXD syntax; the Proxmox equivalent is a dev0:/dev1: line in /etc/pve/lxc/<CTID>.conf, cf. hal0's docs/getting-started/proxmox.mdx:91-101) when inside a container and modprobe remedies when not. apply_cpu_gpu_fallback() (:235-257) then degrades the whole install to CPU coherently — it rewrites GPU_BACKEND, CAP_*, and clears compose overlays in one place, rather than letting a half-set backend leak downstream. select_cpu_fallback_tier() (:259-274) picks a tier from RAM alone.
  • CPU budget (detection.sh:92-147): get_docker_available_cpus() prefers docker info --format '{{.NCPU}}' over nproc, then calculate_llama_cpu_budget() clamps per-backend limit/reservation to what's actually available. llama-memory-budget.sh does the same for RAM, taking min(host, docker) so Docker Desktop VMs don't get over-committed.
  • progress.sh (25 lines, quoted in full in D.2) — a GUI progress protocol that is a complete no-op unless ODS_INSTALLER_GUI=1.
  • docker-images.sh / compose-images.sh — validate an image tag exists before compose runs (validate_docker_image_or_fallback), and discover the external image set from the resolved compose stack rather than a hardcoded list.
  • readiness-summary.sh (128 lines) — takes name|health_url|container|open_url lines on stdin, probes each, prints Ready now: N/M with per-service state (ready/starting/not detected/needs attention) plus a "Next:" block.
  • model-lifecycle-lock.sh (83 lines) — flock mutual exclusion between the installer and the detached background upgrader, keyed by a cksum of the resolved install dir so an SSH session and a systemd unit with different XDG_RUNTIME_DIR still contend on the same lock.
  • sudo.sh — ods_prepare_sudo() sets ODS_SUDO_AVAILABLE; ods_sudo() then skips root-only extras rather than hanging on a password prompt under --non-interactive.
  • podman-registries.sh (234) repairs Podman short-name search config idempotently (Bash-3.2-safe empty-array guard); packaging.sh (312) is the apt/dnf/yum/pacman/zypper abstraction; lib/rootless-ownership.sh (repo root) provides ods_fix_rootless_ownership() for rootless-Docker bind-mount UID mapping, called from phases/06-directories.sh:1161 and from ods-cli.
A.5 Installer UX: the narrator

installers/lib/ui.sh:97-103 is the entire narrator API:

ai()       { echo -e "  ${GRN}▸${NC} $1" | tee -a "$LOG_FILE"; }
ai_ok()    { echo -e "  ${BGRN}✓${NC} $1" | tee -a "$LOG_FILE"; }
ai_warn()  { echo -e "  ${AMB}⚠${NC} $1" | tee -a "$LOG_FILE"; }
ai_bad()   { echo -e "  ${RED}✗${NC} $1" | tee -a "$LOG_FILE"; }
signal()   { echo -e "  ${GRN}░▒▓█▓▒░${NC} $1" | tee -a "$LOG_FILE"; }

Every one of them tees to $LOG_FILE. That single detail is why ODS can hand a user a log after a failure and why ods-doctor can mine install-report-*.txt later.

show_phase (ui.sh:115-125) prints PHASE n/6 — NAME with a wall-clock timestamp and an EST. TIME: line supplied by the caller; call sites give honest ranges (phases/08-images.sh:21: "~5-10 min + ~30 min ComfyUI build"). spin_task (ui.sh:236-266) drives a braille spinner with an [mm:ss] counter, optionally rendering live progress from a .part file; format_download_progress (ui.sh:187-208) is deliberately pure so it is BATS-testable.

pull_with_progress (ui.sh:305-370) is the retry-protected image pull: timeout 3600 docker pull, up to ODS_DOCKER_PULL_MAX_ATTEMPTS (default 4) attempts with backoff 5/15/30s then doubling (_docker_pull_retry_delay, ui.sh:269-302), a post-pull docker inspect validation step, and a non-retryable classifier that bails immediately on unauthorized|denied|not found|404|no space left on device|cannot connect to the docker daemon. check_service (ui.sh:375-460) is the health-check twin: exponential backoff capped at 8s, distinguishing curl exit 124 (timeout), 7 (refused), 56/52 ("starting up"), while simultaneously inspecting the container — if it is exited|dead|missing it stops retrying instead of burning 30 attempts.

Other surfaces: show_hardware_summary/show_tier_recommendation (ui.sh:462-503); show_install_menu (ui.sh:507-597, Full/Core/Custom, which auto-disables ComfyUI on tier 0/1 with an explanation); show_success_card (:600-632); a .desktop file + GNOME favourites pin (phases/13-summary.sh:229-254); the readiness summary (:365-396); and the --summary-json writer (:435-502), a versioned payload (version, installer_version, tier, runtime.{gpu_backend,backend_service,llm_model,compose_flags,dry_run}, hardware_class, preflight_report) written with mkstemp + fsync + os.replace. docs/SETUP-CARD.md covers a printable 4×6 QR card generator; docs/POST-INSTALL-CHECKLIST.md is a six-step "did it work" ritual.

A.6 Bootstrap mode: tiny model first, hot-swap later

ODS's signature reliability trick; hal0 has no equivalent.

installers/lib/bootstrap-model.sh (57 lines) pins a ~1.22 GiB Qwen3.5-2B (file, URL, SHA256, model id, 64K context floor) and defines bootstrap_needed(): true only when tier > 0, the full GGUF is not already on disk, --no-bootstrap was not passed, and the install is not offline / cloud / external-Lemonade.

phases/11-services.sh:673-690 stashes the full-model config into FULL_* vars and swaps GGUF_FILE/GGUF_URL/LLM_MODEL/MAX_CONTEXT to the bootstrap values, so the entire rest of the phase (download, models.ini, .env patch, compose up) runs unmodified against the tiny model — the user is chatting in ~2 minutes. :1329-1372 then persists retry metadata to data/bootstrap-upgrade.args (mode 600) and launches the upgrader detached, inside a subshell that closes inherited non-stdio FDs first so the parent's flock FDs aren't held for the duration of a multi-GB download:

(
    _phase11_close_inherited_fds_for_daemon
    exec nohup bash "$SCRIPT_DIR/scripts/bootstrap-upgrade.sh" \
        "$INSTALL_DIR" "$FULL_GGUF_FILE" "$FULL_GGUF_URL" \
        "$FULL_GGUF_SHA256" "$FULL_LLM_MODEL" "$FULL_MAX_CONTEXT" \
        "$BOOTSTRAP_GGUF_FILE" \
        > "$INSTALL_DIR/logs/model-upgrade.log" 2>&1
) &

scripts/bootstrap-upgrade.sh (3412 lines) runs set -uo pipefail without -e on purpose ("we handle errors explicitly to avoid killing the background process on transient failures"). Its shape:

  • Status file: write_status() writes data/bootstrap-status.json atomically (status, percent, bytesDownloaded/Total, speedBytesPerSec, eta); a monitor_download loop refreshes it every 2s from the .part size — this is what drives the dashboard progress bar.
  • Download (:2189-2325): script-owned resume, not curl's internal retry (a curl retry after a long reset restarts at byte zero and truncates a good multi-GB .part). Bounded wall-clock budget with rounds; over/undersized .part detection; SHA256 verify; failure preserves the .part for resume.
  • Transaction snapshot (:305-380): snapshot_active_model_config() copies .env, config/llama-server/models.ini, config/litellm/lemonade.yaml into a mktemp -d under data/, recording absence as a .missing sentinel so restore can delete a file that didn't exist. restore_active_model_config() is the exact inverse.
  • Swap: .env promotion, models.ini rewrite, then docker compose … up -d --force-recreate --no-deps llama-server with env -u GGUF_FILE -u LLM_MODEL -u MAX_CONTEXT -u CTX_SIZE so the freshly written .env wins compose interpolation over inherited shell vars.
  • Post-swap assertion (:2712-2745): inspects the recreated container's .Config.Cmd and fails loudly if --model still points at the old GGUF — catching the "compose started rather than recreated" bug that would otherwise surface hours later as a 502.
  • Rollback (restore_docker_llama_server_after_swap_failure, :386-452): restore snapshot → recreate with retry → poll health 60×5s → on AMD also restart LiteLLM and prove a completion through the restored route before declaring rollback successful. Called from four failure sites (:2826, :2845, :2865, :3046), each writing a human-readable write_status "failed" … "<what happened>. <rollback state>. Re-run to retry."
  • Cleanup (:3200-3202): the bootstrap GGUF is deleted only when HOT_SWAP_VERIFIED=true.

Net effect: a failed upgrade leaves the user on a working small model with a resumable partial download and an explanatory status string. That is the single best reliability idea in the repo.

A.7 Reliability tooling
Tool File What it is
Preflight engine scripts/preflight-engine.sh (362) Takes --tier/--ram-gb/--disk-gb/--gpu-backend/--gpu-vram-mb/--platform-id/--compose-overlays, emits blockers/warnings + JSON report (/tmp/ods-preflight-report.json) and an --env mode for shell integration. Pure inputs → pure verdict, so it is fixture-testable.
Preflight fixtures tests/contracts/test-preflight-fixtures.sh (132) Six named scenarios (linux-nvidia-good, windows-mvp-good, macos-mvp-good, disk-blocker, …) asserting .summary.blockers. Runs in make test.
Runtime preflight ods-preflight.sh (332) Post-install service probe with bounded curl (--connect-timeout 3 --max-time 10); --install-env delegates to scripts/linux-install-preflight.sh (456) for a JSON environment report.
Doctor scripts/ods-doctor.sh (1730), docs/ODS-DOCTOR.md --json report with capability_profile, preflight, install_artifacts, diagnoses[] (stable IDs like ODS-COMPOSE-CWD-MISMATCH, ODS-DOCKER-IMAGE-UNRESOLVED, plus a whole ODS-RUNTIME-* inference-contract family) each carrying severity/confidence/evidence/impact/next_steps, runtime, summary, autofix_hints. Exit 0/1.
Support bundle scripts/ods-support-bundle.sh (792), docs/SUPPORT-BUNDLE.md Redacted tarball: doctor output, extension audit, compose resolution + validation, docker info, container log tails, config/env.redacted. Best-effort — a failing probe is recorded in the bundle, not fatal.
Compose failure report installers/lib/compose-failure-report.sh (187) On compose failure writes install-report-<ts>.txt: privacy note, phase, compose command, cached flags, model/runtime env, likely failed image(s) grepped out of the installer log, per-port occupancy with the owning process, docker version/info, redacted compose config tail, compose ps -a, and 160 lines of installer log. The redactor (_ods_report_redact_stream) reads .env and masks both key-name matches and the literal secret values.
Update ods-update.sh check/status/backup/update/rollback/changelog/health. snapshot_pre_update() (:235-320) copies .env*, all docker-compose*.yml, .compose-flags, config/{litellm,n8n,openclaw,searxng}/, .version into data/backups/pre-update-<ts>/ with a snapshot.json integrity marker and MAX_BACKUPS pruning. cmd_update = snapshot → git pull → run migrations/migrate-v*.sh in order → restart → wait_for_healthy (:374-407). Any non-zero step calls _update_rollback() (:409-450), which restores + restarts and, on restore failure, prints literal manual recovery commands.
Migrations migrations/ + scripts/migrate-config.sh Versioned migrate-vX.Y.Z.sh with a .migration-state ledger; check/diff/migrate/backup.
Backup / uninstall / stack test ods-backup.sh, ods-restore.sh, ods-uninstall.sh, test-stack.sh Retention count + free-space precheck; --keep-models/--keep-data/--force; test-stack.sh runs --quick/--stress/--voice with no set -e and exits 0/1/2.
Policy docs docs/KNOWN-GOOD-VERSIONS.md, docs/ADR-IMAGE-TAG-PINNING.md, docs/RELEASE_VALIDATION.md, docs/VALIDATION-MATRIX.md, docs/HIGH_RISK_CHANGE_MAP.md Tested-version baselines per platform; a written, reasoned decision to not pin three :latest tags; the "User Green" gate (Zero-prereq bootstrap / Install / Product / Capability / Model Switchboard / Lifecycle Green); the four-layer validation matrix; and a risk table mapping changed area → required validation.
Gate + tests Makefile, tests/ make gate = lint + test + bats + smoke + simulate; test alone runs ~60 named contract scripts. 33 BATS files, 32 contracts/, 6 smoke/, plus scripts/simulate-installers.sh (292) which runs install-core.sh --dry-run --non-interactive --skip-docker --force --summary-json, the macOS installer in MVP mode, the preflight engine as a Windows stand-in, and doctor — merging all of it into artifacts/installer-sim/SUMMARY.{json,md}.
A.8 Tauri desktop installer

/home/user/ods/installer/ is a Vite + React + Tauri v2 app (7 pages: Welcome → Prerequisites → SystemCheck → GpuDetected → Features → Installing → Complete; src-tauri/src/ is 1246 lines of Rust). installer.rs::run_install (:33-172) is thin by design: ensure_checkout() does git clone --depth 1 --branch, then it maps GUI feature toggles to installer flags and spawns install.sh (or install.ps1) with ODS_INSTALLER_GUI=1, piping stdout through parse_progress_line(). stderr is collected on a separate thread and the last 10 lines are surfaced in the error message. PROGRESS_INTEGRATION.md documents the one-line-per-phase contract. It is a wrapper, not a second installer — the shell path stays the single source of truth.

A.9 Platform breadth

The phase model is replicated, not shared. Windows: installers/windows/install-windows.ps1 dot-sources phases/01-preflight.ps1 … 07-devtools.ps1 plus 15 lib/*.ps1 modules — including a PowerShell tier-map.ps1, readiness-summary.ps1, install-report.ps1, compose-diagnostics.ps1. macOS: installers/macos/install-macos.sh (3327 lines) is one file but speaks the same show_phase 1 6 … "30 seconds" vocabulary (:1132, :1337, :1451, :1567, :1929) with its own lib/{ui,detection,tier-map,env-generator,preflight-fs,bridge-manager}.sh; llama-server runs natively with Metal on the host and Docker services reach it via host.docker.internal through a loopback Colima bridge. WSL: installers/dispatch.sh routes linux|wsl → install-core.sh, with divergences isolated in detection and covered by tests/smoke/wsl-logic.sh. The cost of replication is the "Generated Config Writers" table in docs/INSTALLER-ARCHITECTURE.md — an explicit admission that four writers must be edited together.


B. hal0 today (verified)

Everything below was grepped, not assumed.

Install flow. curl -fsSL https://hal0.dev/install.sh | sudo bash → installer/bootstrap.sh (638) → installer/install.sh (4002). Bootstrap's provenance chain is materially stronger than ODS's: a digest-pinned cosign (per-arch sha256 constants, kept fresh by scripts/update-cosign-pin.sh and a weekly .github/workflows/cosign-pin.yml), keyless-OIDC verification of the channel manifest bytes before any JSON is parsed, then sha256 + Sigstore verification of the release tarball. Channel (stable|preview|nightly) is validated before admission (bootstrap.sh:241-261).

Installer structure. One 4002-line file with 16 ui_step calls (UI_STEP_TOTAL=16, install.sh:246, with tests/install/test_ui_step_total.py asserting the constant matches the call count). No step files, no lib/step separation beyond lib/{ui,distro,preflight,run-as-hal0}.sh, no Expects/Provides headers. The ERR trap (install.sh:248-269) is genuinely good — it names CURRENT_STEP and prints step-specific recovery advice via a case, including the exact ss -ltnp sport = :PORT invocation — but that advice is hardcoded in the trap rather than owned by the step.

UI. installer/lib/ui.sh (326) is a competent narrator: ui_banner (:122), ui_step (:149, ── (n/16) Title ──), ui_spinner_run (:178 — background command + braille spinner + live tail of the command's last line, replaying the last 50 captured lines on failure), ui_box (:270, ANSI-width-aware), and info/warn/err/die (:116-119) with UI_WARN_COUNT/UI_ERR_COUNT so preflight_all can say "passed with N warning(s)". HAL0_PLAIN=1/NO_COLOR/ non-TTY all degrade cleanly with ASCII glyph fallbacks — better than ODS.

Preflight. installer/lib/preflight.sh (2127) is deeper than ODS's engine on substance: hal0_lxc_kind() (:155) classifies none|lxc-privileged|lxc-unprivileged and decides which remedies are even possible from inside; preflight_gpu() (:994) prints the exact Proxmox LXC dev0/gid fix and has a soft/hard mode (HAL0_GPU_GATE=1) with distinct return codes (HAL0_GPU_RC_BROKEN_GID, HAL0_GPU_RC_NO_DEVICE) so the installer can smart-block a broken passthrough rather than "succeed" into CPU-only; preflight_container_runtime() (:682) auto-installs podman under HAL0_CONTAINER_REQUIRED=1; preflight_ports() (:1536) exempts hal0's own units from "port in use". Dual-mode — sourced by install.sh, executed directly by hal0 doctor (:1900-1907). It emits no JSON and has no fixture corpus.

Doctor. Stronger modelling than ODS. src/hal0/diagnostics.py defines frozen dataclasses Evidence(kind, summary, data), NextStep(kind, label, target), Diagnosis(id, severity, confidence, summary, detail, evidence, next_steps, fixable), a frozen DIAGNOSIS_IDS taxonomy (~30 stable IDs, snapshot-tested) and overall_verdict(). src/hal0/health_report.py (298) holds the pure classifiers shared by hal0 doctor verify and GET /api/doctor, with a layering test (tests/diagnostics/test_layering.py) forbidding hal0.api → hal0.cli. cli/doctor_all.py (1285) is one read-only evidence pass; doctor_commands.py (2272) holds perms/models/migrations/profiles/ports with --fix autofixes.

Support bundle. src/hal0/cli/doctor_bundle.py:452 (build_bundle) — hal0 doctor bundle writes manifest.json, commands.tsv, system/, config/ (redacted via the canonical hal0.api._redact.redact_config), diagnostics/ (doctor JSONs + API payloads), logs/ (journalctl), doctor-summary.txt. hal0 already has this.

Updater. src/hal0/updater/updater.py (5006 lines): manifest → 0700 root-only staging → sha256 re-derived from the same open file object that gets extracted → cosign verify-blob (:870) against a GitHub-Actions OIDC identity derived from the release kind → extract to /usr/lib/hal0-<version>/ → config migrations → _atomic_symlink_swap() (:735, os.symlink(tmp) + os.replace) → re-pip into the venv, rolling the symlink back if the re-pip fails → record /var/lib/hal0/hal0.previous. rollback() (:4851) swaps back and warns on schema downgrade. Typed error codes (system.update_cosign_failed, system.update_swap_failed, system.update_rollback_unavailable).

Components converge. src/hal0/components/registry.py — one ComponentDef per updatable component (id, kind, service_id, pinned(), installed(), converge()), with status.py merging catalog × recorded state × live probes. A declarative reconcile model ODS has no analogue for.

Setup. hal0 setup is hidden=True and internal-only (cli/setup_command.py:1-20); install.sh drives it as hal0 setup --auto --no-pull --no-extensions. The Stage-2 handoff prompt was deliberately deleted (install.sh:3985-3997) with a comment forbidding its return. README:42 states the contract: "The installer is the whole setup."

Post-install output. A qrencode QR of the dashboard URL (install.sh:3907-3917, soft-skips when absent), then ui_box "hal0 is ready" with CLI/config/data paths, dashboard/chat URLs, an IPv4+IPv6 reachability list, a repeated Verify FAILED: block for failed post-install smoke probes, and next steps.

Tests. CONTRIBUTING.md:123-215 defines three tiers: α unit (make test, pure pytest, ~3s, 425+ tests), β integration (documented, but the Makefile notes it was retired in v0.2), γ release-gate (make release-test, SSH into an hal0-test LXC, seven-row matrix, tests/release-gate-report.json, statuses pass|fail|skip|deferred). Plus make harness (scripts/harness.sh: installer → cli → runtime → agents → cleanup, merged JSON), make harness-install (scripts/fresh-test-ct.sh, clones a Proxmox CT template for a full install→smoke→uninstall→destroy cycle, appending JSONL), and scripts/release-check.sh (pre-tag ritual). tests/installer/ has 30 files; tests/install/ 18; tests/golden_paths/ 5. tests/release-validation/ adds an agent-driven RC kit — kit.toml, boxes.toml, known-issues.yaml (do-not-re-report), regressions.yaml (every prior finding re-probed each release), lanes/{readonly,stateful,update}/*.md, committed reports/*.md — which is ahead of ODS's VALIDATION-MATRIX.md on carry-forward learning.

Verified absent in hal0: installer summary JSON; any persisted install log file (no tee, no LOG_FILE — output is terminal-only); a failure-report artifact written to disk; per-step time estimates; a GUI/machine-readable progress protocol; image-pull retry/backoff (the only pull is a fire-and-forget (podman pull … || true) & at install.sh:2177); a preflight JSON report or fixture corpus; a background model download with hot-swap and rollback; an installer --dry-run; a documented per-step contract table; a high-risk change map; a known-good-versions baseline doc.


C. Better / worse / equivalent

ODS is better:

  1. Installer decomposition. 14 phase files with written contracts + 19 pure libs vs one 4002-line script. This is the biggest gap and it compounds: it is why ODS can BATS-test tier-map, compose-select, detection, progress, and ui as units, and hal0 cannot.
  2. Install-time forensics. tee "$LOG_FILE" on every narrator call + write_compose_failure_report() + an install-report-*.txt that doctor later mines. hal0 loses the install transcript the moment the terminal scrolls.
  3. Bootstrap-then-upgrade model flow. ~2 min to first token vs a 15–31 GB blocking decision at install time, plus snapshot/rollback around the swap.
  4. Retry discipline on network operations. pull_with_progress (bounded attempts, backoff table, post-pull validation, non-retryable classifier) and check_service (backoff + curl-exit-code classification + container-state short-circuit). hal0 has wait_active — a flat 0.5s poll — and no pull retry.
  5. Machine-readable install output. --summary-json + preflight JSON + doctor JSON, all merged by simulate-installers.sh into one artifact set.
  6. Installer dry-run as a CI lane.
  7. Written per-phase contracts, a reinstall contract, and a high-risk change map.
  8. Time estimates and a progress percentage in the UX.

hal0 is better:

  1. Update integrity. cosign keyless-OIDC + sha256 re-derived from the same fd + atomic symlink swap + venv re-pip rollback, vs git pull origin main. Not close.
  2. Diagnosis modelling. Typed Diagnosis/Evidence/NextStep, a frozen ID taxonomy with a snapshot test, one shared classifier for CLI and API. ODS's doctor has the same idea but implemented as 1730 lines of bash with the shape enforced only by convention.
  3. Preflight substance. LXC privilege classification, GPU device-node + gid-mapping remediation, soft/hard gating with distinct return codes, auto-install of missing runtimes. ODS's preflight-engine.sh is a threshold-checker by comparison — more testable, not more capable.
  4. Degradation quality. HAL0_PLAIN/NO_COLOR/non-TTY with ASCII glyph fallbacks throughout ui.sh; ODS's type_line merely skips the animation.
  5. Declarative component convergence (components/registry.py).
  6. Release-validation carry-forward (regressions.yaml + known-issues.yaml).
  7. Single-entry-point discipline — no first-run wizard, enforced by tests/installer/test_install_single_entry_point.py. ODS still ships a dashboard setup wizard alongside its installer.

Equivalent: support bundle (both redact, both best-effort); QR code; final summary card; distro/package-manager abstraction; interactive-vs-piped detection; idempotent re-run as a stated contract; ADR practice; install-time secret generation.


D. Port candidates, ranked

D.1 — Module header convention (impact: high, effort: trivial, risk: none)

ODS: docs/INSTALLER-ARCHITECTURE.md "File Header Convention". hal0 target: installer/lib/*.sh, and every extracted step file from D.4. Size: ~12 lines per file. Deps: none.

Adopt verbatim, renamed:

#!/usr/bin/env bash
### ============================================================================
### hal0 installer — <Module Name>
### ============================================================================
### Part of: installer/lib/   (or installer/steps/)
### Purpose: <one line>
### ### Expects: <globals/functions read>
### Provides: <globals/functions defined>
### ### Modder notes:
###   <when and why you'd edit this file>
### ============================================================================

Do this first — it is the prerequisite that makes D.4 reviewable.

D.2 — Progress protocol (impact: medium-high, effort: trivial, risk: none)

ODS: installers/lib/progress.sh (25 lines) — copy essentially verbatim. hal0 target: installer/lib/ui.sh (add to the existing lib).

### Emit structured progress events for a GUI/CI consumer.
### Format: HAL0_PROGRESS:<percent>:<phase_id>:<human message>
### Complete no-op unless HAL0_INSTALLER_GUI=1.
hal0_progress() {
  local percent="$1" phase="$2" message="$3"
  if [[ "${HAL0_INSTALLER_GUI:-0}" == "1" ]]; then
    echo "HAL0_PROGRESS:${percent}:${phase}:${message}"
  fi
}

Then one call at the top of each ui_step. Immediately buys: a parseable install transcript for scripts/fresh-test-ct.sh, a progress bar for any future GUI, and a way for the harness to assert which step a failed install reached.

D.3 — Persisted install log + failure report (impact: high, effort: low, risk: low)

ODS: installers/lib/logging.sh (the | tee -a "$LOG_FILE" idiom) and installers/lib/compose-failure-report.sh:91-187. hal0 target: installer/lib/ui.sh (info/warn/err) plus a new installer/lib/failure-report.sh. Size: ~30 + ~150 lines. Deps: a writable log dir. Risk: low — must not break --dev (no root) or non-TTY.

Two parts. First, tee the narrator — every info/warn/err also appends its ANSI-stripped form to $HAL0_INSTALL_LOG. Second, a hal0_write_failure_report invoked from the existing ERR trap, copying ODS's report shape: privacy note → step name → environment (systemctl --failed, podman info, podman images — each under timeout 10, a timed-out probe recorded as evidence rather than hanging the trap on the very subsystem that failed; correction 2026-10-04) → port occupancy with owning process → redacted api.env/hal0.toml → last 160 lines of the install log → journalctl -u hal0-api -n 100. Redact in two passes (correction 2026-10-04: the key-name redactor alone is not enough for a shareable file): hal0.api._redact.redact_config for the structured api.env/hal0.toml section (it masks by key name, _redact.py:52-68, and its text pass knows only a few token shapes), then — as ODS's _ods_report_redact_stream does (compose-failure-report.sh:51-83) — collect the literal values of every sensitive key and replace them across the whole assembled report, because a provider token masked in the config section can still sit verbatim in the install log, a command line, a URL or the journal under an innocuous key. Print the path in the trap so the user has one file to attach. Payoff: hal0 doctor gains an install_artifacts surface exactly as ODS's does, and a HAL0-INSTALL-* diagnosis family can key off it.

D.4 — Decompose install.sh into steps (impact: very high, effort: high, risk: medium)

ODS: install-core.sh + installers/phases/01..13. hal0 target: installer/install.sh as a ~300-line orchestrator + installer/steps/01-preflight.sh … 16-summary.sh. Size: 4002 lines redistributed, ~0 net new logic. Deps: D.1. Risk: medium — one global bash namespace, and any missed set -u interaction bites at install time. Mitigate by moving one step per PR, keeping UI_STEP_TOTAL and its test green throughout, and running make harness-install per move.

Keep ODS's exact discipline: installer/lib/* defines functions only and is safe to source; installer/steps/* executes on source; the orchestrator sets CURRENT_STEP (already exists) before each source. Then write docs/internal/install-step-contracts.md modelled on docs/INSTALLER_PHASE_CONTRACTS.md — Owns / Inputs / Outputs / Idempotency / Failure modes per step — plus hal0's own Reinstall Contract (hal0 claims idempotency in the README; nothing states what that guarantees). This is the change that converts hal0's installer from "a script we're careful with" into "a system with per-unit tests".

D.5 — Retry-protected image pulls (impact: high, effort: low, risk: low)

ODS: installers/lib/ui.sh:269-370. hal0 target: a ui_pull_with_retry alongside ui_spinner_run, plus the Python pulls in src/hal0/components/runner_images_arm.py / cli/runner_image_commands.py. Size: ~70 lines bash, ~40 Python.

Copy three ideas, not the code: (1) a backoff table with env override (5 15 30, then doubling); (2) a post-pull validation step (podman image inspect) — a "successful" pull that produced nothing is a real failure mode; (3) a non-retryable classifier that bails immediately on unauthorized|denied|not found|404|no space left on device|cannot connect. hal0's current (podman pull … || true) & at install.sh:2177 silently accepts every one of those.

D.6 — Preflight JSON report + fixture corpus (impact: high, effort: medium, risk: low)

ODS: scripts/preflight-engine.sh + tests/contracts/test-preflight-fixtures.sh. hal0 target: installer/lib/preflight.sh gains a --report PATH / HAL0_PREFLIGHT_REPORT writer; tests/installer/test_preflight_fixtures.py gains the corpus. Size: ~80 lines in preflight.sh, ~120 test lines. Deps: none.

Do not port ODS's engine — hal0's checks are better. Port the contract: every preflight_* appends {id, severity, summary, detail, next_steps} to an array; preflight_all serialises it, reusing the Diagnosis field names from src/hal0/diagnostics.py so the JSON is already doctor-shaped, with HAL0-PREFLIGHT-* IDs reserved in DIAGNOSIS_IDS. Then add the fixture table — named scenarios with injected inputs asserting blocker counts:

linux-cpu-good           → 0 blockers
lxc-unprivileged-no-gpu  → 0 blockers, ≥1 warning
lxc-broken-gid           → 1 blocker  (HAL0-GPU-BROKEN-GID)
disk-below-floor         → 1 blocker
port-8080-foreign        → 1 blocker
port-8080-own-unit       → 0 blockers  (the exemption path)

hal0's preflight already has the hardest part (the detection); it just has no way to prove the verdicts stay stable across refactors.

D.7 — Bootstrap-then-upgrade model flow (impact: very high, effort: high, risk: medium-high)

ODS: installers/lib/bootstrap-model.sh (57), phases/11-services.sh:673-690 and :1329-1372, scripts/bootstrap-upgrade.sh (3412), installers/lib/model-lifecycle-lock.sh (83), installers/lib/background-tasks.sh (194). hal0 target: src/hal0/install/agent_model.py (which already computes a curated plan and a size string) + a new src/hal0/install/background_pull.py + a hal0-model-pull@.service transient unit. Size: ~400 lines Python if written natively. Deps: the slot-config writer, the registry, systemd-run. Risk: medium-high — this is the one item where copying the implementation would be a mistake.

hal0's shape is already better suited: it has slots, a registry, and a brain model that is the "tiny model first". The port is conceptual:

  1. Keep the unconditional lfm2.5-2.6b brain pull (that is the bootstrap model).
  2. Replace the blocking 15–31 GB agent-model prompt with: ask, then hand the pull to a transient systemd unit (systemd-run --uid=hal0 --gid=hal0 --unit=hal0-model-pull@…; correction 2026-10-04: --unit= only names the unit — without --uid/--gid the pull inherits the installer's root identity and leaves model directories, status files and registry state root-owned for the User=hal0 API, packaging/sudoers/hal0-systemctl:3) rather than nohup — hal0 is a systemd product and gets journald logging and systemctl status for free. Restart is not free (correction 2026-10-04): a transient unit defaults to Restart=no, so a registry or network hiccup leaves it failed; pass --property=Restart=on-failure --property=RestartSec=30s (and a StartLimitBurst the pull can survive), or use a static hal0-model-pull@.service template that declares them.
  3. Copy ODS's status-file contract (data/bootstrap-status.json: status/percent/bytesDownloaded/bytesTotal/speedBytesPerSec/eta, written atomically) so the dashboard can render progress.
  4. Copy the transaction snapshot shape verbatim (bootstrap-upgrade.sh:305-380) — snapshot the slot TOML + registry entry before binding, restore on failure, and use a .missing sentinel so restore can delete a file that did not previously exist: bash snapshot_file_state() { # writes <snap> or <snap>.missing if [[ -f "$src" ]]; then cp -p "$src" "$snap"; else : > "$snap.missing"; fi } restore_file_state() { if [[ -f "$snap" ]]; then cp -p "$snap" "$dst" elif [[ -f "$snap.missing" ]]; then rm -f "$dst"; fi }
  5. Copy the verified-before-cleanup rule — only delete after the new slot has answered a real completion (HOT_SWAP_VERIFIED=true, bootstrap-upgrade.sh:3200-3202) — but make the what reference-aware (correction 2026-10-04). ODS replaces one bootstrap model in one server; here lfm2.5-2.6b stays bound to the brain slot while the agent model lands in a separate slot, so there is no "old model" to remove, and treating the brain model as one would strip a live slot of its weights. Cleanup must consult registry.discover.referenced_model_ids() (discover.py:551) and go through the refcount GC in registry/gc.py, reaping only models no slot or stack references — never the brain model.
  6. Copy the lock: model-lifecycle-lock.sh is 83 lines and directly reusable — the installer and a background pull must not both rewrite a slot config. hal0 already has hal0_config_file_lock in config/loader.py; extend that rather than adding a second mechanism.
D.8 — --summary-json (impact: medium, effort: low, risk: none)

ODS: installers/phases/13-summary.sh:435-502, including the mkstemp+fsync+os.replace atomic write. hal0 target: the install.sh summary block, writing ${VAR_DIR}/install-summary.json by default. Size: ~40 lines. Payload: version, channel, install_root, models_dir, slots[], services[] with start verdicts, smoke_failed[], preflight_report path. scripts/fresh-test-ct.sh currently parses terminal output; this replaces that with a contract.

D.9 — Time estimates on steps (impact: medium, effort: trivial, risk: none)

ODS: show_phase 4 6 "Downloading Modules" "~5-10 min + ~30 min ComfyUI build". hal0 target: ui_step "Python environment" "~2 min" — one optional second arg, rendered dim at the right of the rule. Honest ranges matter more than precision; ODS's ComfyUI line is the model.

D.10 — Installer --dry-run + simulate harness (impact: medium, effort: medium, risk: low)

ODS: every phase guards mutations with if $DRY_RUN; scripts/simulate-installers.sh runs it in CI and merges artifacts. Feasible for hal0 only after D.4 — a 4002-line script cannot be retrofitted with dry-run guards safely. Add HAL0_DRY_RUN=1 per step, then a scripts/simulate-install.sh emitting artifacts/install-sim/SUMMARY.json from D.8 + D.6.

D.11 — Governance docs (impact: medium, effort: low, risk: none)

Port three, adapted: docs/HIGH_RISK_CHANGE_MAP.md → a CONTRIBUTING.md section mapping area → risk level → required validation (hal0 has the tiers α/γ/harness/ rc-validate but no table saying which to run for a given diff — cheapest reliability win available); docs/KNOWN-GOOD-VERSIONS.md → docs/reference/known-good-versions.mdx with tested kernel/podman/ROCm/driver baselines per hardware class, as the support-triage companion to the existing hardware-matrix.mdx; and INSTALLER_PHASE_CONTRACTS.md's Reinstall Contract → installer/README.md, because hal0 asserts idempotency without defining it.


E. Do NOT copy

  1. ods-update.sh's update mechanism. git fetch && git pull origin main with no signature verification, into the live install directory. hal0's cosign-verified, staged, atomic-symlink updater is a different class of thing; copying here would be a downgrade. (The pre-update snapshot discipline at :235-320 is worth reading, but hal0.previous + config migrations already cover that ground.)
  2. The CRT/"lore" theme — LORE_MESSAGES[], show_stranger_boot(), type_line_dramatic(), "THE GATEWAY IS OPEN", the terminal bell. The mechanism underneath (print something every N seconds so a long pull doesn't look hung) is worth having, and ui_spinner_run already does it better by tailing the command's real output. The ideology is not hal0's voice and adds 100+ lines of untestable string data.
  3. type_line() character-by-character animation (ui.sh:28-49) — a 0.035s sleep per character is dead wall-clock on every install and mangles log capture.
  4. scripts/bootstrap-upgrade.sh as a file — 3412 lines with Windows Lemonade, macOS native-PID, and Linux Docker paths interleaved under set -uo pipefail without -e. Port the five ideas in D.7; not the artifact.
  5. The Tauri desktop installer. hal0 installs curl | sudo bash onto a headless box; a GUI wrapper solves a problem hal0 does not have. The one transferable piece is the 25-line progress protocol (D.2), useful without a GUI.
  6. Duplicating the phase model per platform (installers/windows/phases/*.ps1, installers/macos/lib/tier-map.sh). ODS pays for this with the "Generated Config Writers" sync table. hal0 is Linux+systemd only — do not acquire that cost speculatively.
  7. ADR-IMAGE-TAG-PINNING.md's conclusion (retain :latest for three third-party images). hal0 pins by digest with lockstep tests (tests/installer/test_owui_pin_lockstep.py, test_hindsight_pin_lockstep.py) and a weekly cosign-pin workflow. The ADR's format is worth emulating; its decision is not.
  8. ods-preflight.sh (the runtime one) — hal0 doctor verify + health_report.py already cover that surface with typed results.

F. Decisions for the owner

  1. Decompose install.sh — yes or no? The load-bearing decision. 4002 lines in one file is why hal0 has no per-step tests, no dry-run, no per-step contracts, and why the ERR trap hardcodes recovery advice for four named steps. Cost: several weeks of one-step-at-a-time PRs with make harness-install between each. Recommendation: yes, sequenced D.1 → D.3 → D.4, so the headers and the install log land first and make the decomposition reviewable.

  2. Background model pull: systemd unit or nohup? ODS uses nohup + a PID registry because it has no service-manager guarantee; hal0 does. Recommend systemd-run --unit=hal0-model-pull@<slot> — journald logging, systemctl status, and restart policy for free, with the status JSON reduced to a dashboard-progress concern rather than the only observability.

  3. Should preflight emit Diagnosis rows? Reusing diagnostics.py's field names from bash means preflight output drops straight into hal0 doctor --json and the support bundle with no translation layer, and preflight IDs join the frozen taxonomy. The cost is a contract installer/lib/preflight.sh must not drift from. Recommend yes, reserving HAL0-PREFLIGHT-* IDs up front (ODS reserves IDs before the emitter lands; same pattern).

  4. Where does the install log live? ODS uses /tmp/ods-install.log, lost on reboot. Options: ${VAR_DIR}/logs/install-<ts>.log (survives; needs the dir before the filesystem step) or journald via systemd-cat (survives, no ordering problem, harder to attach to an issue). Recommend the file, created in step 1, path echoed by the ERR trap — and have hal0 doctor bundle glob it and any failure report into diagnostics/, so a support case gets evidence the user never thought to send.

  5. Is "the installer is the whole setup" negotiable? Currently enforced by a test and a do-not-reintroduce comment. Every port above respects it — D.7 strengthens it by removing the one install-time question big enough to make people defer. Worth re-affirming before D.4 starts, since decomposition is exactly when a "quick post-install wizard step" gets proposed.

  6. What hal0 does not need from ODS: multi-platform installers, compose-stack resolution, tier→model mapping (slots + the curated ladder already generalise it), and the desktop GUI. Do not let the breadth of ODS's installer tree imply hal0 is missing surface area — hal0 is missing structure and instrumentation on the surface it already has.

04-models.md Models

Model handling: ODS vs. hal0

Scope: selection, storage, download, switching, rollback, and discovery. ODS paths are relative to /home/user/ods/ods/ unless given in full; hal0 paths are absolute under /home/user/hal0/.

A. ODS mechanism

A.1 Catalog + selection

config/model-library.json (schema "version": 2, 56 entries) is the single versioned catalog. Each entry carries id, family, gguf_file/gguf_url/gguf_sha256, size_bytes/size_mb, vram_required_gb, context_length/max_context_length, quantization, specialty, tokens_per_sec_estimate, llama_server_image, and — the distinctive part — an app_compatibility block per consumer app (openai_chat, hermes_talk, perplexica, agent_viability, …) with status, label, reason, evidence (a path into a fleet-test run directory), recordedAt, productSha/harnessSha, sometimes hostScope. This is real fleet-test provenance, not hand-written metadata: qwen3.5-2b-q4's hermes_talk.status is verified across six named hosts while openai_chat is unsupported_until_revalidated because two consumers truncated output on one host (config/model-library.json:9-63). Some entries also carry publisher_evidence, quality_evidence, source_evidence, rejected_artifacts, and runtime_profiles (VRAM/arch-gated alternate launch recipes).

scripts/select-model.py (562 lines) is a deterministic, offline selector invoked by the installer and by ods model swap. It loads only source ∈ {"", "curated"} records (load_catalog, :121-131) — HF imports in data/model-imports.json are excluded from auto-selection. usable_memory_gb() (:134-144) computes a memory envelope: unified-memory hosts get max(ram_gb*0.55, 2.0) GB, CPU/no-GPU hosts get min(max(ram_gb*0.35,3.0),8.0) GB, else raw VRAM/1024. selector_required_memory_gb() (:177-182) adds an estimated KV-cache footprint on top of on-disk size: estimated_param_billions() (:151-167) infers B-params from declared fields or a regex on id/name/gguf filename, then estimated_context_kv_gb() (:170-174) scales params_b*0.12 (clamped 0.35–3.5 GB) by context/32768. score_model() (:237-259) ranks by specialty weight (Code=4.4 > Quality=4.1 > General=3.8 > … > Bootstrap=1.0), a family-affinity bonus, a context bonus, a size term, and a headroom penalty above 92%/98% memory utilization. MODEL_PROFILE is qwen | gemma4 | auto (:38-44), auto resolving to qwen on CLOUD/0 tier and gemma4 elsewhere (:68-73); family_allowed() (:230-234) filters the catalog to one family.

Unified-memory routing (arch_policy_model, :326-364): on the qwen profile, when the tier is NV_ULTRA on arm64 (Spark/GB10) or memory_type == "unified" and the otherwise-selected model is qwen3-coder-next (is_spark_aarch64_excluded_model, :367-372), the selector force-substitutes qwen3.6-35b-a3b-ud-q4 under policy tag unified-memory-coder-next-a3b-v1, because "qwen3-coder-next produces all-? tokens on every chat completion" on those backends (:24-31,100-108). Output is JSON or --env shell assignments (LLM_MODEL, GGUF_FILE, GGUF_URL, GGUF_SHA256, MAX_CONTEXT, LLM_MODEL_SIZE_MB, MODEL_RECOMMENDATION_*, and, when a runtime_profile matched, MODEL_RUNTIME_PROFILE*/LLAMA_SERVER_IMAGE).

installers/lib/tier-map.sh (393 lines) is the static, non-catalog tier table actually used for install-time defaults and ods model swap: set_qwen_tier_config()/set_gemma4_tier_config() hard-code LLM_MODEL/GGUF_FILE/GGUF_URL/GGUF_SHA256/MAX_CONTEXT/LLM_MODEL_SIZE_MB per tier (CLOUD, ARC, ARC_LITE, NV_ULTRA, SH_LARGE, SH_COMPACT, 0-4). Two in-source incident notes matter: the NV_ULTRA block (:92-118) documents the same coder-next bug on Blackwell-aarch64, and the SH_LARGE block (:119-138) documents a live 2026-05-19 Strix Halo repro ("bootstrap-upgrade downloaded the 48.5GB coder-next-Q4_K_M.gguf, stalled at 91%, and the model would not have worked on this hardware even if the download completed") and the fix (substitute Qwen3.6-35B-A3B-UD-Q4_K_M, ~21.1 GB). tier_to_model() (:333-393) is what ods model swap <tier> uses to resolve a tier without re-running detection.

config/backends/{amd,nvidia,apple,cpu}.json are small per-backend service contracts (engine, API port/path, provider URL, runtime image) — amd.json:1-21 pins llm_engine: lemonade, lemonade-server:v10.2.0, linux_backend: rocm / windows_backend: vulkan. config/hardware-classes.json maps (platform_id, gpu_vendor, memory_type, vram_mb range) → {backend, tier, compose_overlays}; strix_unified_large matches AMD unified memory ≥ 92160 MB → SH_LARGE+docker-compose.amd.yml (:3-18). docs/CAPABILITY-PROFILE.md documents the derived CAP_* env contract install-core consumes for tier/backend/overlay normalization.

installers/lib/llama-memory-budget.sh (56 lines) is narrower than its name: it reads Docker-visible memory, takes min(host, Docker) RAM, and derives the NVIDIA llama-server container memory limit — reserve 3 GiB under 16 GiB RAM else 4 GiB, clamp usable to [1,64] GiB (:33-56). Context/batch sizing itself lives in tier-map.sh's static MAX_CONTEXT, not a separate budgeting module.

docs/MODEL-SWITCHBOARD.md (938 lines) is a PR plan, not all-implemented status: its header says the state store/data-plane/reconciler/consumer-migration work "are not implemented" as of its audit date (:5-7), but extensions/services/model-router/app/main.py (PR 3's target) exists and is wired into every non-cloud compose file (A.4) — the doc is architecture rationale/roadmap; cite code for current truth.

docs/ENGINE-PROVIDER-MODES.md defines four provider modes — local, cloud (LiteLLM → remote APIs), hybrid (local-first + cloud fallback), lemonade (ODS-managed or external) — with a shared env contract (ODS_MODE, LLM_BACKEND, LEMONADE_EXTERNAL, LEMONADE_BASE_URL/LEMONADE_CONTAINER_BASE_URL, LEMONADE_API_BASE_PATH=/api/v1) and a capability-proof contract (liveness/chat/embeddings/STT/TTS/rerank/stats must each pass through the app-visible route or the mode fails readiness) (:14-60). LiteLLM (config/litellm/{local,lemonade,hybrid,cloud,switchboard}.yaml) is the OpenAI-compatible front door in every mode.

A.2 Storage + download

Models live at data/models/*.gguf (one file per model, flat). scripts/pre-download.sh (425 lines) is legacy/independent: its own nano/edge/pro/cluster tiers map to non-GGUF HF repo ids (e.g. Qwen/Qwen2.5-32B-Instruct-AWQ) via huggingface_hub.snapshot_download (:162-176) — it shares no tiers/ids/filenames with config/model-library.json and has no SHA256 check. Not the production path.

The real download path is installers/phases/11-services.sh: checks an existing file's SHA256 against GGUF_SHA256 and deletes-and-redownloads on mismatch (:697-722); downloads with curl -fSL -C - --connect-timeout 30 --max-time 3600 --retry 3 --retry-delay 5 --retry-all-errors into a .part file so an interrupted install resumes (:751-762); on curl failure falls back to scripts/download-hf-artifact.py (89 lines), which parses a /resolve/<rev>/<file> URL and calls huggingface_hub.hf_hub_download — the documented hf_xet fallback for Xet-backed HF files plain curl can fail against (:1-7,36-56); re-verifies SHA256 after either path and hard-fails the install if it never passes (:791-829,894-897). scripts/download-hf-snapshot.py (74 lines) is the same helper for whole-repo snapshots.

installers/lib/bootstrap-model.sh pins the fast-start model as constants (BOOTSTRAP_GGUF_FILE=Qwen3.5-2B-Q4_K_M.gguf, sha, LLM_MODEL=qwen3.5-2b, MAX_CONTEXT=65536, :16-24); bootstrap_needed() (:34-52) gates on tier>0, full model absent, no --no-bootstrap, not offline/cloud/external-Lemonade. .env gets GGUF_FILE/LLM_MODEL/MAX_CONTEXT/MODEL_RECOMMENDATION_* from select-model.py --env. config/llama-server/models.ini is a plain per-model INI stanza (installers/phases/11-services.sh:929-939):

[${LLM_MODEL}]
filename = ${GGUF_FILE}
load-on-startup = true
n-ctx = ${MAX_CONTEXT}

Background hot-swap: phase 11 backgrounds scripts/bootstrap-upgrade.sh (3412 lines) with the full model's file/url/sha/name/context (installers/phases/11-services.sh:1328-1362); it downloads, re-verifies sha (:242-275), and hot-swaps llama-server with minimal downtime, safely re-triggerable by ods start/ods restart/re-install on failure.

No mmproj/vision-projector handling exists anywhere in ODS's model catalog, download scripts, or models.ini generation — ODS's local models are text-only.

A.3 Switching + rollback

ods model {current,list,swap} (ods-cli:3584-3710): current reads .env's LLM_MODEL/TIER; list prints the static tier table; swap <tier> resolves the tier via tier-map.sh, looks up the catalog id for that tier's GGUF_FILE via jq, requires the file already downloaded (:3623-3625), then POSTs {model_id, tier, context_length} to the host agent's POST /v1/model/activate (bearer-authed, key fed over stdin so it never appears in argv), treating anything but HTTP 200 + status:"activated" as failure (:3635-3697).

The activation transaction, _do_model_activate() in bin/ods-host-agent.py (12,834 lines total; handler at :7635), snapshots every file it's about to touch before mutating — .env, models.ini, config/litellm/{lemonade,local,switchboard}.yaml, config/model-router/endpoints.json, the activation receipt, Hermes/OpenCode/Perplexica config, plus container state (:7836-7876). restore_backups()/restore_previous_runtime()/rollback_and_prove() (:7878-8020) implement rollback: on any failure past the mutation point, every snapshot is restored, the previous runtime relaunched, and — critically — the previous model must itself re-pass identity+completion proof before rollback is reported successful (:7967-8020); a rollback that can't re-prove the old model is reported as a failed restoration, not a silent success. installers/lib/model-lifecycle-lock.sh is a flock-based, per-install-dir advisory lock so a background bootstrap-upgrade and a foreground ods model swap/install can never race on the same files (:15-83).

docs/MODEL-MANAGEMENT.md (466 lines) documents the six-verb dashboard flow (discover → download → load → use-through-every-app → restore → delete): multi-GPU reassignment re-runs the topology planner when a target model's VRAM envelope exceeds the current GPU subset (:137-160); context above a model's declared max is allowed with a warning but must pass runtime identity+context proof or roll back (:101-131); HF imports pin an immutable revision and reject projectors/adapters/split files (:38-70); the .env/models.ini fields ODS's activation transaction owns are read-only in the Dashboard settings editor specifically so a manual edit can't bypass verification/rollback (:194-199).

A.4 The model-router service

extensions/services/model-router/app/main.py (1126 lines) is fully implemented, not just planned. It is the Switchboard's "PR 3" data plane: a public stable alias (ods/current, default) every ODS app can hardcode, decoupling app config from the concrete model. Key mechanics: reads data/model-state.json with strict shape validation and only trusts an active route with completed identity+completion proof and a monotonic seq/routeSeq (_has_verified_active_route, :493-538) — regressed/mutated-at-same-sequence state is rejected; endpointId resolves only through a read-only, install-generated allowlist so state can never name an arbitrary upstream (:317-347,541-562); forwards only /v1/chat/completions|completions|responses (:84-88), stripping client-auth headers and injecting its own upstream credential; rewrites model back to the requested alias on every response and every SSE chunk (_SSERewriter, :680-786) while emitting X-ODS-{Request-Id,Requested-Model,Routed-Model,Backend,Route-Seq} headers; bounded queue-during-reload returns HTTP 503 model_swap_in_progress with Retry-After on timeout (:894-935); signed fleet-probe route evidence (HMAC-SHA256 over a UUID, :605-620) is recorded in a bounded TTL'd map behind an internal bearer-authed endpoint, never prompts/generations/credentials.

It exists alongside LiteLLM, not instead of it: at design time LiteLLM's alias-to-model mapping was a static YAML mount with no DB-backed management plane, so changing ods/current would otherwise mean rewriting LiteLLM's YAML and restarting it on every swap (docs/MODEL-SWITCHBOARD.md:94-98). model-router is the small, ODS-owned, hot-reloadable indirection layer; LiteLLM keeps owning auth/OpenAI-compat/multi-provider policy and always points ods/current/default/"*" at http://model-router:9099/v1 (confirmed live in config/litellm/switchboard.yaml, docker-compose.base.yml:225,299-303,328).

extensions/services/llama-server/ ships only a manifest (no compose.yaml — its compose lives in docker-compose.base.yml as a core service); the manifest declares aliases: [llm] (so ods restart llm works), port: 8080, health: /health, gpu_backends: [amd, nvidia].

A.5 AMD Strix Halo specifics

config/backends/amd.json pins the Lemonade runtime: ghcr.io/lemonade-sdk/lemonade-server:v10.2.0, linux_backend: rocm, windows_backend: vulkan, health at /api/v1/health. config/gpu-database.json has exact device matches: strix_halo_395 (device_ids: ["0x1586"], "Radeon 8060S"/"RYZEN AI MAX+ 395"/"Strix Halo", memory_mb: 98304, rdna-3.5, bandwidth_gbps: 256, compute_units: 40) → tier SH_LARGE; strix_halo_390 (RYZEN AI MAX 390, memory_mb: 65536) → SH_COMPACT (:29-68). config/hardware-classes.json gates SH_LARGE at min_vram_mb: 92160, SH_COMPACT at 65536 (both AMD, unified, linux/wsl).

SH_COMPACT ships qwen3-30b-a3b (18.6 GB, 128K ctx); SH_LARGE ships qwen3.6-35b-a3b (21.1 GB UD-Q4_K_M, 128K ctx) — not qwen3-coder-next, per the routing note in A.1/A.3 (installers/lib/tier-map.sh:119-146). docs/LEMONADE-SDK-COMPAT.md documents wrapping an already-running Lemonade SDK install (--use-existing-lemonade) without ODS managing that process or downloading its own GGUF, auto-detecting the first non-image model from Lemonade's /api/v1/models unless LEMONADE_MODEL is set, with Phase 12 proving the model via a real chat completion (:90-118). docs/AMD-ODS2-GAMMA-BRIEF.md is a go-to-market/appliance-onboarding deck (QR setup, magic-link auth, mDNS) with Strix Halo as flagship demo hardware and no model-selection content of its own; it self-flags "code present, packaged appliance images need end-to-end validation."

The one fact hal0 most needs from this domain: ODS's own fleet testing found qwen3-coder-next (an 80B-A3B MoE, ~48.5 GB Q4_K_M) produces garbage output ("all-? tokens on every chat completion") specifically on unified-memory backends — verified independently on Strix Halo (AMD, gfx1151) and NVIDIA Grace-Blackwell aarch64 (Spark/GB10) — and now hard-routes both away from it to Qwen3.6-35B-A3B-UD-Q4_K_M (installers/lib/tier-map.sh:100-138; scripts/select-model.py:24-31,326-372). The model stays in the catalog (install_recommendation: true) for manual/discrete-GPU use; it's excluded only on unified memory.

B. hal0 today

Registry (src/hal0/registry/): store.py (624 lines) is mid-cutover — TomlModelRegistry is the original atomic-tmpfile+fsync+os.replace implementation (_atomic_write, :305-341) plus a cross-process flock on a stable-inode sidecar (registry_write_lock, :101-134) and an mtime-cache, but the public name ModelRegistry is rebound at the bottom of the file to SqliteModelRegistry (:594-612) — SQLite is the actual runtime source of truth since the "ML-1" migration, TOML kept alive only for export/import-sqlite. gguf_header.py (379 lines) mmaps up to 8 MiB and walks the real GGUF v1-v3 KV block rather than trusting the filename, extracting general.architecture, general.embedding_length, general.tags, <arch>.context_length, <arch>.pooling_type, <arch>.attention.causal (:1-36,85-90). detect.py (579 lines) layers capability inference cheapest-first: header pooling_type (high confidence) → tag heuristics → filename tokens (medium/low), extracts quantization from general.file_type or a filename regex, and _hf_repo_name_from_path() (:214-229) walks up a path looking for a models--ORG--REPO HF-cache dir to recover org/repo when scanning blob-hash-named cache files directly. discover.py (717 lines) is the filesystem scanner (POST /api/models/scan + startup auto-scan), skipping HF-cache internals and non-standalone accessory dirs (mmproj, vae, clip, controlnet, …).

model_fit / capacity: model_fit.py (118 lines) — evaluate_model_fit() is a pure, reason-bearing verdict (allowed|blocked|degraded) combining model classification, slot-type match, registry resolvability, and profile/device-class compatibility (NPU/image mismatches hard-block, GPU/CPU degrade). slots/capacity.py's gtt_feasibility_verdict()/estimate_file_size_kv_mb() (:177-282) is a live memory-fit check: it reads AMD amdgpu sysfs GTT counters at request time, explicitly because on a UMA/LXC host /proc/meminfo can lie ("the actual weight allocation fails with cudaMalloc failed: out of memory", :222-228), returning fits|tight|exceeds|exceeds_total|unknown — advisory only, never blocking. POST /api/models/feasibility (src/hal0/api/routes/models.py:604-692) batches this per (model, ctx) pick for the picker UI, degrading any bad row to "unknown" rather than 500ing the batch.

profiles/: profiles are device-agnostic "logical tune" templates (chat template, sampler, KV type, batch sizes) — explicitly not hardware flags (NGL/threads/tensor-split live on the slot). 10 shipped seeds cover one profile per runtime family plus generic chat/cpu-chat/embedding/reranking; 8 workload variants (long-context, dense, MoE, named family recipes like chadrock-dense with --no-mmproj/mtp=true toggles tuned to a real model card) are demoted to legacy_seed_profiles.toml, auto-migrated once into an existing install as editable custom entries, never re-injected fresh. -fa auto (never -fa on) is a deliberate policy so a generic profile never forces an unschedulable fused-attention kernel.

capabilities/: a UX overlay grouping flat slots into "capability cards" (embed=embed+rerank, voice=stt+tts, img) via a _CHILD_TO_SLOT bridge; selections persist in capabilities.toml, and CapabilityOrchestrator.apply() reconciles against slots/*.toml through slot_config.SlotConfigStore (compute-only apply() → ChangeSet, atomic commit()/revert() across both files). NPU picks (device=npu) don't spawn a standalone slot — they drive one shared flm serve anchor serving chat+embed+ASR coresident ("NPU trio"), a multiplexing model ODS's single-llama-server design has no equivalent of. slot_config/ is the single reconciled-write owner for both files.

API routes: models.py exposes GET /api/models, GET /api/models/catalogue, POST /api/models/scan{,/preview}, POST /api/models/add-from-path, POST /api/models (register), POST /api/models/validate, GET /api/models/pulls, GET /api/models/updates/check + POST /api/models/{id}/update (HF LFS-sha diffing against the stored sha, src/hal0/registry/update_check.py:1-17 — "is a newer upstream build available," which ODS's pinned-revision catalog doesn't offer), POST /api/models/feasibility, GET/PUT/DELETE /api/models/{id}, POST .../{seed-profile,default}, POST .../pull + GET .../pull/{status,stream} (SSE) + POST .../pull/cancel, POST /api/models/inspect. hf.py adds GET /api/models/huggingface/search (thinner than ODS's — see D).

Download (src/hal0/registry/pull.py, 2169 lines): per-file streaming SHA256 (computed while downloading, :739), true HTTP Range resume via a .part+.part.json sidecar keyed by model id and filename (:546-610), and — where curl-based ODS only verifies when a hash is pre-pinned in its own catalog — hal0 extracts HF's own advertised LFS sha256 off X-Linked-ETag (:659-684) so an unpinned community pull still gets verified against what HF claims to serve. PullFile.kind: "model"|"mmproj" and a documented two-file pull shape (:128-152,973-1111) give hal0 first-class mmproj/vision support ODS has none of; CuratedModel.mmproj_file (src/hal0/registry/curated.py:107-116) lets an entry declare a companion projector downloaded alongside the main file and stamped onto Model.mmproj directly.

Curated catalog (curated.py, 1299 lines, 41 entries): code-shipped, not remote-fetched, "so users can't end up on an outdated pick list," sized for "Strix Halo's 100 GB unified pool." Its own header is candid about gaps: STT/TTS curated picks are blocked because Moonshine/Kokoro/VibeVoice weights don't fit the single-file pull shape yet (:17-58) — the wizard's STT/TTS dropdowns are empty on a fresh install. Entries carry tool_calling: bool|None ("verified, not aspirational") plus free-text notes, but no cross-app, cross-host pass/fail ledger like ODS's app_compatibility.

UI (ui/src/dash/): tests for model-filters, model-usage, model-drawer — a drawer/picker pattern rather than ODS's dedicated Models.jsx page, consistent with the flatter "everything is a slot" surface.

Brain steward install-time download (src/hal0/install/brain_model.py, 573 lines): provisions the dashboard sidebar-chat "steward" model (hal0/brain) at install time, defaulting to a hardware-blind LFM2.5-2.6B Q8_0 GGUF loadable by both the FPX runner and stock llama.cpp. Never fatal (a failed pull leaves the slot model-less, not the install broken), reuses the same run_pull_and_activate gate as every model activation, idempotent via already_pulled(). A long in-source postmortem (:53-70) documents a real 0.9.8→1.0.0 upgrade regression where scaffold/seed-loop ordering silently reverted a successful bind — the same "write the incident, then the invariant" style as ODS's tier-map.sh notes.

C. Better / worse / equivalent

ODS ahead: 1. Fleet-verified catalog metadata — app_compatibility (real run evidence, per-consumer pass/fail, host scope, product/harness SHAs) has no hal0 analogue; hal0's tool_calling flag + free-text notes cover a fraction of the same signal. 2. HF discovery + verified import UX end to end — routers/models.py's pipeline (_hf_gguf_artifacts, _hf_llm_runtime_compatibility, Hub-parsed context length, revision pinning, projector/adapter/split-file rejection, gated-repo HF_TOKEN handling) is more complete than hal0's hf.py search endpoint plus the general pull path. 3. Automatic rollback with re-proof of the old model — rollback_and_prove requires the restored previous model to itself pass identity+completion proof before rollback counts as successful. hal0 has no equivalent transactional multi-file-snapshot + app-config-fanout + verified-rollback path; a failed slot load is a state-machine transition, not a snapshot/restore across config plus every downstream consumer. 4. Stable-alias routing layer (model-router) decoupling every app from the concrete model — hal0 has no equivalent; each slot binds to a concrete model id, and a swap is a reconfigure+restart, not an atomic alias flip behind a shared gateway. 5. Static, pre-flight memory-envelope selection with documented hardware exclusions (the unified-memory coder-next routing) — hard-won hardware knowledge hal0's selection logic doesn't encode anywhere (see F). 6. Bootstrap-then-hot-swap fast start — tiny model first, full model swapped in transparently in the background, resumable independent of the foreground install; hal0's brain-model install is "never fatal" but a single pull, not a fast-start/upgrade pair.

hal0 ahead: 1. Multi-slot architecture — ODS runs one llama-server process; hal0 runs N independent hal0-slot@<name> containers (chat/embed/rerank/stt/tts/img, plus an NPU trio) with per-slot lifecycle. 2. Registry as real infrastructure, not a catalog — atomic writes, cross-process flock, mtime-cache, one-level-deep merges with explicit null-clears, and (ML-1) a SQLite-backed store behind the same interface. ODS has nothing playing this role for locally-discovered/imported models beyond flat data/model-imports.json. 3. GGUF magic-byte + KV-header parsing for capability/context/quant, vs. ODS's filename/catalog-declared metadata only. 4. Live, hardware-truth memory-fit check (gtt_feasibility_verdict against real amdgpu sysfs counters) per-request, vs. ODS's static, install-time-only envelope math. 5. mmproj/vision as a first-class, documented two-file pull shape — ODS has none. 6. HF update-checking for already-pulled models (LFS sha diff) — ODS's pinned-revision catalog doesn't offer this (and arguably shouldn't — see E). 7. Capability cards + NPU multiplexing — a richer UX/runtime model for grouping embed/voice/image slots and sharing one NPU process across chat+embed+ASR.

Roughly equivalent: both do atomic-write+fsync+rename config persistence; both have real streaming-SHA256 verification with resume; both classify GGUF-vs-non-GGUF; both fall back to a sane on-disk model when the configured one is missing (ODS's rank_models pool vs. hal0's fallback_local_model heuristic) — hal0's is arguably more careful (explicit looks_diffusion_or_nontext guard after a real incident where a 25 GB video model was picked for a chat slot).

D. Port candidates

Ordered by value/effort. "Risk" reflects integration risk in hal0, not code quality.

  1. Unified-memory model exclusion/substitution policy — port the knowledge, not the code. Source: installers/lib/tier-map.sh:100-138, scripts/select-model.py:24-31,326-372. hal0 target: a check in src/hal0/registry/curated.py (flag qwen3-coder-next, :260-275) and/or model_fit.py/profile_fit.py gating it off unified-memory hosts, at minimum a notes= warning and demotion from the hal0-Max bundle's chat.coder slot (ARCHITECTURE.md:844). Size: ~20-40 lines + one bundle-manifest edit. Risk: low (pure catalog data) — but verify against hal0's actual llama-server/ROCm build first; ODS's repro was against llama.cpp b9014 specifically (see F.1).
  2. Fleet-verified per-app compatibility metadata shape — port the schema idea: app_compatibility.{consumer}.{status,label,reason,evidence,recordedAt} (config/model-library.json:9-63) as an optional field on CuratedModel/Model.metadata. Size: ~30 lines schema addition; populating it is a separate, larger effort. Risk: medium — no fleet harness exists in hal0 today to populate it.
  3. Re-verify-on-existing-file SHA256 check — hal0 already streams SHA256 during pull (pull.py:739); the portable idea is ODS's install-time re-verify-on-existing-file check (installers/phases/11-services.sh:697-722) so a corrupted on-disk file is caught and auto-redownloaded outside the pull path. hal0 target: a verify_installed(model_id) helper in registry/pull.py, callable from a CLI/API path. Size: ~40-60 lines. Risk: low.
  4. Bootstrap-then-hot-swap fast start — for hal0-Lite/hal0-Default first-run tiers: a small model synchronously so hal0-api is usable within seconds, then background-pull + hot-swap the tier's real pick. Source pattern: installers/lib/bootstrap-model.sh + installers/phases/11-services.sh:670-690,1328-1362 + scripts/bootstrap-upgrade.sh. hal0 target: generalize src/hal0/install/brain_model.py or a new install/bootstrap_chat.py paired with the bundle picker (ARCHITECTURE.md:825-859). Size: medium (200-400 lines — needs a background-task mechanism plus a hot-swap reusing run_pull_and_activate). Risk: medium — hal0's installer currently appears synchronous.
  5. model-router-style stable alias — not a straight port (different runtime shape), but the idea — one stable identifier a consumer binds to once — maps to hal0's existing hal0/brain virtual-model pattern and could generalize to hal0/chat for chat.primary so external consumers survive a model_swap unreconfigured. Size: large (src/hal0/dispatcher/). Risk: medium-high — touches the dispatcher's routing table and the omni-router's tool-calling gate; needs its own design pass.
  6. "Read-only after activation" config fields — ODS makes model/tier/context fields read-only in its Settings editor specifically so a hand-edit can't bypass verified activation (docs/MODEL-MANAGEMENT.md:194-199). hal0 target: audit slot_config's write_slot_toml() single-write-path claim for any UI/CLI path that still writes [model].default outside run_pull_and_activate. Size: audit only, <1 day. Risk: low.

E. Do-not-copy

  • scripts/pre-download.sh — stale, a different tier vocabulary and non-GGUF HF repos than the rest of ODS, no SHA256 check. Would only confuse hal0's already-cleaner pull model.
  • docs/MODEL-SWITCHBOARD.md as a literal process to follow — a planning artifact whose "not implemented" disclaimer is stale (model-router is built); cite the current code (extensions/services/model-router/app/main.py) instead of the plan.
  • A single-process, single-model inference server as the target shape — ODS's one-llama-server/tier-swap-the-whole-model design is a worse fit for hal0's multi-slot/capability-card architecture than what hal0 already has. Port the verification discipline (D.3/D.6), not the single-process shape.
  • ODS's static memory-envelope heuristics wholesale (estimated_param_billions's filename regex, estimated_context_kv_gb's flat per-B-param constant) — hal0's live GTT-sysfs check is strictly better evidence where it applies; only worth adapting as a fallback for non-AMD/no-live-telemetry hosts.
  • Bundling fleet-evidence JSON paths (fleet-test/runs/...) into a public catalog file — meaningless dead links outside ODS's private harness. If hal0 adopts the schema idea (D.2), populate evidence from hal0's own repo/CI.

F. Owner decisions

  1. Verify or retire qwen3-coder-next as hal0's flagship Strix Halo coder pick — the single highest-value finding here. hal0's hal0-Max first-run bundle tier ("100 GB Strix Halo," hal0's own reference hardware class) sets chat.coder = Qwen3-Coder-Next-80B-A3B (ARCHITECTURE.md:844), and curated.py:260-275's entry is explicitly described as "Sized for a Strix Halo unified memory pool (~100 GB)" / "needs the full Strix Halo pool," pulling the same unsloth/Qwen3-Coder-Next-GGUF repo family ODS pulls. ODS's own fleet testing found this exact model produces "all-? tokens on every chat completion" on unified-memory backends including Strix Halo, verified on two different unified-memory architectures (installers/lib/tier-map.sh:100-138), and now hard-routes every unified-memory host away from it. Load this model on a real Ryzen AI Max+ 395 box and check for garbled/?-only output before shipping hal0-Max — if it reproduces on hal0's llama.cpp/ROCm build, this is a correctness bug in the flagship pick for hal0's own target hardware, not a nice-to-have.
  2. Decide whether hal0 wants a fleet-verification ledger at all. ODS's app_compatibility metadata is expensive (private, six-host harness) but genuinely differentiates its catalog. hal0 could invest in an equivalent (even single-host, CI-driven) pass, or explicitly document that curated entries are unverified beyond "it loads."
  3. Decide the rollback bar for slot/model swaps. ODS requires the previous model to re-prove identity+completion before a rollback counts as successful — meaningfully stronger than "the state machine transitioned back." Worth adopting the same discipline for chat.primary/brain slots that gate the dashboard's own usability.
  4. Decide whether mmproj pull failures need a louder story — with mmproj already a first-class two-file pull (pull.py:973-1111), is a partial pull (GGUF landed, mmproj failed) ever left silently vision-less rather than failing loud? Worth a targeted test if one doesn't exist.
  5. Scope any model-router analogue deliberately. Decide whether the actual pain point is "apps hardcode a concrete model id" (already solved for one slot by the hal0/brain virtual-model pattern) versus a genuine need for LiteLLM-style multi-consumer alias indirection — different problems, different right-sized solutions.
05-network-dns.md Networking, mDNS, auth posture

Networking, DNS/mDNS, LAN access, onboarding, and auth posture: ODS vs. hal0

Scope: /home/user/ods/ods (ODS, Apache-2.0) vs. /home/user/hal0 (hal0). All paths below are repo-relative to those roots unless given in full. Both repos were read-only; nothing was modified.

A. ODS mechanism

A1. mDNS and the single-:80 entry

ODS's .local story has two independent layers, and the docs are explicit about how they compose (docs/MDNS.md:5-12, docs/ODS-PROXY.md:1-20):

  • Name → IP: bin/ods-mdns.py (381 lines) is a standalone Python daemon, installed as scripts/systemd/ods-mdns.service, using python3-zeroconf — not a container, not host networking, a plain systemd unit as the install user. It polls .env every 30s (ODS_MDNS_POLL_INTERVAL, default 30, bin/ods-mdns.py:93) and republishes on change; it's a no-op on macOS (Bonjour already does this) and unimplemented on Windows (bin/ods-mdns.py:338-350). _build_services() (bin/ods-mdns.py:173-259) emits two record shapes: direct-port SRV records (<device>-chat._http._tcp.local → :3000), gated on BIND_ADDRESS actually being LAN-facing (_direct_ports_lan_reachable, lines 161-171); and per-subdomain A records (chat., dashboard., auth., api., hermes., talk., bare <device>.local) that always resolve to the proxy's IP:port regardless of BIND_ADDRESS, since the proxy is the one thing meant to be reachable. The installer apt/dnf/pacman/zypper-installs python3-zeroconf or falls back to pip install --user (installers/phases/07-devtools.sh:479-511) and renders/enables the systemd unit (lines 518-563) — non-fatal on failure ("device still reachable by IP").
  • Port → service: extensions/services/ods-proxy is Caddy (caddy:2.11.3-alpine, extensions/services/ods-proxy/compose.yaml:5) doing host-based routing, not path-based — an earlier path-based draft broke Open WebUI's root-relative assets, websockets, and OAuth callbacks (docs/ODS-PROXY.md:22-30). The Caddyfile (extensions/services/ods-proxy/Caddyfile, 156 lines) has one http:// block per subdomain, e.g.:

caddyfile http://chat.{$ODS_DEVICE_NAME:ods}.local { request_body { max_size 200MB } reverse_proxy open-webui:8080 { header_up X-Forwarded-Proto {scheme} header_up X-Forwarded-Host {host} } } http://{$ODS_DEVICE_NAME:ods}.local { redir http://chat.{$ODS_DEVICE_NAME:ods}.local{uri} 302 } (extensions/services/ods-proxy/Caddyfile:66-78,154-156)

The proxy is the one exception to the loopback-by-default rule: ODS_PROXY_BIND defaults to 0.0.0.0 while every other service keeps BIND_ADDRESS=127.0.0.1 (extensions/services/ods-proxy/compose.yaml:26-31, extensions/services/ods-proxy/manifest.yaml:55-60). The ods-session cookie is Domain=<device>.local-scoped (ODS_COOKIE_DOMAIN, installer-set), so one magic-link redemption on auth.<device>.local authenticates chat/dashboard/hermes subdomains too (docs/ODS-PROXY.md:32-37). TLS is out of scope for v1 — HTTP only, with Tailscale certs or a self-signed CA documented as follow-ups (docs/ODS-PROXY.md:80-89).

docs/NETWORK.md covers a third, unrelated thing: Wi-Fi joining (dashboard → dashboard-api → ods-host-agent(root) → nmcli, docs/NETWORK.md:19-28), Linux+NetworkManager only, 501 elsewhere.

A2. Docker-internal DNS and ports

docker-compose.base.yml:7 names the project ods; the network is pinned explicitly: networks: default: name: ods-network (docker-compose.base.yml:473-475). Services address each other by Compose service name on the internal port (http://llama-server:8080, docker-compose.base.yml:94,217,251; searxng:8080, line 117). Only dashboard-api gets extra_hosts: host.docker.internal:host-gateway (docker-compose.base.yml:197-198), used for the Apple-silicon path where llama-server runs natively on the host and containers reach it via host.docker.internal:8080 (ARCHITECTURE.md:120).

config/ports.json (135 lines) is the canonical external-port contract — one JSON array of {env_var, external_default, service_id, internal_port, manifest_service} per service (llama-server→11434/8080, open-webui→3000/8080, dashboard→3001, dashboard-api→3002, litellm→4000, qdrant→6333/6334, etc.). Every port binding in every compose file is "${BIND_ADDRESS:-127.0.0.1}:${WEBUI_PORT:-3000}:8080"-shaped (docker-compose.base.yml:32,172,200,454), so one env var demotes/promotes every service at once. --lan sets exactly that var (install-core.sh:267: --lan) BIND_ADDRESS="0.0.0.0"...), and ods-proxy's own bind is a separate, proxy-only override (ODS_PROXY_BIND) precisely so --lan isn't required to get the friendly URL — docs/ODS-PROXY.md:57 explicitly warns against setting global BIND_ADDRESS=0.0.0.0 "just for this."

Phase 04 (installers/phases/04-requirements.sh:150-297) checks port conflicts with a three-tool fallback (lsof → ss → netstat, lines 154-213), special-cases a running host ollama process (lines 232-256), and only checks the ports of enabled features (lines 275-283) — voice, workflows, qdrant, comfyui ports are conditional.

A3. Headless / first-boot

Three complementary documents, all pointing at real code (docs/HEADLESS-SETUP.md:49-59 is the index):

  • AP mode (docs/AP-MODE.md, scripts/ap-mode.sh, scripts/systemd/ods-ap-mode.service) — the device hosts its own ODS-Setup-XXXX Wi-Fi AP via hostapd+dnsmasq, DNATs every :80/:443 request to the gateway IP (captive-portal trick, docs/AP-MODE.md:22-34), and lands the phone on the dashboard's /setup wizard through ods-proxy. Disabled by default — "Bringing up an AP is destructive... auto-enabling that would disconnect [users] from their own Wi-Fi" (docs/AP-MODE.md:7-11). Linux-only, requires NetworkManager release/reclaim of the wireless interface.
  • Setup cards (docs/SETUP-CARD.md, scripts/generate-setup-card.py) — a 4×6", 300 DPI PNG/PDF with a Wi-Fi QR (WIFI:T:WPA;S:...;P:...;;) and a setup-URL or owner-magic-link QR side by side, meant to ship in the box (docs/SETUP-CARD.md:6-16).
  • Terminal QR fallback (lib/qrcode.sh, 153 lines) — print_dashboard_qr() shells out to qrencode -t ANSIUTF8 if present, else falls back to an ASCII box (lib/qrcode.sh:32-53,56-68); print_success_card() prints a boxed local+LAN URL summary plus the QR (lib/qrcode.sh:75-127). LAN IP detection reads the src field of ip route get 1.1.1.1 (lib/qrcode.sh:16-28).
  • installers/mobile/install-mobile.sh (23 lines) is a stub — it detects android-termux/ios-ashell and then exits 1 with "Mobile preview installer is split into a follow-up PR" (lines 12-18). Not implemented today.
  • EDGE-QUICKSTART.md is marked "Status: Planned — Not Yet Available" (line 3); docker-compose.edge.yml does not exist.
A4. Remote access + auth
  • Tailscale (docs/TAILSCALE.md, extensions/services/tailscale/) — network_mode: host so the tailnet IP lands on the host itself (extensions/services/tailscale/compose.yaml:21); the doc is emphatic that this alone does nothing — ods-proxy + BIND_ADDRESS=0.0.0.0 are still required for anything to answer on the tailnet interface (docs/TAILSCALE.md:47-54). Opt-in, Linux-only, no Funnel by default.
  • remote-provider-egress / remote-provider-ssh-tunnel are not user-facing remote access — they are internal-only (port: 0, external_link: false), an "egress boundary" that injects a private API key at the last hop for optional cloud LLM providers (extensions/services/remote-provider-egress/manifest.yaml:22-26) and an SSH transport supervisor for the same (extensions/services/remote-provider-ssh-tunnel/manifest.yaml:22-25). Do not confuse with Tailscale — this is outbound-to-a-vendor, not inbound-for-a-user.
  • hermes-proxy gates the Hermes agent with real cryptographic verification, not header-presence: Caddy forward_auth to dashboard-api:3002/api/auth/verify-session (extensions/services/hermes-proxy/Caddyfile:101-129), which HMAC-verifies the ods-session cookie (extensions/services/dashboard-api/routers/auth.py:65-107). The Caddyfile comment is candid about the prior, weaker design: "An earlier draft... only checked that the cookie HEADER was present... any LAN user can set Cookie: ods-session=foo" (hermes-proxy/Caddyfile:8-12). docs/HERMES-SSO.md:9-13 is equally candid that this is gateway auth, not per-user auth: Hermes itself is one process with one session token baked into the SPA HTML, so "anyone with a valid invite" shares one identity.
  • Magic links (extensions/services/dashboard-api/routers/magic_link.py, 896 lines) — 32-byte secrets.token_urlsafe tokens, only the SHA-256 hash persisted, single-use guest links (60 min TTL) vs. reusable revoke-only owner cards, 5-failed-attempts/min rate limit, QR rendered server-side via the qrcode PyPI package into a data:image/png;base64 URL (magic_link.py:376-395). dashboard-api/security.py (43 lines) separately gates the admin /api/* surface with a Bearer DASHBOARD_API_KEY, auto-generated and written to /data/dashboard-api-key.txt (mode 0600) if unset (security.py:13-23).
  • docs/MULTI-USER-SETUP.md is explicit that the out-of-the-box posture is single-user: "Binds every service to 127.0.0.1... Launches llama-server with --parallel 1" (docs/MULTI-USER-SETUP.md:15-19); going multi-user is a six-step opt-in the operator does deliberately. docs/OAUTH_PROVIDER_SETUP.md is a separate, narrower thing — an OAuth callback capture helper for Hermes skill setup, metadata-only, no secrets committed.
A5. Security posture summary

Default posture: everything on 127.0.0.1 except ods-proxy, which is LAN-open by design once enabled (SECURITY.md:55-63). --lan (or BIND_ADDRESS=0.0.0.0 + ods restart) widens every service at once (SECURITY.md:57-68); the host-agent has its own separate, narrower ODS_AGENT_BIND that auto-detects the Docker gateway IP on Linux rather than defaulting to 0.0.0.0 (SECURITY.md:84-102). Behind the proxy, each backend keeps its own check — dashboard-api's Bearer key, Open WebUI's own WEBUI_AUTH, hermes-proxy's forward_auth — the proxy itself "adds NO auth layer... duplicating without strengthening" (docs/ODS-PROXY.md:62-72). privacy-shield (extensions/services/privacy-shield, port 8085) is content-layer, not access-layer — a PII-scrubbing proxy in front of the LLM, unrelated to who's allowed to connect.

B. hal0 today (verified by grep)

Correction to the task's framing: grep -rn -i 'avahi\|mdns\|\.local\|zeroconf\|bonjour' across hal0 returns over 300 matches in src/, installer/, ui/, tests/, and docs/ — hal0 already has a real, tested mDNS subsystem. It is architecturally different from ODS's, not absent.

  • src/hal0/services/mdns.py (150 lines) is the mirror of bin/ods-mdns.py, but it does not run its own responder — it writes standard avahi service-group XML files that the system's avahi-daemon inotify-watches (services/mdns.py:10-11), the same mechanism ODS's macOS path leans on natively. mdns_hostname() returns <HAL0_HOSTNAME or socket.gethostname()>.local (services/mdns.py:40-49); advertise()/withdraw() write/prune one hal0-addon-<id>.service file per LAN-published, mDNS-capable service (services/mdns.py:100-141):

python def _service_group_xml(name: str, port: int) -> str: safe = escape(name) return f"""<?xml version="1.0" standalone='no'?> <service-group> <name replace-wildcards="yes">{safe} on %h</name> <service> <type>_http._tcp</type> <port>{port}</port> </service> </service-group> """ (services/mdns.py:81-97, abridged)

Only two services are mdns=True in the catalog: services/registry.py:65-114 — OpenWebUI (:3001) and ComfyUI (:8188). Hermes (:9119) and Hindsight (:9177) are loopback_port-only with no mDNS/host:port fallback at all (registry.py:96-113) — closer to ODS's "loopback stays loopback" instinct than the headline framing suggests. - This is wired to a real UI: ui/src/dash/services.jsx has a DiscoveryCard ("DISCOVERY (mDNS)") with an avahi active/inactive dot and an advertise/withdraw toggle (services.jsx:279-315), backed by GET/POST /api/services/mdns (api/routes/services.py:269-311, useMdnsAdvertise/useMdnsStatus in useServices.ts:114-119), and covered by tests/api/test_services_page.py:262-309 and tests/installer/test_avahi_hostname.py (avahi-daemon.conf host-name= pinning, installer/install.sh:1372-1451). - Gap found in this subsystem: mdns.status() reports base_advertised: (services_dir()/"hal0.service").is_file() (services/mdns.py:76), i.e. whether hal0-api itself (not an addon) is announced. Nothing in the current tree writes that file — the docstring says "if something else drops" it (services/mdns.py:3), and CHANGELOG.md:3676 records a stale avahi/hal0.service unit being deleted, not a service-group file being added. In practice base_advertised is very likely always False on a stock install; the dashboard's own copy says as much ("No base hal0.service avahi file — the installer writes it when avahi is present," services.jsx:304) — a promise the installer does not currently keep.

Bind posture — confirmed exactly as briefed, with one nuance the brief undersells:

  • src/hal0/install/network.py:43: DEFAULT_BIND_HOST = "0.0.0.0" — this is what install.sh seeds into /etc/hal0/api.env. hal0-api binds 0.0.0.0:8080 on a stock install, no --lan equivalent needed.
  • src/hal0/openwebui/env_writer.py:164: OpenWebUI's companion follows the same HAL0_BIND_HOST (also defaulting to 0.0.0.0) — but this is a fix, not the original design: CHANGELOG.md:3091 documents #1515/#1514, where OpenWebUI used to hardcode -p 0.0.0.0:3001:8080 regardless of the operator's HAL0_BIND_HOST, so someone who bound the API to loopback got "a wide-open chat UI on :3001 regardless." WEBUI_AUTH=False unless HAL0_OWUI_TRUSTED_EMAIL_HEADER is set (env_writer.py:166-172).
  • Nuance: inference slot containers (llama-server, the NPU trio, ComfyUI-as-slot) are not on HAL0_BIND_HOST — they publish loopback-only by default ([slots].publish_host, fail-soft to 127.0.0.1, src/hal0/providers/container.py:17-23,550-561), and under Network=host the code rewrites any --host 0.0.0.0/--listen 0.0.0.0 in the child's own argv back to loopback, since host networking has no PublishPort fence (container.py:610-651). So: control-plane API + chat UI default open, raw inference ports default closed. src/hal0/config/network.py:34's own library fallback (no env at all) is also loopback — it's specifically the installer's choice to write 0.0.0.0 into api.env that makes the shipped default LAN-wide.
  • src/hal0/api/agents/_auth.py:1-8 states the underlying incident plainly: "exposing a long-running JSON-RPC bridge to the hermes runtime over an unauthenticated 0.0.0.0:8080 is LAN-RCE."

src/hal0/service_identity.py (307 lines) resolves the box's own service-to-service key (HAL0_ADMIN_KEY/HAL0_CLIENT_KEY, env then /etc/hal0/api.env, service_identity.py:71-88) for hal0's own internal callers, and owns key rotation (rotate_api_env_key, atomic tmp+rename, 0640, service_identity.py:148-234) plus keeping hindsight-llm.env in sync (refresh_hindsight_llm_env, lines 245-296).

src/hal0/security/ — exposure.py (436 lines) is a single ordered, first-match-wins classification table (RULES, lines 153-326) sorting every route into OPEN / BOOTSTRAP / CLIENT / ADMIN (AuthClass, lines 71-77), deny-by-default for anything unlisted (classify(), lines 346-354). ratelimit.py (138 lines) backs the login and key-rotation rate limiter. This is consumed by src/hal0/api/auth.py (542 lines): AuthEnforcementMiddleware is a pure-ASGI gate (lines 431-530) resolving a request to anon/client/ admin via cookie → Bearer → ?api_key= (resolve_principal, lines 289-321), plus an Origin-based CSRF check for browser-driven state changes (_origin_allowed, lines 344-382). Crucially, require_auth_enabled() defaults to False unconditionally (api/auth.py:128-157) — a documented reversal: an earlier KB-1 design auto-enabled enforcement whenever the bind was non-loopback or a key existed, and that "locked operators out of a dashboard that shipped no login UI... so they disabled auth wholesale" (api/auth.py:137-145). So today: LAN-open bind and auth-off are both the shipped default, independently.

src/hal0/ports/ — authority.py (389 lines) is PortAuthority, the single writer of port ownership (one port_claim DB row per live slot, partial-unique-indexed, authority.py:1-19); __init__.py (229 lines, not fully read) is the read-side harvester. Unlike ODS's static config/ports.json contract, hal0's port map is dynamic and allocated at runtime per slot — there is no single canonical file enumerating "which port serves what," except the small, fixed services/registry.py catalog for the four companion services.

Docs already say this out loud: README.md:563-565 ("Auth ships off by default. A fresh v1.0 install is trusted-LAN-open — the right posture for a homelab appliance"), docs/operate/auth.mdx (336 lines, full three-tier reference, <Aside type="caution"> at lines 15-27), and docs/concepts/security.mdx:3 ("hal0 ships open on your LAN by default, but v1.0 adds a real, opt-in three-tier auth layer"). Install itself prints a QR to the dashboard URL via qrencode when present (installer/install.sh:3908-3916, HAL0_NO_QR=1 to skip) and an explicit, if quietly-styled, line in the same summary box: "Auth open on the trusted LAN — front with a reverse proxy if exposed" (installer/install.sh:3934).

Issue #1822, pinned to the exact gap: src/hal0/cli/doctor_all.py:62-83, the one function whose entire job is to flag auth misconfiguration:

def check_auth_posture(auth: dict[str, Any] | None) -> Check:
    """... The only misconfiguration we flag is "auth required but no admin
    key configured" (nobody can log in); an intentionally open dev install
    passes with a note.
    """
    ...
    if not required:
        return Check("auth", "Auth posture", _PASS,
                      "open (auth not required — dev/loopback)")

grep -n 'bind_host\|BIND_HOST\|0\.0\.0\.0' src/hal0/cli/doctor_all.py returns nothing: this check never looks at HAL0_BIND_HOST at all. A stock install — 0.0.0.0 bind, auth off — gets an unconditional green PASS whose own message ("dev/loopback") describes a configuration the box is not actually in. hal0 doctor all / hal0 doctor verify and GET /api/doctor (security/exposure.py:252 classifies it ADMIN) are the tools built precisely to catch this class of drift, and none of them cross-check bind against auth. The install-time line (install.sh:3934) and the Settings ▸ Security page (ui/src/dash/settings/pages/server/ SecurityPage.jsx:154: "Auth is off — hal0 runs trusted-LAN open") are the only two surfaces that say it, and both require the operator to already be looking in the right place, once, rather than being told by the tool whose job is to tell them what's wrong.

One historical note for anyone tracing "why no Caddy/auth already": CHANGELOG.md:5050-5053 records v0.3.0-alpha.1 as "Caddy and the auth surface are removed" per ADR-0012, which superseded ADR-0001's softer plan. docs/adr/ on disk (at 108b366) only goes up to 0006 and has no 0012 (reconstructed into the tree by #2244 on main, together with 0004, 0013, 0015, 0020 and 0023) — this is not drift, CHANGELOG.md:14 explains ADRs are filed under a docs/internal/ tree that is gitignored (issue #638), same convention as this repo's current docs/.devdocs//docs/superpowers/. So ADR-0012's own reasoning is genuinely unavailable in this checkout; only its effects (the Caddy+auth removal, and the later KB-1 opt-in re-addition) are reconstructable from CHANGELOG.md and the code. ARCHITECTURE.md: 111-113 ("dedicated auth packages... were removed") is accurate about that historical cut but silent about api/auth.py/security/exposure.py existing today as an opt-in, off-by-default replacement — a live instance of this repo's own "verify against source, not memory" rule.

C. Better / worse / equivalent

Dimension ODS hal0 Verdict
Name→IP mechanism Own zeroconf daemon + systemd unit + pip/apt dep System avahi-daemon + plain XML files, no extra process hal0 lighter, equivalent outcome
Port→service Single Caddy on :80, host-based vhosts, one URL Each own port; dashboard derives links from request Host (_behind_proxy/_resolve_host, services.py:38-73) ODS better UX; hal0 needs no proxy at all for LAN-IP/raw-port use
Canonical port map config/ports.json, static, one file services/registry.py (4 svcs) + dynamic PortAuthority DB for slots Different shapes, each fits its own domain
Default bind (control plane) Loopback; proxy is the sole opt-in LAN surface LAN-open (0.0.0.0) by default ODS safer default
Default bind (inference) Loopback (BIND_ADDRESS) Loopback ([slots].publish_host) Equivalent
Auth model Cookie SSO across Caddy subdomains + per-service checks + magic links Deny-by-default classifier, three tiers, off by default hal0 more rigorous design; ODS safer default
Auth-off visibility N/A — LAN needs --lan, an explicit act Told once at install + in Settings, but doctor_all.py:75 actively says "PASS" ODS better — hal0's own health-check contradicts its docs
Headless/QR onboarding Full stack: AP mode, setup cards, lib/qrcode.sh, mobile stub One QR to the dashboard URL at install; no AP mode, no card ODS far ahead
Remote access Tailscale extension, documented prerequisites, opt-in None ODS ahead (feature gap)
Egress-boundary services remote-provider-egress/ssh-tunnel: real, scoped, internal-only No equivalent (upstreams/ handles external providers differently) Different architectures, not a gap

D. Port candidates

# Candidate hal0 target file(s) Size Risk Notes / excerpt
1 Close the #1822 gap: cross-check bind + auth in the doctor src/hal0/cli/doctor_all.py:62-83 (check_auth_posture); needs the daemon's bind as the signal — GET /api/config/urls (api/routes/config.py; add the bind host to it if it does not carry one) or a parse of /etc/hal0/api.env, the unit's EnvironmentFile (install.sh:1605), the way service_identity.py:47-57 already reads it — not hal0.config.network.bind_host() from the CLI process: doctor runs in the operator's shell, so that call sees no HAL0_BIND_HOST and returns its loopback fallback (network.py:37-45), reproducing the false PASS (correction 2026-10-04) ~20-30 line diff + 1 test Very low — read-only, additive, no behavior change Demote to _WARN when <daemon bind> not in _LOOPBACK_BIND_HOSTS and not required; message: "open on a LAN-reachable bind — run 'hal0 auth require on' or set HAL0_BIND_HOST=127.0.0.1". This is the single highest-leverage, lowest-risk fix in this whole report.
2 Write the missing base avahi file src/hal0/services/mdns.py (extend advertise()/add a advertise_base()); hook from installer/install.sh near the existing avahi block (install.sh:835-851) ~30-50 lines Low — same fail-soft, tmp+rename pattern already proven for addons Makes base_advertised (services/mdns.py:76) true on a stock install, and the dashboard's own caption (services.jsx:304) stop describing a promise nothing keeps.
3 Single-port LAN gateway (ODS-proxy-equivalent) New packaging/hal0-proxy/Caddyfile + a systemd/quadlet unit; installer/install.sh hookup; must respect X-Forwarded-Host in _behind_proxy/_resolve_host (services.py:38-73) so links don't regress to raw ports ~150-250 lines Medium — WS/SSE passthrough needs the same header-stripping care hermes-proxy/Caddyfile:104-119 documents; must not fight hal0's existing request-Host link trick (useConfigUrls.ts:1-12) Ports almost verbatim from ods-proxy/Caddyfile's chat.<device>.local block (lines 66-78), service names swapped for 127.0.0.1:8080/:3001/:8188. Correction (2026-10-04) — two prerequisites this row omitted. (a) A stock install seeds API_BIND_HOST="0.0.0.0" (installer/install.sh:157) into api.env and publishes Open WebUI on ${API_BIND_HOST}:3001 (install.sh:1567), so a forward_auth listener on :80 leaves :8080 and :3001 reachable directly and the auth is bypassable; the gateway work must take every upstream off the LAN interface and keep that converged across upgrades before it is the single audited edge — and not by HAL0_BIND_HOST=127.0.0.1 alone: Open WebUI and every bridge-network extension reach hal0-api through host.docker.internal, the podman bridge gateway address, not loopback (openwebui/env_writer.py:102-128), so hal0-api needs a bridge-reachable listener (second bind or shared internal network) or the LAN cut is an nftables rule that exempts the bridge subnet (report 02 §D.5 (iii)). (b) src/hal0/services/mdns.py:82-96 renders _http._tcp service records on the machine hostname and publishes no A/AAAA records, so chat.<host>.local needs an address publisher (avahi-publish -a per name, or a zeroconf daemon like ODS's bin/ods-mdns.py:173-187) whose name set is pinned to the Caddyfile the way ODS's tests/test_mdns_subdomains.py pins it.
4 Headless setup card + two-QR onboarding New installer/lib/qrcode.sh-equivalent (model: lib/qrcode.sh, 153 lines) + a hal0 setup-card command reusing detect_lan_ips (install/network.py:58-96) ~100-150 lines Low — additive, operator-invoked, no daemon hal0 already has the terminal-QR primitive (install.sh:3908-3916); the gap is a printable card and, separately, a phone-driven first-boot path — install.sh is a foreground console script, not a served wizard.
5 Elevate the install-summary auth line to a real warning installer/install.sh:3934 (currently ${DIM}-styled) Trivial Very low Cheapest partial mitigation for #1822 if the doctor fix (D1) is deferred.
6 AP mode / true out-of-box Wi-Fi setup Greenfield: hostapd/dnsmasq/iptables orchestration, captive portal, wizard integration 300-600+ new lines + unit + hardware testing Medium-high — ODS itself ships this disabled, calling it "destructive" (docs/AP-MODE.md:7-11) Worth it only if hal0 moves toward a sealed-appliance framing, not "operator runs the installer on their own box."

E. Do-not-copy

  • ODS's dedicated zeroconf daemon wholesale. bin/ods-mdns.py + python3-zeroconf + its own systemd unit exists because ODS wants fine-grained control over SRV and subdomain records independent of whatever avahi happens to be doing. hal0's existing avahi-XML approach (services/mdns.py) gets the same .local resolution with zero extra process, zero extra dependency, and no polling loop — for hal0's simpler shape (one host, a couple of LAN ports) it is strictly less to run and maintain. Adopting ODS's daemon would be a regression in simplicity for no discovery-fidelity gain hal0 currently needs.
  • The ODS_COOKIE_DOMAIN=<device>.local multi-subdomain SSO trick in isolation. It solves a problem (one login across chat./dashboard./ hermes. origins) that only exists because ODS chose host-based routing. hal0's hal0_session cookie (api/agents/_auth.py) is already host-only and origin-checked; grafting a Domain= scope on without first adopting subdomain routing does nothing useful and widens the cookie's reach for no reason.
  • AP mode, unless the product direction genuinely wants a sealed appliance. ODS itself treats it as dangerous enough to ship disabled (docs/AP-MODE.md:7-11); it is Linux-only, chipset-dependent (docs/AP-MODE.md:137), and every failure mode stops the box from joining any network at all.
  • remote-provider-egress/ssh-tunnel as a "remote access" model. They are outbound egress-boundary services for optional cloud LLM providers, unrelated to letting a user reach the box from outside. Don't reach for them when the actual ask is "Tailscale for hal0."
  • The full magic-link/owner-card persistence subsystem (magic_link.py, 896 lines) as a literal port. It's built for a multi-person-household product frame (revoke-only owner cards, guest invites, redemption audit trail). hal0's KB-1 model (one admin-equivalent browser session, two static keys) is deliberately simpler for a single-operator appliance; porting the whole subsystem imports a product assumption hal0 hasn't made yet.

F. Owner decisions

  1. Fix #1822 now, cheaply. Candidate D1 (doctor cross-check) is a ~20-line, no-risk change that stops hal0 doctor all from actively asserting "PASS... dev/loopback" on a box that is neither. This should ship regardless of anything else below.
  2. Pick a LAN-exposure model deliberately, don't drift into one. Either (a) keep today's "each service its own port, links follow request Host" shape and simply default HAL0_BIND_HOST to loopback the way hal0.config.network's own library fallback already does (config/network.py:34), pushing LAN exposure behind an explicit opt-in the way ODS's --lan does — not as a one-line default change (correction 2026-10-04): a loopback-only hal0-api is unreachable from Open WebUI and bridge extensions, which use host.docker.internal, so (a) also needs the bridge-reachable second listener or LAN-ingress firewall cut described in D3 and report 02 §D.5 (iii); or (b) build the single-gateway proxy (D3) and accept the added moving part in exchange for one URL and one audited edge. KB-1's own history (api/auth.py:137-145) shows an auto-enable-on-LAN-bind policy was tried and reverted because there was no login UI to catch the user — that objection no longer holds now that docs/operate/auth.mdx and the Settings ▸ Security page exist, so revisiting a safer default bind is worth a fresh look, not a repeat of the same argument.
  3. Decide how much headless/appliance investment is warranted. ODS's full stack (AP mode, setup cards, mobile installer) is aimed at ship-in-a-box, hand-to-a-relative use. If hal0 stays "operator runs the installer on their own hardware," candidates D4/D5 (QR card, sharper warning) are enough; D6 (AP mode) is only worth it if hal0 explicitly adopts the sealed-appliance framing.
  4. Close the dead base_advertised path (D2) — small, and removes a UI caption that currently describes something the installer doesn't do.
  5. Scope remote access separately from egress. If Tailscale-for-hal0 is wanted, design it as its own feature against hal0's actual shape (single systemd unit, not docker-compose extensions) rather than by analogy to ODS's remote-provider-* services, which solve a different problem.
06-mcp-memory.md MCP and memory

06 — MCP & Memory: ODS vs hal0

Source-level comparative study. All paths absolute. ODS = /home/user/ods, hal0 = /home/user/hal0.

Headline correction to the brief. The task premise — "how Hermes's cli-config.yaml.template declares mcp_servers (stdio vs http, env passing, per-server toolsets)" — is false against the tree on disk. ODS ships no MCP anywhere in its product runtime. There is no mcp_servers block, no MCP client, no MCP server, no MCP UI, no MCP CLI verb. Conversely, hal0 already ships ~70% of the "user-added MCP servers" feature the owner asked for (registry, REST API, CLI, manifest resolver, SSRF guard) — it is unfinished, not absent. Both findings reframe the deliverable: there is nothing to port from ODS on the MCP axis, and the hal0 work is closing a loop, not starting a feature.


Addendum, 2026-10-04 (post-review). This report is pinned to hal0ai/hal0@108b366. Two things have happened since.

  1. Main shipped most of section E. ADR-0015 (docs/adr/0015-mcp-supervisor-and-exposure.md, accepted for http/sse) and #2253 (df60d5b) added [secrets]/[tools]/[exposure] to InstalledServer, the Hermes/brain exposure join (src/hal0/mcp/hermes_join.py), hal0 mcp test|allow|gate|block|expose, and POST /{id}/test, PATCH /{id}/tools, PATCH /{id}/exposure. The stdio supervisor is deferred (409 mcp.exposure_needs_supervisor); the dashboard drawer is #2245. docs/adr/ now carries 0004, 0012, 0013, 0015, 0020 and 0023 (#2244), which resolves the bookkeeping remarks in §E.2 and §G.3.
  2. Three design errors in §E.2–E.3 were found in review and are corrected inline below, each marked Correction (2026-10-04). In short: (i) the builtin loop at hermes_provision.py:2012-2021 attaches hal0's own admin service key, so it must not be reused verbatim for third-party URLs — main's hal0.mcp.probe.build_headers sends only X-hal0-Agent, the record's [env] literals and its resolved [secrets], which is the right behaviour; (ii) mirroring [tools] into hermes.toml describes a policy but does not enforce it, because Hermes's own MCP client connects straight to mcp_servers.<id>.url and AgentMCPClient is never instantiated in production (true at 108b366, still true at main c177aa5) — the two builtins are gated server-side inside hal0's /mcp mount, a third-party server has nothing in its path; main inherited this and it is filed as hal0 issue #2303; (iii) a stdio server cannot run as a free-standing systemd unit, because its client must own its stdin/stdout.

A. ODS mechanism

A.1 MCP in ODS: it does not exist

An exhaustive case-insensitive sweep over the ODS tree returns seven files, and every one of them is either a Claude Code developer-tooling artifact or a passing English mention:

  • /home/user/ods/.claude/commands/{code-review,deep-research,team-plan}.md — Claude Code slash commands that call mcp__pal__* / mcp__rube__* tools on the maintainer's workstation. Not shipped, not installed, not part of ods/.
  • /home/user/ods/ods/docs/MDNS.md:43 — "These exist for MCP clients, service-discovery tools, and the eventual ODS mobile app". Aspirational prose about mDNS _http._tcp records.
  • /home/user/ods/ods/bin/ods-mdns.py:179 — same, in a comment.
  • /home/user/ods/ods/extensions/library/services/gaia/README.md:6 — describes a third-party service (GAIA) that has MCP support; ODS does not wire it.
  • /home/user/ods/ods/extensions/services/dashboard/package-lock.json — an npm transitive substring match.

Point-by-point against the brief's specific claims:

Claim Verdict Evidence
Hermes cli-config.yaml.template declares mcp_servers False /home/user/ods/ods/extensions/services/hermes/cli-config.yaml.template is 186 lines: model, providers, auxiliary, agent.disabled_toolsets, terminal, platforms, compression. No mcp_servers.
OpenCode is given MCP servers False /home/user/ods/ods/installers/phases/07-devtools.sh:196-274 writes opencode.json with $schema/model/small_model/provider only. /home/user/ods/ods/extensions/services/opencode/manifest.yaml and README.md never mention MCP.
Open WebUI gets MCP/tool servers False /home/user/ods/ods/docker-compose.base.yml:100-168 is the full open-webui env block: web search, RAG embeddings, ComfyUI, STT, TTS. No TOOL_SERVER_CONNECTIONS, no mcpo.
n8n MCP nodes wired False /home/user/ods/ods/config/n8n/ holds 19 workflow JSONs; the three RAG ones (06-rag-demo.json, rag-pipeline-trigger.json, document-qa.json) contain only manualTrigger + stickyNote nodes — they are documentation stubs, not pipelines.
Dashboard has MCP UI False No mcp string in /home/user/ods/ods/extensions/services/dashboard/src/.
APE mediates MCP tool calls False (and worse — see A.3) APE is a standalone HTTP service nothing calls.
Users can add their own MCP servers N/A No mechanism, no docs, no preservation story. grep -c mcp /home/user/ods/ods/ods-cli → 0.

The nearest thing ODS has to "user extends the agent's tool surface" is subtractive: cli-config.yaml.template:101-104 sets agent.disabled_toolsets: [terminal, browser], relying on Hermes's upstream built-in toolsets. Widening the surface means hand-editing data/hermes/config.yaml per the upstream docs (docs/HERMES.md:121-130).

A.2 Memory in ODS

memory-shepherd (/home/user/ods/ods/memory-shepherd/) is not pruning, compaction, or backup of a vector store. It is a 338-line bash script implementing a baseline-reset contract on a markdown file:

  • The agent's MEMORY.md is split by a --- separator (memory-shepherd.sh:112). Above the line is operator-owned identity/rules/pointers; below is agent scratch.
  • Every 3 hours a systemd timer archives everything below the separator to archives/<agent>/<TIMESTAMP>.md and atomically restores the baseline (memory-shepherd.sh:151-180).
  • Safety rails: refuses to reset from a baseline under min_baseline_size (default 500 bytes, memory-shepherd.sh:134), full-file backup when the separator is missing (:172-174), lock file, 30-day archive retention.
  • Rationale in README.md:7-13: agents drift from role, bloat context, and "sometimes rewrite their own instructions".

Install wiring is odd: it lives in /home/user/ods/ods/installers/phases/10-amd-tuning.sh:63-95 (the AMD tuning phase), enabling memory-shepherd-workspace.timer and memory-shepherd-memory.timer. Its shipped baselines still target /home/deploy/.../.openclaw/workspace/MEMORY.md — OpenClaw, the deprecated agent (ARCHITECTURE.md:63). So the mechanism is real and well-built but points at a retired surface.

Hermes memory is upstream Hermes's own: HERMES_HOME=/opt/data mounted from data/hermes/ with sessions/, memories/, skills/ (docs/HERMES.md:68-77). ODS contributes only compression tuning (cli-config.yaml.template:171-186: threshold: 0.75, target_ratio: 0.50, protect_last_n: 40, each with a paragraph of live-testing rationale). There is no ODS-side memory API, no cross-agent memory, no namespace model.

Qdrant + TEI: deployed, unwired. This is the sharpest memory finding. qdrant runs (/home/user/ods/ods/extensions/services/qdrant/compose.yaml), embeddings runs TEI (extensions/services/embeddings/compose.yaml:3, BAAI/bge-base-en-v1.5), and the README asserts "These vectors are stored in Qdrant" (extensions/services/embeddings/README.md:7). But nothing writes to Qdrant. Open WebUI is pointed at TEI for embeddings only (docker-compose.base.yml:123-129: RAG_EMBEDDING_ENGINE=openai, RAG_OPENAI_API_BASE_URL=http://embeddings:80/v1) and keeps its own default vector DB — there is no VECTOR_DB=qdrant / QDRANT_URI env anywhere in the compose tree. Perplexica, n8n, and Hermes never reference it. Qdrant is a running, port-bound, API-keyed service with zero consumers.

Open WebUI local-service wiring (the part that does work, docker-compose.base.yml:115-167): SearXNG search, TEI embeddings, ComfyUI image-gen with an inline SDXL-Lightning 4-step workflow JSON (:141-156), Whisper STT at http://whisper:8000/v1, Kokoro TTS at http://tts:8880/v1. Every integration is expressed as OpenAI-compatible base URLs — a clean, copyable pattern.

Langfuse is opt-in and reaches LLM traffic only through LiteLLM: the container entrypoint appends 'langfuse' to litellm_settings.success_callback when LANGFUSE_ENABLED=true (/home/user/ods/ods/extensions/services/litellm/compose.yaml:49-54). token-spy rides the same seam as a callbacks entry (:46-47). Traffic that bypasses LiteLLM (Open WebUI → llama-server direct, Hermes → llama-server direct) is untraced.

A.3 Agent tool governance: APE

APE (/home/user/ods/ods/extensions/services/ape/main.py, 1078 lines) is a genuinely sophisticated policy engine — and nothing calls it.

The design is five layers evaluated in order in POST /verify (main.py:793-951):

  1. Circuit breaker (main.py:574-612) — trips to deny-all when the deny ratio over a rolling 300s window exceeds 0.5 with ≥20 samples; tripped state persists across restarts.
  2. Legacy per-minute rate limit (main.py:437).
  3. Intent classification (main.py:635-654) — verb-token matching on the tool name, falling back to arg-shape inference (command/cmd → ExecuteCommand, url → NetworkFetch).
  4. Policy evaluation (main.py:659-695) — four modes: allow, deny, allowlist (base-command allowlist + regex deny patterns), path_guard (realpath prefix check).
  5. Windowed multi-tier caps (main.py:482) — per-intent 5min/hour/day limits, each tier either deny or require_approval.

The require_approval tier is the best-engineered piece. /verify mints an approval_token bound to an args fingerprint (main.py:891-908); /approve (main.py:954-1010) consumes the token and mints a one-shot grant keyed to {session, tool, intent, args_hash}. The next matching /verify consumes the grant (strictly one-shot, deleted on use), counts the retry against the window so the bypass costs a sample, and re-escalates on a second retry (main.py:857-889). This is notably tighter than a plain "approve tool X" model.

Everything lands in an append-only audit.jsonl (main.py:700-706) including grant_consumed/approver provenance. State is persisted under a process lock plus advisory flock with bounded sample lists (main.py:255-262).

But: APE_STRICT_MODE defaults to false (compose.yaml:14), which logs without blocking (main.py:107-108 warns about exactly this). The manifest calls it a "Drop-in governance layer for OpenClaw" (manifest.yaml:26) — OpenClaw is deprecated. A tree-wide grep for callers of :7890 or /verify outside extensions/services/ape/ finds only dashboard health/restart plumbing (dashboard-api/config.py:544, settings.py:37). Hermes has no APE hook. The path_guard allowed_paths were even updated for Hermes's /opt/data/{workspace,skills,memories,sessions,...} (config/ape/policy.yaml:53-60) — someone maintained the policy for an integration that was never built.

A.4 The one genuinely excellent ODS pattern: generated-config discipline

/home/user/ods/ods/docs/INSTALLER-ARCHITECTURE.md:118-131 carries a Generated Config Writers table: every config surface × {Linux writer, macOS writer, Windows writer, upgrade/runtime writer}. The framing line is "When a bug involves generated config, check every writer before calling the fix done. This is the most common way install-time surprises survive a patch."

Two implementations of that discipline are directly reusable:

scripts/patch-hermes-config.py (324 lines) — a line-oriented, block-scoped YAML patcher that never round-trips through a YAML parser (so comments survive). _top_level_block / _child_block / _set_key (:17-59) locate a block by indentation and mutate only inside it. Crucially it encodes two distinct merge policies:

### operator wins — only fill a missing key (patch-hermes-config.py:114-115)
if max_tokens and not _has_key(lines, block, "max_tokens", 2):
    _set_key(lines, block, "max_tokens", str(max_tokens), 2)

### ODS converges — unconditionally re-assert (patch-hermes-config.py:215-218)
block = _set_key(lines, block, "enabled",      "true", 2)
block = _set_key(lines, block, "threshold",    "0.75", 2)
block = _set_key(lines, block, "target_ratio", "0.50", 2)
_set_key(lines, block, "protect_last_n",       "40",   2)

It returns changed: bool and writes only on diff (:291-294) — idempotent by construction. It is invoked against both the shipped template and the live config (scripts/bootstrap-upgrade.sh:1491-1506), plus an in-container variant (:1508-1518).

The OpenCode jq merge (installers/phases/07-devtools.sh:225-274) does the same for JSON with a three-way fallback: fresh-write if absent → surgical jq merge owning only .provider["llama-server"] → regenerate-from-template if the existing file is unparseable (issue #332).


B. hal0 today

hal0's MCP surface is substantially larger than the brief describes.

Two hosted servers. hal0-admin at /mcp/admin and hal0-memory at /mcp/memory, mounted as FastMCP sub-ASGI apps by /home/user/hal0/src/hal0/api/mcp_mount.py. That file solves a real problem well: FastMCP runs tool handlers in a lifespan-scoped anyio task group, so a Starlette middleware contextvar never reaches the handler — every write silently collapsed to the shared namespace (mcp_mount.py:15-27, issue #413). The fix reads headers off the MCP SDK's own request_ctx at call time. It also exposes HAL0_MCP_ALLOWED_HOSTS / HAL0_MCP_ALLOWED_ORIGINS to widen FastMCP's automatic localhost-only DNS-rebinding lockdown (:71-121).

Three-tier tool gating. AUTONOMOUS_READ_TOOLS / AUTONOMOUS_WRITE_TOOLS / GATED_TOOLS in src/hal0/mcp/admin.py; gated calls enqueue into an in-memory ApprovalQueue (src/hal0/mcp/approval_queue.py) and return {"status": "pending_approval", "approval_id": ...}. The queue dedups on (tool_name, primary_target) with a hit_count (approval_queue.py:16-22, 59-69) so a retrying agent doesn't flood the inbox. The owner resolves via /api/agent/approvals/{id}/{approve,deny} with an SSE tail that replays the pending set on subscribe (src/hal0/api/routes/approvals.py:13-21). approve executes the bound executor (approval_queue.py:30-33).

Default-deny MCP client policy. src/hal0/agents/mcp_client.py — pure, synchronous, no wire IO. enabled_servers() (:137-145) is the server axis; classify() (:149-180) is the tool axis, checking blocked → gated → allow → unknown_tool (default-deny). guard() raises ToolNotPermittedError on hard-reject. rewrite_path() (:235-258) resolves filesystem-MCP args against the workspace and raises WorkspaceEscapeError on ../ or out-of-tree absolutes. The schema (src/hal0/config/schema.py:2726-2838) enforces allow/gated/blocked disjointness at load time with the offending tool named (ToolPolicy.lists_are_disjoint, :2766-2793) — no silent "which check wins" behaviour. MCPServerConfig requires a URL for non-builtin servers (:2833-2838); auth is bearer-from-env only, so tokens never live in TOML (:2682-2723).

A user-installed server registry that already exists. src/hal0/mcp/installed.py (issue #305) persists one TOML per server at /etc/hal0/mcp-servers/<id>.toml with install / uninstall / patch_config / list_installed, fcntl advisory locking on read-modify-write (:261-278, issue #382), 0700/0600 perm hardening because the env block holds API keys (:111-133), a tight id charset, and bundled-id reservation (:139-161).

A REST API and a CLI. src/hal0/api/routes/mcp.py (945 lines) serves GET /servers|clients|catalog|resolve|stream|{id}/logs, POST /install, DELETE /{id}, PATCH /{id}/config. src/hal0/cli/mcp_commands.py (376 lines) gives hal0 mcp list|status|install|uninstall|restart and hal0 mcp catalog list|refresh. src/hal0/mcp/manifest.py resolves oci://, npm:/npx:, uvx:/uv:, git+https://, and generic manifest URLs behind an SSRF guard that blocks loopback/private/link-local/CGNAT and refuses redirects (:69-79), with a 256 KiB body cap (:65).

Hermes wiring. _phase_mcp_wire (src/hal0/agents/hermes_provision.py:2840) reads the per-agent allow-list at /etc/hal0/agents/hermes.toml, probes each server both raw and through the Hermes venv's own MCP client, and hard-fails when the client can't connect to a provably live server (issue #2021 — "server-side reachability alone is NOT wiring"). The rendered set is written via hermes config set key-value pairs (:2012-2021): mcp_servers.<n>.{type,url,headers.X-hal0-Agent,timeout} plus a bearer. The seed write merges, never clobbers, operator [mcp.servers.*] blocks (:248-250).

Memory. An explicit MemoryProvider ABC (src/hal0/memory/provider.py:1-14) as the anti-lock-in seam, Hindsight as the only engine (ARCHITECTURE.md:404), a closed two-valued namespace grammar (src/hal0/memory/namespace.py, ADR-0005) that fails closed when a read resolves to no addressable bank (:37-43, issue #1451), and a Hermes-side plugin (installer/agents/hermes/plugins/hal0-memory/) that treats recalled material as untrusted data, never instruction and never interpolates it into a system/tool position (provider.py:14-18).

Known memory gaps (per the brief; #1833/#1834 corroborated in CHANGELOG.md:1869-1872, 2320-2324): no document-ingest endpoint (#2016), a fresh uuid4 doc per turn (#2017), an uncapped extraction queue that fails to drain on CPU-only boxes (#1834), memory_add with no preflight (#1930), and no guide for pointing external coding agents at hal0 memory (#2153).

(Status on main c177aa5: gap 2 closed by #2253/ADR-0015 for http/sse servers — with the enforcement caveat in the §E.3 correction; gap 1 open for stdio. See the addendum at the top.)

The three real gaps in MCP:

  1. No supervisor. POST /api/mcp/{id}/{action} raises McpNotImplemented with code mcp.supervisor_unavailable "pending ADR-0015" (routes/mcp.py:934-945). Installed servers are hard-coded state="stopped" (:565). A stdio server therefore never runs.
  2. No exposure wiring. /etc/hal0/mcp-servers/*.toml and /etc/hal0/agents/*.toml [mcp.servers.*] are two unjoined registries. Installing a server never reaches Hermes's mcp_servers, never reaches the allow-list, never reaches brain/OpenWebUI/OpenCode. The install route sets env={k: "" for k in resolved.env_required} (:871) and stops.
  3. No install UI. manifest.py:3-6 references "the dashboard's InstallDrawer paste-box"; grep -rn InstallDrawer ui/ returns nothing. ui/src/dash/connections.jsx renders a read-only MCP panel plus config-builders that export hal0's servers outbound to Claude Desktop / Codex / Cursor (:688-704).

C. Better / worse / equivalent

Axis Verdict Why
MCP support hal0 vastly better ODS: zero. hal0: 2 hosted servers, registry, REST, CLI, resolver, SSRF guard, per-agent policy. Nothing to port.
Tool-call policy model hal0 better hal0's server-axis + tool-axis default-deny with load-time disjointness beats APE's verb-heuristic intent classifier. APE's classify_intent guesses from the tool name (main.py:636); hal0 matches exact (server, tool).
Policy enforcement reach ODS worse (nil), hal0 partial APE is wired to nothing. hal0's AgentMCPClient is only consulted by the two bundled servers' own gating plus Hermes's allow-list gate; a third-party MCP server would be un-mediated because it never runs.
Rate limiting / circuit breaking ODS better hal0 has no equivalent to APE's sliding-window per-intent caps or its deny-ratio circuit breaker.
Approval model Roughly equivalent, different strengths hal0 dedups + executes on approve + SSE inbox, but is in-memory only (approval_queue.py:11-14) so a restart drops the queue. APE persists approvals and mints args-fingerprinted one-shot grants that count against the window — strictly tighter, but it can't execute anything.
Audit hal0 better Structured hal0.mcp.audit events through structlog→journald, consumed by GET /api/mcp/stream and /{id}/logs. APE's audit.jsonl is append-only text with no reader other than GET /audit.
Memory architecture hal0 vastly better ODS memory is upstream Hermes files + a markdown-reset timer + an orphaned Qdrant. hal0 has a provider ABC, a closed namespace grammar with fail-closed reads, private/shared banks, curation/mental-models/directives, and prompt-injection-aware recall framing.
Memory reliability ODS arguably better within its scope memory-shepherd's contract is trivially correct and observable. hal0's extraction queue is documented as unreliable on CPU-only boxes (#1834). Simple and working beats sophisticated and flaky.
Config-generation discipline ODS better The Generated Config Writers table + patch-hermes-config.py's explicit operator-wins/ODS-converges split is a discipline hal0 lacks a named equivalent for.
RAG plumbing ODS worse than it looks Qdrant + TEI deployed, only TEI consumed, README claims otherwise. A documentation-vs-source divergence of exactly the kind hal0's CLAUDE.md warns about.

D. Port candidates

Ranked by value/risk. Nothing on the MCP axis ports from ODS — these are memory/governance/ process-discipline items.

D1. Block-scoped idempotent config patcher — HIGH value, LOW risk, ~150 LOC
  • From: /home/user/ods/ods/scripts/patch-hermes-config.py:17-59, 114-115, 215-218, 269-294
  • To: new src/hal0/agents/hermes_config_patch.py, called from hermes_provision._phase_config_write
  • Why: hal0 writes Hermes config through hermes config set key-value pairs (hermes_provision.py:2012-2021). That is fine for scalars but has no way to remove a stale mcp_servers.<name> block when a user uninstalls a server, and no way to express "operator wins" vs "hal0 converges" per key. The ODS patcher gives both, plus comment preservation.
  • Borrow specifically: the _has_key guard (operator wins) vs unconditional _set_key (hal0 converges) distinction, and the changed: bool / write-only-on-diff contract.
  • Risk: low — pure function over lines, trivially unit-testable.
D2. Generated Config Writers table — HIGH value, ZERO risk, docs only
  • From: /home/user/ods/ods/docs/INSTALLER-ARCHITECTURE.md:118-131
  • To: a new section in /home/user/hal0/ARCHITECTURE.md, or docs/reference/paths-and-files.mdx
  • Why: hal0 has more generated-config surfaces than ODS (/etc/hal0/agents/*.toml, /etc/hal0/mcp-servers/*.toml, hermes.env, api.env, runtime.json, the Hermes config.yaml overlay) written from at least three places (installer, hermes_provision, REST routes). One table naming every writer per surface is the cheapest high-value doc in this report — and it becomes load-bearing the moment the MCP registry starts writing into Hermes config (§E).
D3. Windowed rate caps + circuit breaker for gated tools — MEDIUM value, MEDIUM risk, ~250 LOC
  • From: /home/user/ods/ods/extensions/services/ape/main.py:482-536 (windowed limits), :574-612 (breaker), :255-262 (bounded sample lists)
  • To: src/hal0/mcp/approval_queue.py or a sibling src/hal0/mcp/limits.py
  • Why: hal0's approval queue dedups but does not cap. An agent in a retry loop against a distinct-args tool generates unbounded approval rows. Per-tool-class sliding windows that escalate to approval before hard-denying map cleanly onto hal0's existing three tiers.
  • Adapt, don't copy: key on hal0's (server, tool) pair, not APE's guessed intent class.
D4. Persisted, args-fingerprinted one-shot approval grants — MEDIUM value, LOW risk, ~80 LOC
### ODS: extensions/services/ape/main.py:973-989 — the grant is bound to the exact call
gkey = _grant_key(rec["session"], rec["tool_name"], rec["intent"], rec["args_hash"])
_state.setdefault("grants", {})[gkey] = {..., "approver": req.approver, ...}
  • To: src/hal0/mcp/approval_queue.py
  • Why: hal0's queue is explicitly in-memory (approval_queue.py:11-14) — an hal0-api restart drops pending approvals silently. Persisting the pending set (and binding an approval to an args hash so an approved model_pull qwen3:0.6b cannot authorise model_pull something-else) is the hardening; the module's own docstring anticipates it: "A future ADR can promote this to a persisted table." It is not table-only, though (correction 2026-10-04): each entry executes through an in-memory _executor closure bound at enqueue time (approval_queue.py:127,216, built in admin.py:1887-1900), and approve() fails with no executor bound to approval when it is missing (approval_queue.py:262-276). A persisted approval therefore needs a trusted dispatcher that rebuilds execution from the stored (server, tool, args, args_hash) after re-running ToolPolicy.classify and session validation (admin.py:1769-1795,1860) — the closure itself cannot be serialised.
D5. Baseline/scratch separator for self-edited agent files — LOW value, note only

memory-shepherd's --- contract (README.md:133-148, memory-shepherd.sh:151-180) only applies if hal0 ever lets an agent self-edit a persona/SOUL file. It currently does not (personas.py is hal0-authored). Record, don't port.


E. Design: user-added MCP servers for hal0

The owner's ask — "a custom way to add MCP servers rather than just be stuck with the originals that we ship" — is 80% built. The work is finishing three things: a supervisor, an exposure join, and a test verb. Do not build a new registry file format; /etc/hal0/mcp-servers/<id>.toml already exists, is atomic-written, perm-hardened, and lock-protected.

E.1 Schema: extend InstalledServer, don't replace it

Current fields (src/hal0/mcp/installed.py:57-91): id, name, description, spec, transport, tools, resources, prompts, env, enabled, installed_at, source_url, author, verified.

Add five, all defaulted so existing records keep validating:

### /etc/hal0/mcp-servers/github.toml
id           = "github"
name         = "GitHub MCP"
spec         = "npm:@modelcontextprotocol/server-github"
transport    = "stdio"                 # stdio | streamable-http | sse   (NEW: sse)
command      = "npx"                   # NEW — stdio only
args         = ["-y", "@modelcontextprotocol/server-github@<pinned>"]   # NEW — exact version, see below
url          = ""                      # http/sse only (already implied by transport)
enabled      = true

[env]                                  # existing — literal values, 0600
GITHUB_HOST = "github.com"

[secrets]                              # NEW — indirection, never a literal
GITHUB_TOKEN = "GITHUB_MCP_TOKEN"      # name of a key in /etc/hal0/api.env

[tools]                                # NEW — mirrors ToolPolicy exactly
allow   = ["search_repositories", "get_file_contents"]
gated   = ["create_pull_request"]
blocked = ["delete_repository"]

[exposure]                             # NEW — which consumers see this server
hermes    = true
brain     = false
openwebui = false
opencode  = false

Correction (2026-10-04) — pin the executable. npx -y <pkg> resolves whatever upstream publishes at start time, so a bridge restart could run a newer release with the configured GitHub token while the TOML never changed. hal0 mcp install must resolve the exact version once (npm view <pkg> version, a uv lock for uvx:), or better fetch the artifact into a hal0-owned cache with its integrity hash, persist that in the record, and render the pinned command into the supervised unit — the same digest-pinning posture hal0 already applies to runner images.

Design notes, each tied to an existing hal0 decision:

  • [secrets] is a reference, not a value. hal0 already has a secrets store at /etc/hal0/api.env behind /api/settings/secrets with an ^[A-Z][A-Z0-9_]{0,63}$ name gate (src/hal0/api/routes/secrets.py:129-143), and the agent-config schema already refuses to hold tokens in TOML (AgentAuthConfig, config/schema.py:2682-2689). Keeping literal env for non-secret settings and secrets for indirection preserves both rules. New vs both repos.
  • [tools] reuses ToolPolicy verbatim (config/schema.py:2726), inheriting the disjointness validator and the empty-by-default posture. A freshly installed server therefore has zero callable tools until the operator promotes some — which is exactly the documented intent of ToolPolicy's docstring (:2737-2740). This is the "default-deny + approvals for dangerous tools" requirement, satisfied by reuse, not new code.
  • [exposure] is new. It is the join that currently doesn't exist. hermes and brain are the two hal0-side consumers that can be honoured immediately; openwebui / opencode should ship disabled and unimplemented rather than silently ignored — return 501 mcp.exposure_unsupported if set, following the precedent of McpNotImplemented (routes/mcp.py:57-69), which is a pattern worth keeping because the dashboard can key on the code.
E.2 Supervisor (the ADR-0015 gap)

stdio servers need a process. Follow hal0's own bundled-agent precedent — sandboxed sibling systemd units (ARCHITECTURE.md:395-397) — rather than inventing a supervisor:

  • One transient unit per enabled stdio server: hal0-mcp@<id>.service, User=hal0, ProtectSystem=strict, LoadCredential= for each [secrets] entry (which is how AgentAuthConfig's bearer-from-env tokens already arrive, mcp_client.py:199-231), ReadWritePaths= limited to the server's own scratch dir.
  • POST /api/mcp/{id}/{start,stop,restart} replaces the 501 with systemctl --user-equivalent calls; list_servers replaces the hard-coded state="stopped" (routes/mcp.py:565) with the real unit state.
  • http/sse servers need no supervision — they are already reachable; only the exposure join and the tool policy apply.

Correction (2026-10-04). The first bullet is wrong as written. A stdio MCP server speaks JSON-RPC over its own stdin/stdout, so whoever starts the process must be its client: a free-standing hal0-mcp@<id>.service would leave an npx/uvx server with nobody on the other end of the pipe, and the renderer in §E.3 (which emits only {type,url,timeout}) would then point Hermes at an empty url. Two shapes actually work:

  1. Hermes spawns it. Render command/args into Hermes's own mcp_servers.<id> block and let Hermes's client fork the process (only if the pinned Hermes build accepts a stdio entry; hal0's provisioner never writes one today). Cost: the server then runs inside hal0-agent@hermes's sandbox, [secrets] have to reach that unit's environment, and the process sits outside hal0's policy path (see the §E.3 correction).
  2. hal0 runs a bridge. hal0-mcp@<id>.service executes a small hal0-owned adapter that forks command args… as its child, speaks MCP to it over the pipe, and serves the same session as streamable-http on a loopback port or unix socket. LoadCredential= still delivers [secrets] — as files under $CREDENTIALS_DIRECTORY, so the adapter must read each one and set the mapped variable in the child's environment; a stdio server (like hal0's own client, agents/mcp_client.py:222) looks up a plain environment name, not a credential file (correction 2026-10-04). Nor can the unit point LoadCredential= at api.env: that copies the whole file into one credential (LoadCredential=ID:PATH), and the box's secrets live in that one aggregate KEY=value file (service_identity.py:47-57), so a per-[secrets] mapping needs a root broker — an ExecStartPre=+ helper that extracts only the selected keys into per-unit systemd-creds encrypted files consumed through LoadCredentialEncrypted= — or a per-secret store (second correction 2026-10-04) — ProtectSystem=strict still applies, and the exposure join renders the bridge's URL — which makes a stdio server indistinguishable from an http one to everything downstream, including the policy proxy the §E.3 correction asks for.

Recommend (2). ADR-0015's "Deferred: stdio supervisor" section still sketches the free-standing unit; whoever picks that work up should start from the bridge shape.

Write ADR-0015 first. routes/mcp.py:943 already promises it, hal0's CLAUDE.md requires an ADR before changing behaviour it covers, and docs/adr/ currently holds only 0001, 0002, 0003, 0005, 0006 while source cites 0004, 0008, 0012, 0013, 0015, 0020 and 0023 sixty times. (See §G.)

E.3 Exposure wiring — the Hermes join, done the ODS way

This is where the ODS port earns its place. On any registry mutation (install / uninstall / patch / enable / disable), regenerate the Hermes mcp_servers block idempotently and surgically, exactly as patch-hermes-config.py does:

  1. Build the desired set = _default_mcp_servers() (hermes_provision.py:1462-1497) plus every installed record with enabled = true and exposure.hermes = true.
  2. For each, write mcp_servers.<id>.{type,url,timeout} + headers.X-hal0-Agent, reusing the hermes config set mechanism behind the existing loop at hermes_provision.py:2012-2021 — but not its bearer. Correction (2026-10-04): that loop attaches service_key(prefer="admin") (hermes_provision.py:2011,2020-2021) because its only targets are hal0's own ADMIN-classed /mcp mounts. Reusing it verbatim for a user-installed URL would hand hal0's platform-admin credential to a third-party server. User-installed entries carry only X-hal0-Agent, the record's [env] literals, and its [secrets] resolved from api.env (an AUTHORIZATION = "<api.env key>" entry is how a server that wants a bearer gets its own); the hal0 service identity never leaves the box. This is what main's hal0.mcp.probe.build_headers (#2253) does. Correction (2026-10-04) — no TLS gate exists. Neither the record schema (installed.py:110, url: str = Field(default="")) nor the join (hermes_join.py:136-140,237, non-empty check only) rejects a plaintext http:// URL on a non-loopback host, so a [secrets]-backed AUTHORIZATION header would cross the LAN in the clear. The install verb must require https:// whenever the host is not loopback (unix-socket and 127.0.0.1 bridges exempt) and the record carries [secrets], with at most an explicit, logged allow_insecure_http override — filed as hal0 issue #2304, since main has the same gap.
  3. Remove mcp_servers.<id> blocks that hal0 previously wrote and no longer wants — the capability hermes config set cannot express, and the reason D1 is a prerequisite rather than a nice-to-have.
  4. Mirror the record's [tools] into /etc/hal0/agents/hermes.toml [mcp.servers.<id>] so AgentMCPClient.classify() (mcp_client.py:149) can report the policy (hal0 mcp test, the dashboard verdicts). Preserve the existing merge-never-clobber contract (hermes_provision.py:248-250). Correction (2026-10-04): the mirror does not enforce anything on the call path, and the original wording implied it did. AgentMCPClient has no production instantiation (grep -rn "AgentMCPClient(" src/ is empty at 108b366 and at main c177aa5); Hermes's own MCP client connects directly to mcp_servers.<id>.url. The two builtins are safe only because their gate is server-side: hal0's /mcp mount classifies every call (mcp/admin.py:1769-1795 ToolPolicy.classify → run|gated|denied|refused, is_gated + ApprovalQueue; api/mcp_mount.py:290-310). A third-party server has no such server in front of it, so blocked is not blocked and gated never reaches the approvals inbox — and ADR-0013's "zero callable tools until promoted" default is void for exactly the servers it was written for. The design fix is to put hal0 on the wire: expose user-installed servers to Hermes only through a hal0-hosted proxy mount (e.g. /mcp/ext/<id>, ADMIN-classed like /mcp) that forwards initialize/tools/list/tools/call upstream, hides tools outside allow ∪ gated, routes gated calls through the same ApprovalQueue, rejects blocked, and injects the record's [secrets] upstream while receiving the hal0 service bearer from Hermes like the builtins do. That also collapses step 2's bearer question into one rule (hal0 identity on the Hermes→hal0 hop, the record's own token on the hal0→upstream hop) and gives a stdio bridge (§E.2) the same front door. Until the proxy exists, exposure.hermes = true honestly means "every advertised tool is callable" and should either say so in the UI/CLI or be refused for records whose [tools] has a non-empty gated/blocked list. Main's #2253 shipped the mirror without the proxy; tracked as hal0 issue #2303.
  5. Re-run the mcp_wire probe for the new server only, so a bad URL surfaces at install time rather than at first agent turn.

Ownership marker. Write hal0-managed blocks between sentinel comments (# >>> hal0 mcp registry — do not edit <<< … # >>> end <<<) so step 3 can delete precisely what hal0 owns and never touch an operator's hand-added block. This is a small addition to the ODS block-scoping approach, which locates blocks by key name only. (Main's #2253 keeps the ownership list in a side-car manifest, /var/lib/hal0/mcp/hermes-managed.json, instead of sentinel comments, because comments do not survive the yaml.safe_dump hal0 already applies to this file — ADR-0015 "Rejected". Same guarantee, different carrier.)

E.4 CLI

hal0 mcp list|status|install|uninstall|restart already exist (cli/mcp_commands.py:72,119,204,239,271). Additions:

  • hal0 mcp test <id> — new. Runs the existing probe path (hermes_provision._probe_mcp_server, already reused by cli/doctor_all.py:806-828) against one server and prints the advertised tool list with each tool's current allow/gated/blocked/ unknown_tool verdict from classify_many (mcp_client.py:264-273). This is the single highest- value new verb: it turns "I added a server" into "here is exactly what it can do and what I have permitted".
  • hal0 mcp allow|gate|block <id> <tool> — new. Edits [tools] through patch_config, which already holds the lock (installed.py:281-314).
  • hal0 mcp expose <id> --hermes/--no-hermes — new. Flips [exposure] and triggers §E.3.
  • Keep add as an alias of install and remove as an alias of uninstall if the owner prefers that vocabulary; do not rename the existing verbs.
E.5 API and dashboard

REST needs three additions to src/hal0/api/routes/mcp.py: POST /{id}/test (probe + classify), PATCH /{id}/tools, PATCH /{id}/exposure. The existing PATCH /{id}/config already covers env/enabled.

Dashboard: build the missing InstallDrawer that manifest.py:3-6 already assumes — paste a spec, GET /api/mcp/resolve for the preview (SSRF-guarded, manifest.py:69-79), edit env/secrets, install. Then extend the existing McpServerRow (ui/src/dash/connections.jsx:727) — which already renders a per-tool drawer — with a three-state control per tool writing to PATCH /{id}/tools, and an exposure toggle row. Reuse useMcpServers (ui/src/api/hooks/useMcp.ts:126).

E.6 Provenance
Element Source
Per-server TOML registry, atomic write, 0600, lock hal0, existing — src/hal0/mcp/installed.py
Three-tier tool policy + disjointness hal0, existing — src/hal0/config/schema.py:2726
Approval inbox for gated tools hal0, existing — src/hal0/mcp/approval_queue.py
Manifest resolve + SSRF guard hal0, existing — src/hal0/mcp/manifest.py
Idempotent block-scoped config regeneration, operator-wins vs converges ODS, ported — scripts/patch-hermes-config.py:114-115, 215-218
Three-way fallback on unparseable generated config ODS, ported — installers/phases/07-devtools.sh:225-274
Writers-per-surface documentation table ODS, ported — docs/INSTALLER-ARCHITECTURE.md:118-131
Windowed caps / circuit breaker (optional, D3) ODS, adapted — extensions/services/ape/main.py:482-612
[secrets] indirection block New
[exposure] targets block New
hal0-owned config region with an ownership record (sentinels proposed; main shipped a side-car manifest) New (extends ODS block-scoping)
hal0 mcp test New
stdio bridge unit — hal0-mcp@<id> running a hal0-owned stdio↔streamable-http adapter (§E.2 correction) New
Policy proxy mount fronting user-installed servers (§E.3 correction) New

F. Do not copy

  1. APE as a deployed service. A policy engine nothing calls is worse than no policy engine: it consumes a port, a container, and a maintained policy file (config/ape/policy.yaml was updated for Hermes paths that never route through it) while providing zero enforcement. hal0's policy belongs in the call path (AgentMCPClient), where it already is. Port APE's algorithms (D3, D4), never its topology.
  2. APE_STRICT_MODE=false as a default (extensions/services/ape/compose.yaml:14). Advisory-by- default governance trains operators to ignore it. hal0's default-deny posture is correct; preserve it.
  3. Intent classification by tool-name verb tokens (ape/main.py:635-654). Guessing that a tool named create_pull_request is a WriteFile (it contains "create") is a category error. hal0's exact (server, tool) matching is strictly better.
  4. Deploying a vector DB with no consumer, and documenting it as if it had one. Qdrant in ODS is a running, API-keyed, port-bound service with zero writers, while extensions/services/embeddings/README.md:7 states "These vectors are stored in Qdrant". hal0's own CLAUDE.md names this failure mode; ODS is a live example. Do not add infrastructure ahead of a consumer.
  5. memory-shepherd's periodic destructive reset as a model for hal0 memory. Hindsight-backed memory is the durable record; a timer that truncates it on a schedule would be a data-loss feature. The separator contract is portable (D5); the reset is not.
  6. Wiring memory into the AMD tuning phase (installers/phases/10-amd-tuning.sh:63-95 installs the memory-shepherd timers) — unrelated concerns in one phase, and presumably no timers at all on non-AMD hardware.

G. Owner decisions

  1. Confirm the ODS-MCP finding. ODS contributes nothing to hal0's MCP story. If the owner expected otherwise, the expectation may come from a newer ODS branch than the one on disk (ARCHITECTURE.md header: Version 2.6.0). Worth a one-line confirmation before any further ODS-MCP archaeology is commissioned.
  2. Write ADR-0015 (MCP process supervisor) — blocking. routes/mcp.py:943 promises it by name. Decide: systemd-unit-per-server (recommended, matches ARCHITECTURE.md:395-397) vs an in-process subprocess pool vs http/sse-only (drop stdio support entirely, which halves the work and covers most hosted MCP servers). Resolved on main (#2253): ADR-0015 accepted for http/sse, stdio deferred — read the §E.2 correction before implementing the deferred part.
  3. Resolve the ADR bookkeeping gap. Source cites ADR-0004, 0008, 0012, 0013, 0015, 0020, and 0023 (60 times); docs/adr/ holds only 0001, 0002, 0003, 0005, 0006. Separately, ARCHITECTURE.md:392-393 states "hal0 keeps no ADR tree" while CLAUDE.md says "docs/adr/ holds the accepted decision records". One of those is stale. This matters here because the MCP supervisor decision is supposed to land as an ADR. Resolved on main (#2244): docs/adr/ now holds 0004, 0012, 0013, 0015, 0020 and 0023 alongside the original five.
  4. Scope of [exposure]. Ship hermes + brain only in v1, with openwebui/opencode returning 501 mcp.exposure_unsupported? Or defer the whole exposure block and wire Hermes unconditionally (simpler, but re-creates the "one hard-coded consumer" shape the owner is trying to escape)?
  5. [secrets] vs [env]. Confirm that MCP server credentials should live in /etc/hal0/api.env alongside hal0's own secrets, rather than in a separate /etc/hal0/mcp-secrets.env. The former reuses /api/settings/secrets and its name validation for free; the latter isolates third-party credentials from hal0's own. Recommend the former.
  6. Persist the approval queue now or later? D4 is ~80 LOC and closes a real hole (restart drops pending approvals). It becomes more urgent once third-party servers can enqueue gated calls, because the blast radius of a silently-dropped approval widens.
  7. Adopt the Generated Config Writers table (D2)? Zero-risk docs, but it only pays off if it is maintained. It should probably become a CONTRIBUTING.md checklist item ("did you update the writers table?") rather than a standalone doc that rots.
  8. Rate-limit gated MCP tools (D3)? Genuine hardening, ~250 LOC, and the only axis where ODS is ahead. Reasonable to defer until a third-party server actually runs.
  9. Memory issues are out of scope for MCP-adds but interact with it. #1930 (memory_add no preflight) and #1834 (uncapped extraction queue) both get worse when more MCP clients can write. Consider sequencing at least #1930 before third-party servers gain memory_* access.
07-dashboard-ux.md Dashboard and self-explanation

Dashboard & self-explanation: ODS vs. hal0

Comparative read of how ODS (Osmantic Deployment System) makes itself legible to a non-expert, and where hal0 — a homelab appliance for a more technical operator — matches, exceeds, or falls short of that bar. All citations are file:line against the two repos on disk (/home/user/ods/ods/…, /home/user/hal0/…).

A. ODS UX inventory

Nav / IA. The sidebar (extensions/services/dashboard/src/components/Sidebar.jsx) and route registry (extensions/services/dashboard/src/plugins/core.js:25-121) expose only seven top-level items — Dashboard, GPU Monitor (hidden unless gpu_count>1, core.js:44), Extensions, Integrations, Models, Remote GPU, Settings — with Usage and "Setup / Owner" deliberately kept off the top nav (core.js:87-90: "a factory/distributor/service-provider flow, not a day-to-day dashboard surface"). Agents (Hermes), Privacy, Workflows, Features and Updates are not separate pages — they're sections/cards embedded in Dashboard and Settings, keeping the nav shallow while the underlying surface area is large.

Page Question it answers Actions offered Status vocabulary Empty/error handling
Dashboard (pages/Dashboard.jsx) "Is my system healthy, and what can I turn on?" Click a Feature card to launch it; jump to GPU Monitor healthy/degraded/not_deployed rolled into computeHealth() (Dashboard.jsx:30-40, "3/4 core services online."); feature status enabled/available/services_needed/insufficient_vram normalized to ready/disabled for display (:132-144) Loading: "Linking modules... reading telemetry..." (:679); feature fetch failure silently degrades to status-only view (comment :623)
FirstBoot wizard (pages/FirstBoot.jsx, 717 lines) "What is this box and who am I?" 4 taps: label the setup → name the first user → pick a stack (Chat only / Chat+Agents / Full Stack) → Finish, generating a QR owner card n/a (linear wizard) Owner-card-unavailable and finish-error states both explain why and what to do next (:547-562)
Extensions (pages/Extensions.jsx, 1328 lines) "What can I install, and what state is it in?" Install/Enable/Disable/Remove/Purge, per-extension console, live install progress 10-value enum each with a plain-English STATUS_DESCRIPTIONS sentence (:60-84, e.g. unhealthy → "Container is running but health check is failing — check logs") friendlyError() maps raw backend strings to sentences (:38-56); a "polling lost" banner with auto-recovery tracking (:111-160)
Integrations (pages/ServiceMap.jsx) "How do these services actually talk to each other?" None (read-only topology) 4-color legend: healthy/degraded/down/not deployed (:332-337) "Topology data unavailable: {error}" (:314)
Models (pages/Models.jsx, 1970 lines) + HuggingFaceModelBrowser "What models do I have, what can I add?" Search HF, pull, assign, insights panel download/insufficient-VRAM style badges —
Remote GPU (pages/RemoteProvider.jsx) "Is my cloud-burst route working?" "Test route" probe, refresh StatusPill Inline red banner with the raw error
Usage (pages/Usage.jsx) "What is this costing me, and can I trust the number?" Enable/Restart Token Spy inline readiness missing/unconfigured tones; a "Tracking Source Guide" panel labelled "Honest by design" explains each cost-source tag (actual_billed, priced_from_tokens, local_zero_cost, untracked) (:761-799) UsageReadinessPanel shows message + detail + a real "Enable Usage Tracking" button with busy state (:511-562)
Setup / Owner (pages/Invites.jsx) "Who has a key to this box?" Create/revoke owner cards and guest invites (QR) — Icon+heading+explanation+CTA empty states (:304-339, "No owner cards yet")
GPU Monitor (pages/GPUMonitor.jsx) "Is my GPU the bottleneck?" none per-GPU aggregate bars "GPU data unavailable" / {error} (:38-49)
Settings (pages/Settings.jsx + components/settings/EnvEditor.jsx) "What can I change, and what happens if I do?" Save / Apply changes (recreates only the affected containers) field-level required/enum/type errors in plain English (settings.py:244-289) BehaviorCards literally state Save/Restart/Apply behavior (EnvEditor.jsx:268-279); Apply is disabled with an explanation until it has a plan (:271-273)

Feature/Services/Extensions split. "Features" (Chat, Voice, Document Q&A, Workflows…) are outcome-level, declared as data inside each service's manifest.yaml (extensions/services/llama-server/manifest.yaml:24-39) and computed server-side into enabled/available/services_needed/insufficient_vram (dashboard-api/routers/features.py:19-110). "Services" are the technical containers; "Extensions" is the installer/marketplace layer over them. A NON_USER_FACING_LINK_SERVICES set (Dashboard.jsx:73-84) hides infra services (litellm, qdrant, embeddings…) from click-through — deliberate complexity hiding. Hardware tiers get human names (installers/lib/tier-map.sh: "Entry Level", "Prosumer", "Pro", "Enterprise", "Strix Halo 90+", "Cloud (API)"), and GET /api/features returns plain-English GPU-tier advice (features.py:162-177, e.g. "Entry-level GPU — focus on chat first").

Self-explanatory API shapes. features.py:130-144 turns "available"/ "services_needed" into first-person suggestions ("Your hardware can run Document Q&A. Enable it?" / "Voice needs whisper, tts to be running."). GET /api/features/{id}/enable (:188-250) returns numbered steps + deep links per feature, rendered by components/FeatureDiscovery.jsx:246-300 as a modal — click "Enable Document Q&A" and you get "1. Ensure Qdrant vector database is running… 2. Open Open WebUI…" with a working "Open Chat" button. routers/privacy.py:44-51, 68-90 puts a human message on every response ("Privacy Shield is active" / "…is not running. Check: docker compose ps privacy-shield"). routers/workflows.py:166-168 turns a 400 into "Missing dependencies: qdrant. Enable these services first." components/TroubleshootingAssistant.jsx pairs Symptoms → Likely cause → Solutions with copy-to-clipboard commands, auto-surfacing the ones relevant to whatever is currently unhealthy.

First-run / guidance surfaces. installers/lib/ui.sh:97-103 defines a consistent "AI narrator" voice (ai/ai_ok/ai_warn/ai_bad/signal); :154-168 carry brand-voice lore lines; :507-597 show_install_menu ("Choose how deep you want to go. I can install everything, or keep it minimal.") auto-disables ComfyUI on low-VRAM tiers with an explicit follow-up ("You can enable it later with: ods enable comfyui"); :600-632 show_success_card closes with "THE GATEWAY IS OPEN" / "Your data never leaves this machine." Phase 13 (installers/phases/13-summary.sh) writes a desktop shortcut + GNOME sidebar pin (:229-256), retries preflight validation up to 3× with a spoken wait message (:170-183), and writes a versioned, atomically-written summary JSON (:435-501). docs/SETUP-CARD.md documents a script that renders a physical 4×6, 300-DPI card with two QR codes (Wi-Fi + setup URL) for zero-typing onboarding of a boxed unit. docs/POST-INSTALL-CHECKLIST.md is a 6-step "does it actually work" script. docs/HOW-ODS-SERVER-WORKS.md explains the whole microservice mesh as "an office building" with each service as a named team member — no jargon. The outer README.md:120 links that guide plus "listen to the audio version" (a Spotify episode), then :124-139 gives an "At A Glance" Q&A table and an "If you know Ollama / Open WebUI / AnythingLLM / n8n… ODS adds…" table that anchors newcomers to tools they already know. templates/*.yaml are outcome-named install bundles ("Creative Studio", "Developer Homelab", "Voice Assistant") consumed by components/TemplatePicker.jsx. ods status (ods-cli:1273-1400) prints sectioned ━━━ ODS Status ━━━ / Health Checks / GPU Status blocks with ✓/⚠ per service, plus --json.

B. hal0 UX inventory

Nav / IA. ui/src/dash/chrome.jsx:480-524 (useNavItems) lists Overview, Slots (children: Endpoints, Runner Images, Stacks), Models (children: Profiles), Benchmarks, Agents (children: Memory, MCP), Services, Logs, Settings. Compared to ODS, the vocabulary is implementation-first: "Slots", "Runner Images", "MCP", "Profiles", "Stacks" are hal0's own internal nouns exposed directly, not translated into outcomes ("Chat", "Voice"). This matches the product's own framing — see README.md's target hardware and register — but it does mean the nav teaches hal0's architecture before it teaches what you can do.

Page Question it answers Actions offered Status vocabulary Empty/error handling
Overview (dash/dashboard-redesign.jsx, active; dashboard.jsx is the fallback) "Is the box up and who's using memory?" Quick actions: restart agent / pull model / evict idle / open webui (:166-215) slot phases missing/pulling/starting/serving/ready/idle/stopped/crashed (slot-status.js:8-16) RDAttentionCard: "nothing needs you" when clear (:920); zero-slot state: "No models configured yet… Run hal0 setup in your terminal" (dashboard.jsx:253-263)
Slots (slots.jsx, 1118 lines) "What's loaded, on what device, doing what right now?" Load/Stop/Restart/Swap, per-slot capability toggles color-coded dot + tooltip (slot-status.js:158-226, e.g. "Container stopped (auto-reloads on next request)") pull-progress bar with live label (:106-124)
Models (models.jsx) "What's on disk / pullable from HF?" Pull, filter, assign to slot pull job states —
Agents overview (agents/agents-overview.jsx) "What agents can drive this box?" Restart Hermes, view persona live/health per agent card, rendered as collectible trading cards (rarity, abilities, "pow") (:29-50) roadmap agents shown greyed "coming soon"
Brain chat (board/agent-chat.jsx) "Talk to the steward model directly" send/stop placeholder="Ask hal0-brain…" vs "chat unavailable" (:354) —
Board (board/board-view.jsx) + Approvals (chrome.jsx:1101-1214) "What is an agent waiting on me for?" Approve/Deny gated tool calls "no pending approvals" empty state; reassuring copy: "Calls block until you approve or deny — agents pause cleanly." (:1114) badge is poll-driven, no arrival toast (issue #1994)
Memory (memory-overview-v2.jsx, memory-map.jsx, telemetry-header.jsx) "How much unified memory is free, by whom?" delete/curate facts, approve deletes ruler + bar segments header vs. bar computed "free" from two different bases on some boxes (issues #1928/#1929)
Journal / Logs (activity-log.jsx, routes/journal.py, routes/logs.py) "What just happened?" filter/expand — —
Stacks (stacks.jsx, 876 lines) "What's my named slot+profile+model bundle?" New/Clone/Snapshot/Import/Export/Delete breadcrumb descriptor only: "Stacks · runtime · slot + profile + model bundles" (:793-798) "No stacks yet — create one, snapshot the live config, or import a .hal0stack.json." (:833)
Settings → Capabilities/Model Defaults/Security/Doctor/… (settings/pages/**) "What's configured, and does changing it need a restart?" Save; one-click Restart hal0-api with confirm + live health-poll + toast (settings/shared/RestartApiPanel.jsx) per-field ApplyBadge: green "live" / amber "⟳" / red "⚠ manual restart" (settings/shared/ApplyBadge.jsx:9-16) PanelFooter: "capability probe failed — Save disabled" + Retry (capabilities/shared.jsx:33-46)

Guidance/diagnosis surfaces. hal0's nearest equivalent to ODS's Troubleshooting Assistant + Feature-enable modal is the typed Diagnosis model (src/hal0/diagnostics.py:41-80: id/severity/confidence/summary/detail/evidence/ next_steps/fixable), fed by GET /api/doctor (src/hal0/api/routes/doctor.py) into a single generic DiagnosisPanel.jsx that already renders next_steps as chips ($ command / ↗ doc / • manual, :31-44). The scaffolding is more rigorous than ODS's ad hoc per-router strings — one classifier (src/hal0/health_report.py) is shared verbatim by hal0 doctor verify --json and the API, so the two can't drift. The install-time and per-setting remediation UX (RestartApiPanel, ApplyBadge, PanelFooter) is genuinely strong and, in places, ahead of ODS (a real restart-and-confirm loop with a success/failure toast, not just a banner).

First-run. There is no wizard: README.md:39-45 states this as policy — "⚡ The installer is the whole setup. There is no separate first-run wizard afterwards." README.md:160-183 ("First ten minutes") is entirely CLI/curl. docs/ is a Starlight site with a conventional getting-started/concepts/guides/reference split (docs/getting-started/*.mdx, docs/concepts/*.mdx, docs/guides/*.mdx, docs/reference/*.mdx) — better taxonomy than ODS's flat 60-file docs/, but reference-heavy (cli.mdx, config-schema.mdx, env-vars.mdx, mcp-tools.mdx) rather than beginner-first. docs/getting-started/index.mdx:109-114 does disclose the no-auth-by-default LAN posture up front, in a Starlight <Aside type="caution">.

C. Where hal0 falls short (evidenced) — and where hal0 is ahead

1. The remediation affordance is designed but not wired end-to-end. Diagnosis.next_steps and NextStep(kind="command"|"manual"|"doc") (diagnostics.py:41-56) are real, and DiagnosisPanel.jsx:31-44 already renders them as chips. But the classifier actually wired to /api/doctor (health_report.py:246-283, to_diagnosis()) hard-codes next_steps=[] for every one of its 7 checks — even though the richer CLI path (src/hal0/cli/doctor_commands.py:887,899,911,1208,1251,1272,1304,1331,1645,1934, 1953) already knows real remediation commands for adjacent checks (e.g. hal0 model scan, podman pull <image>). And even where a chip is populated, its target is shown only as a title= tooltip (DiagnosisPanel.jsx:38) — not click-to-run, not click-to-copy. Contrast ODS, where GET /api/features/{id}/enable (features.py:188-250) returns real numbered steps with deep links, and FeatureDiscovery.jsx:246-300 renders them as an actual modal with clickable "Open Chat"/"Open n8n" buttons. This is precisely the shape of issue #1845 (a printed remediation, hal0 slot migrate-flags --apply, that fails because the panel omitted --stop-services even though the error message itself named the fix) — a self-explanation gap in the exact same family, not a one-off typo.

2. Long silent waits with no progress line (issue #1870). slot load, unload, restart, swap, and update --restart-slots can legitimately block 10s to 2.7 hours with nothing printed — no spinner, no elapsed counter, no state line — because the fix for a prior regression (#1832) raised the timeout ceiling without adding the promised progress polling. ODS's closest analogue, extension installs, streams a live progress bar and label (Extensions.jsx:106-124 pattern; poll loop :111-160) and degrades gracefully with a "polling lost" banner rather than silence.

3. A remediation that cannot work as printed (issue #1845). The hal0 update "Convergence incomplete" panel tells the operator to "Stop hal0, then run: hal0 slot migrate-flags --apply" — but that literal command fails, because hal0-slot@* units are separate from hal0.target/hal0-api, and the actually-sufficient invocation (--apply --stop-services) is only visible in the error the failed command itself prints. ODS's install-menu logic shows the alternative discipline: when it auto-disables a feature, it immediately states the real command to reverse that (ui.sh:547-549, "You can enable it later with: ods enable comfyui") in the same breath as the decision, not two failed attempts later.

4. Notification infrastructure exists but isn't wired to the moment that matters (issue #1994). globals-install.ts:26,39-40 installs a working toast system and hooks a "queue" from a zustand store; notifications.jsx already unifies bell + "Needs attention" card. Yet an agent-requested memory delete landing in the approval queue is silent — "the badge is poll-driven and silent," per the issue, filed by the team's own reviewer during the memory-v2 build. ODS's parallel case (a background SDXL/model download finishing) is explicitly narrated at the next natural checkpoint (13-summary.sh:170-183: "SDXL Lightning model download still in progress" / "…completed").

5. Same-screen contradicting numbers (issues #1928, #1929). The memory ruler's header (telemetry-header.jsx:364, total - modelUsedGb) and its bar segment (:389-390, reducing to total - gttUsedGb) compute "free" from two different bases, so on a box where host GTT usage diverges from the per-slot model sum, the same screen shows two different "free" numbers (measured ~16 GB apart in the issue). A second, related bug: memory-map.jsx:141-142 reads dead fallback keys (stats.gtt_total_mb, rawHw.unified_memory_mb) that don't exist on the normalized shapes, so the pool can silently become "system RAM" while still labeled "GPU pool (GTT)." ODS's dashboard computes a comparable figure (sidebar memory bar, Sidebar.jsx:76-89) from a single source per render and explicitly branches on memoryType === 'unified' rather than blending two derivations.

6. Help text explains internals, not consequences. hal0's FieldInfoIcon tooltips are widespread (20 files) and the popover mechanics are solid, but the copy is written for someone who already knows the codebase: "/api/health/system · per-dependency status", "/api/stats/power · sysfs hwmon, sensor-dependent", "hal0.toml [memory] · requires restart to switch" (settings/pages/general/OverviewPage.jsx:59-231, settings/pages/data/ MemoryPage.jsx:69,161). None say what changing the setting does for you or what happens if you don't. ODS's equivalent — STATUS_DESCRIPTIONS (Extensions.jsx:73-84) and the saveHint/restartHint strings (main.py:921-922) — run the other way: consequence first, mechanism only if needed ("Saving writes the .env file directly, keeps existing secret values when left blank… stores a timestamped backup"). This is a register choice consistent with each product's stated audience, but it means hal0 currently explains itself to an engineer, not a first-time user — worth naming explicitly if hal0 wants a broader audience.

7. An unsurfaced security tradeoff (issue #1822). HAL0_BIND_HOST (LAN) and require_auth (off by default) are decided independently and never compared out loud at the moment that matters — install completion or hal0 doctor all. The posture is intentional and is documented (docs/getting-started/index.mdx: 109-114, a caution Aside), so this is "disclosed but not surfaced at the decision point," a narrower gap than "hidden." ODS states comparable consequences inline at the moment of the choice (e.g., ui.sh:547-549 above), which is the standard #1822 is asking hal0 to meet.

8. tool_model has no dashboard path (issue #2108). AgentsBrainPage.jsx persists enabled/read_only/model/max_rounds/ completion_timeout_s but never tool_model, even though it's the one [brain_chat] key with real routing consequences — it's hal0.toml-only. ODS keeps this kind of thing symmetric: anything a manifest or schema declares (.env.schema.json) is automatically surfaced as a labeled, described row in Settings via _build_env_fields (settings.py:194-241) — there's no config-file-only escape hatch for a setting with live behavior.

9. Open WebUI is only partly pre-wired. src/hal0/openwebui/ env_writer.py:106-131 sets chat routing and voice (STT/TTS) into Open WebUI, but never sets RAG_EMBEDDING_ENGINE/RAG_OPENAI_API_BASE_URL, ENABLE_WEB_SEARCH/SEARXNG_QUERY_URL, or ENABLE_IMAGE_GENERATION/ COMFYUI_BASE_URL — confirmed absent by a repo-wide grep. hal0 has an embeddings/rerank surface (/v1/embeddings, /v1/rerank) and an image pipeline (routes/comfyui.py, comfyui-pane.jsx), but neither is reachable through Open WebUI's own document-upload or image-gen buttons — a user who opens the "familiar" chat UI and clicks "generate image" or uploads a PDF gets Open WebUI's un-pointed defaults, not hal0. ODS wires all of this, including a full baked-in ComfyUI node-graph JSON (docker-compose.base.yml:127-146) so Open WebUI's native image button works unmodified.

Where hal0 is genuinely ahead. (a) RestartApiPanel.jsx is a complete, correctly-sequenced repair loop — confirm → mutate → poll /api/health → toast success or a specific failure message pointing at journalctl -u hal0-api — better than any single ODS remediation flow reviewed here. (b) ApplyBadge.jsx gives per-field reload-class information (live/needs-service-restart/needs- manual-restart) with the affected service named in a tooltip, which is more precise than ODS's page-level "Apply changes will recreate: X, Y" banner. (c) One shared classifier (health_report.py) drives both the CLI and the API, structurally preventing the two from disagreeing — ODS's status text is duplicated per-router (SERVICE_DESCRIPTIONS in the frontend, message= strings in each Python router) with no single source of truth. (d) The agents-as-trading-cards treatment (agents-overview.jsx) is a distinctive, memorable identity choice ODS has nothing like — a legitimate strength if delight is a goal, at some cost to a brand-new user's ability to tell what an "ability" with "cost 2" and "pow 90" actually does.

D. Port candidates

From ODS hal0 target Size Risk
next_steps population pattern: features.py:130-144 (human suggestion sentences) + :188-250 (numbered steps/links) Populate health_report.to_diagnosis() (src/hal0/health_report.py:246-283) with real NextSteps for the 7 existing checks (reuse the commands already known to cli/doctor_commands.py); make DiagnosisPanel.jsx's command chips actually copy-to-clipboard or run S–M Low — additive; scaffolding (Diagnosis, NextStep, the panel) already exists and is tested
FeatureDiscovery.jsx:246-300 modal (numbered steps + deep links, rendered from a live API response) Reuse for next_steps once populated above, or for slot-load failures surfaced by #1845/#1870 S Low
Extensions.jsx:60-84 STATUS_STYLES/STATUS_DESCRIPTIONS (status word → one sentence) Add a parallel plain-language map alongside slot-status.js's tooltips — keep the precise phase vocabulary, add a "what this means" clause for crashed/stopped/pulling S Low
installers/lib/ui.sh:547-549 "auto-disabled X — here's the command to re-enable" pattern Apply anywhere hal0 auto-derives a default with a workaround (e.g. tool_model #2108, ComfyUI-gated low-VRAM boxes) S Low
docker-compose.base.yml:82-175 full Open WebUI env wiring (RAG/websearch/image-gen) src/hal0/openwebui/env_writer.py — conditionally add RAG_EMBEDDING_*/ENABLE_IMAGE_GENERATION/COMFYUI_* only when the relevant hal0 slot is actually bound, mirroring ODS's ENABLE_COMFYUI:-false} gating discipline (docker-compose.base.yml:126-127). Correction (2026-10-04): a one-shot gate drifts — openwebui.env is written at install and the writer's preserve_existing merge keeps every existing key on upgrade (env_writer.py:251-267), so a flag set true stays true after the slot is unbound and nothing re-renders when one is bound later. The generated keys need their own owned block, re-rendered (with an Open WebUI restart) from the slot-change hook report 02 §D.5 calls depends_on_slots — or point Open WebUI at stable hal0-api routes that answer with a clear error when no slot is bound, so no env rewrite is needed M Medium — must gate on live slot state or it repeats ODS's own Apple-VRAM-fallback footgun (features.py:27-32) of claiming a capability that isn't really there
README.md:124-139 "At A Glance" + "If you know X, ODS adds Y" tables hal0 README.md, near the top, framed against Ollama/llama.cpp/Open WebUI/LocalAI XS None (copy only)
docs/POST-INSTALL-CHECKLIST.md 6-step "did it work" script docs/getting-started/first-chat.mdx or a new verify.mdx XS–S None
TroubleshootingAssistant.jsx Symptoms → Cause → Solutions with copy buttons A CLI-doctor-backed panel using the already-existing NextStep data once wired (overlaps row 1) — —

E. Do-not-copy

  • The CRT typing-effect boot theatrics (ui.sh type_line/type_line_dramatic, a real per-character sleep). ODS itself gates this off when $INTERACTIVE != "true" — it knows it's for a human's first, foreground, one-time run. hal0's installer is used repeatedly (upgrades, Proxmox CT scripting, fleet installs); adding artificial per-character delay anywhere in that path would actively hurt the audience hal0 has.
  • The LORE_MESSAGES brand-voice copy verbatim (ui.sh:154-168). The pattern of a consistent narrator voice is worth having; the specific ideological copy ("No terms of service. No content policy. Just freedom.") is ODS's brand, not a UX mechanism, and shouldn't be imported as text.
  • ODS's flat, 60-file, ALL-CAPS docs/ directory. hal0's Starlight getting-started/concepts/guides/reference split is already better information architecture (less duplication, clearer entry points); don't flatten it to match ODS.
  • Client-side status re-derivation (Dashboard.jsx's ~250 lines of pickFeatureLink/findHealthyService/normalizeFeatureStatus helpers). hal0 already has the better pattern — one server-side classifier (health_report.py) shared by CLI and API — and should keep extending that, not add a second, client-side resolution layer like ODS's.
  • The Setup/Owner magic-link invite model (Invites.jsx). This solves a multi-tenant "who's allowed in this household appliance" problem hal0's single-operator, trusted-LAN posture doesn't have today. Not applicable unless hal0 adds multi-user access as a feature in its own right.

F. Owner decisions

  1. First-run wizard, or stronger post-install confirmation? hal0's "installer is the whole setup" stance (README.md:39-45) is explicit policy, not an oversight — the question is narrower than "add a wizard": does the end of install.sh need an ODS-style summary (what's running, what isn't, exactly what to run next) rather than relying on the operator to find hal0 status themselves?
  2. Wire next_steps end-to-end for /api/doctor. Low-risk, scoped, and the scaffolding already exists on both the CLI (doctor_commands.py) and UI (DiagnosisPanel.jsx) sides — this is the single highest-leverage fix in this review.
  3. Pre-wire Open WebUI's RAG/search/image-gen, or keep them hal0-native-only? If the dashboard's own comfyui-pane and /v1/embeddings are meant to be the only front door, that's a legitimate "one UI, not two" decision — but it should be a decision, not an omission, and should be stated somewhere a user would find it before they click Open WebUI's native "generate image" button and get nothing.
  4. Who is the copy for? The current technical register (API paths and config keys in tooltips) is coherent for hal0's stated homelab-operator audience. Moving any of it toward ODS's consequence-first, jargon-free register is a deliberate audience decision, not a bug fix — worth scoping explicitly rather than drifting page-by-page.
  5. Give cross-cutting notification gaps (toast-on-approval, #1994) an owner. The issue itself flags this as "cross-cutting… not memory-specific" — worth deciding whether it's a tracked initiative (audit every SSE/poll surface for a matching toast) rather than a one-off patch.
  6. #1822's LAN-bind warning: disclose more loudly, without re-creating KB-1's scar. The doc Aside already exists; the ask is a non-enforcing, non-blocking line in the install summary and a hal0 doctor all row. Given the prior incident (auto-enforcement made things worse), any implementation should be reviewed explicitly against the O19 lesson cited in the issue before shipping.
08-docker-cli-amd.md Container runtime, CLI, AMD tuning

ODS vs hal0 — Container Runtime, Config Contracts, CLI, and AMD/Strix Halo Tuning

Scope: container runtime (Docker/Podman), compose/config contracts, the operator CLI, and AMD Ryzen AI Max ("Strix Halo") host tuning. All citations are file:line against the two repos on disk: ODS at /home/user/ods/ods/ and hal0 at /home/user/hal0/.

A. ODS mechanism

A.1 Compose layering & contracts

ODS assembles its runtime from Docker Compose file merging, documented as a formal contract rather than convention. docker-compose.base.yml (475 lines) defines the always-on core: llama-server as a "GPU-agnostic stub" (docker-compose.base.yml:24-25, image defaults to ghcr.io/ggml-org/llama.cpp:server-b9014), open-webui (:82-186, pinned ghcr.io/open-webui/open-webui:v0.7.2), dashboard-api, model-router, two remote-provider-egress services, and dashboard. Every service in the base file carries a deploy.resources.limits/reservations block (e.g. :173-180, :281-288), a healthcheck (7 of them in the base file), restart: unless-stopped (24 occurrences across base + extensions), and security_opt: [no-new-privileges:true]. dashboard explicitly gates on dashboard-api's health (depends_on.dashboard-api.condition: service_healthy, :463-465).

Hardware overlays (docker-compose.{nvidia,amd,apple,arc,intel,cpu}.yml) only override llama-server, dashboard-api, and open-webui — never redefine the whole stack. docker-compose.amd.yml rebuilds llama-server from a Dockerfile.amd into image ods-lemonade-server:latest (:6-17), wraps it with a custom entrypoint that syncs LEMONADE_CTX_SIZE into Lemonade's cached config.json on every start (:18-24, because Lemonade "only reads the env on first-start"), and passes /dev/dri + /dev/kfd plus VIDEO_GID/RENDER_GID group_add (:48-53). Mode overlays (docker-compose.{cloud,external-llm,lemonade-external,tier0}.yml) use Compose profiles: [local-inference] plus restart: "no" to stub out llama-server without deleting the service reference other files' healthchecks depend on (docker-compose.cloud.yml:8-16) — the "dependency placeholder" pattern docs/COMPOSE_RESOLVER_CONTRACTS.md:75-87 calls out as first-class. Multi-GPU overlays (docker-compose.multigpu-{amd,nvidia}.yml) layer again on top for ROCR_VISIBLE_DEVICES/tensor-split flags.

docs/BACKEND-CONTRACT.md:1-39 and config/backends/{amd,nvidia,cpu,apple}.json define one JSON contract per backend (LLM engine, public port/health URL, provider name/URL — e.g. config/backends/amd.json:1-18 pins ghcr.io/lemonade-sdk/lemonade-server:v10.2.0 and records linux_backend: rocm vs windows_backend: vulkan), loaded by scripts/load-backend-contract.sh and consumed in installers/lib/detection.sh. config/ports.json is the single canonical port map (env_var, external_default, internal_port per service).

Config-contract enforcement: .env.example (592 lines) documents every variable; .env.schema.json (1522 lines, JSON Schema draft 2020-12) requires six secrets (WEBUI_SECRET, SEARXNG_SECRET, N8N_USER, N8N_PASS, LITELLM_KEY, OPENCLAW_TOKEN) and types/enums/ranges for the rest. scripts/validate-env.sh (611 lines) is a from-scratch Bash .env parser (never sources the file, "for security reasons", scripts/validate-env.sh:90-92) that checks required keys, unknown keys, type/enum/range/minLength, and hand-written cross-field "runtime contracts" — e.g. remote-LLM transport requires HTTPS and rejects loopback/link-local hosts (:450-461), EMBEDDING_MODEL must be a HF TEI repo id not a GGUF (:475-485). .github/workflows/validate-env.yml runs this as a 5-way matrix (tiers 0-4), each generating a synthetic .env via a stubbed phase-06 environment before validating (/home/user/ods/.github/workflows/validate-env.yml:13-118).

lib/safe-env.sh is the shared safe loader (load_env_file, load_env_from_output, load_env_from_output_allowlist) — explicitly "no eval, no word-splitting" (lib/safe-env.sh:1-9), and it special-cases UID because Bash's own readonly UID would otherwise abort every lifecycle command under set -e (:33-36). lib/dotenv-quote.sh:1-25 serializes a value into the one-line .env grammar shared by Bash/Compose. lib/rootless- ownership.sh (250+ lines) detects Docker-rootless vs Podman-rootless via two different docker info fields (.SecurityOptions vs .Host.Security.Rootless, :8-29) and repairs per-service bind-mount ownership with a disposable busybox:1.36.1 helper container rather than chowning from the host (:4, :130-150). ODS_UID/ODS_GID (.env.example:487-491) are read by three extension composes directly (extensions/services/hermes/compose.yaml:18-19, n8n/compose.yaml:6, privacy-shield/compose.yaml:10) and by that repair script's per-service UID/GID table (lib/rootless-ownership.sh:220-260).

Image tag pinning is a deliberate, written trade-off: docs/ADR-IMAGE-TAG- PINNING.md accepts (2026-03-04) that OpenClaw, SearXNG, and Whisper stay on :latest because SearXNG has no stable release channel and pinning risks silently missing patches on an optional, already-hardened service — but llama-server, open-webui, and the AMD build args are all pinned to specific tags. docs/KNOWN-GOOD-VERSIONS.md records tested Docker/Compose/ driver baselines per platform. docs/DOCKER-DESKTOP-OPTIMIZATION.md is a Windows/WSL2-specific tuning guide (CPU/memory sliders, disk-image relocation to NVMe) — there is no equivalent for native Linux or AMD.

A.2 Podman

ODS does not run on Podman as a first-class, documented target — there is no Podman row in docs/PLATFORM-TRUTH-TABLE.md or docs/SUPPORT-MATRIX.md. What exists is defensive handling for the case where the docker binary itself is the Podman Docker-compatibility shim, called out as "common on AMD Ryzen AI / Fedora / RHEL hosts" (installers/phases/05-docker.sh:390-401). _runtime_is_podman() greps docker version for "podman" (:398-401); when true, installers/lib/podman-registries.sh:4-234 rewrites a user-level ~/.config/containers/registries.conf.d/99-ods-dockerhub.conf so bare image names resolve against Docker Hub, "because podman has no implicit docker.io like Docker does" (:390-393). This is careful, tested code (idempotent TOML editing in Python, tests/test-podman-rootless-contracts.sh) but a compatibility patch, not a native path: ODS still emits and resolves Docker Compose YAML, never podman-compose or Podman Quadlets. Rootless is handled generically for whichever runtime answers "rootless": true" (lib/rootless-ownership.sh:8-29), and Docker Desktop is its own branch (docs/DOCKER-DESKTOP-OPTIMIZATION.md). Net: ODS treats "docker" as the contract and tolerates Podman when it wears a Docker mask; it is not validated against native Podman networking, Quadlets, or podman machine.

A.3 AMD / Strix Halo host tuning

installers/phases/10-amd-tuning.sh (304 lines) is the concrete, host-level tuning ODS applies only when GPU_BACKEND=amd:

  • Group membership & device presence — adds the installing user to render/video (:34-38), modprobes amdkfd if /dev/kfd is missing (:41-51), and verifies /dev/dri/renderD128 exists (:53-61).
  • sysctl — installs config/system-tuning/99-ods.conf (vm.swappiness=10, vm.vfs_cache_pressure=50) to /etc/sysctl.d/ (:102-111).
  • amdgpu modprobe options — installs config/system-tuning/amdgpu.conf (options amdgpu ppfeaturemask=0xffffffff and gpu_recovery=1) (:113-121).
  • GTT/TTM sizing, scaled by RAM — computes a GTT percentage of total RAM (90% at ≥96 GB, 80% at ≥64 GB, else 65%; :140-152) and writes /etc/modprobe.d/amdgpu_llm_optimized.conf with options amdgpu gttsize=<MB>, options ttm pages_limit=<pages>, options ttm page_pool_size=<pages/2> (:163-170), then rebuilds initramfs (:172-174). A 128 GB Strix Halo box lands at ~115 GB GTT.
  • GRUB kernel cmdline — single-GPU APUs get amd_iommu=off appended for "~6% memory bandwidth improvement" (:186-229); multi-GPU boxes instead require iommu=pt for device passthrough and are only warned, not auto-edited (:192-201).
  • tuned profile — installs tuned if absent and sets accelerator-performance for "5-8% pp improvement" (:245-284).
  • A BIOS banner recommends setting UMA Frame Buffer Size to 512 MB minimum (:123-136), and a reboot-required banner fires whenever GRUB or modprobe files changed (:289-302).

installers/lib/amd-topo.sh provides vendor-neutral multi-GPU topology detection (amd_gpu_id, amd_gfx_version, detect_amd_topo), with a three-tier fallback chain per fact (rocm-smi → amd-smi → sysfs ip_discovery → rocminfo). docs/LEMONADE-SDK-COMPAT.md documents a managed-vs-external Lemonade split (own container vs. wrap a pre-existing host Lemonade via --use-existing-lemonade, with host.docker.internal networking notes and a runtimeMode/managedByODS diagnostic surface). docs/AMD-ODS2-GAMMA-BRIEF.md is a product brief for a Strix-Halo "drop-ship appliance" flow — engineering-adjacent, not a runtime mechanism. The Docker 29.x bug workaround (installers/phases/05-docker.sh:194-244): Docker 29.3.x regresses AMD /dev/dri custom-device passthrough; ODS detects the server version and, on AMD, auto-downgrades to 29.2.1 via the OS package manager, restarting the daemon afterward (:240-243).

hal0 has none of this tuning. A repo-wide grep for gttsize, ttm.pages_limit/page_pool_size, amd_iommu, GRUB_CMDLINE, tuned-adm/ accelerator-performance, and vm.swappiness across installer/, packaging/, src/hal0/hardware, and src/hal0/install returns zero matches. What hal0 does have is device-presence diagnosis, not host tuning: installer/lib/preflight.sh (~950-1050) runs rich /dev/dri / /dev/kfd / /dev/accel gid-mapping checks with named error codes (HAL0_GPU_RC_NO_KFD, HAL0_GPU_RC_KFD_GID, etc., tailored to LXC device passthrough where a render node's gid can map to an unrelated host group like clock), and src/hal0/install/gpu_perms.py:1-56 + installer/systemd/hal0-gpu-perms.service:1-29 re-converge /dev/kfd's group once per boot (since udev recreates the node fresh every boot) — which is a materially more robust mechanism than ODS's one-shot, install-time-only usermod -aG render,video (10-amd-tuning.sh:34-38, which also requires a fresh login shell to take effect and is never re-checked after that). src/hal0/providers/_gpu.py:70-71 hard-codes a last-resort GID fallback table explicitly labelled "Strix Halo LXC values" (render: 993, video: 44). HSA_OVERRIDE_GFX_VERSION exists in hal0 only as a manual, per-slot [server].env escape hatch documented in src/hal0/config/schema.py:298-306 — nothing auto-detects gfx1151 and sets 11.5.1 the way ODS's phase 06 does (referenced from .env.example:477). ROCBLAS_USE_HIPBLASLT, sysctl swappiness, GTT/TTM sizing, GRUB amd_iommu=off, and tuned-adm accelerator-performance are entirely absent. This gap is corroborated by hal0's own upstream image source: /home/user/amd-strix-halo-toolboxes/README.md:179-198 (the fork hal0's ROCm runner images are built from) explicitly documents amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 as required GRUB parameters for Strix Halo, citing a 5-12% performance delta — almost the exact tuning ODS automates — yet nothing in hal0's own installer applies it.

Correction (2026-10-04) — the IOMMU line is not part of the gap. hal0's own Strix Halo guidance is iommu=pt amd_iommu=on (docs/getting-started/drivers.mdx:100-112), and ADR-0003 records why: the community amd_iommu=off tweak silently removes /dev/accel, the XDNA NPU device, so hal0 deliberately recommends it nowhere (docs/adr/0003-onnx-text-generation-npu.md:96-99). The toolbox README's 5-12% and ODS's "~6%" are both that IOMMU-off delta. The gap hal0 should close is GTT/TTM sizing, the sysctls and the tuned profile; the IOMMU parameter stays as hal0 documents it. The GTT/TTM half is also narrower than ODS's default-on: on kernels ≥ 6.14 amdgpu sizes the pool dynamically and hal0 reads the live value per request (drivers.mdx:86-97), so fixed sizing is an opt-in floor (see D.1).

A.4 CLI

ods-cli is a single Bash file, 6,595 lines / 269,480 bytes (ods-cli:1-6595, confirmed via wc -l/ls -la), dispatched from one case "${1:-help}" at the bottom (ods-cli:6562-6595). The verb tree (with aliases) is:

gpu|g          status topology assignment validate reassign
status|s       (table); status-json (separate top-level cmd, not `status --json`)
list|ls        enable/disable-aware service listing
enable/disable/purge   extension lifecycle
preset|p       save|load|list|delete|export|import  (.env + extension state + meta.txt)
mode|m         local|cloud|hybrid  (rewrites LLM_API_URL/ODS_MODE)
model          current|list|swap
remote-provider  list|status|plan
stt            current|status|download
backup/restore/rollback
logs|log|l <service> [lines]   (hardcodes `-f`/follow, no way to opt out)
restart|r / start / stop       (special-cases "hermes" and "llama-server" inline)
update|u, shell|sh, config|cfg (show|edit|validate), chat|c, benchmark|bench|b
doctor|diag|d [--json]
audit, template|tmpl, agent (controls the ODS **host agent** daemon, not Hermes)

32 cmd_* functions total (grep count). cmd_config show (ods-cli:2260-2317) masks secrets by schema lookup before printing .env, and cmd_config validate shells out to the same scripts/validate-env.sh used in CI (:2289-2312). cmd_status_json (:1402-1517) builds one JSON document via jq -n/jq -s and per-service HTTP health probes — but it is its own top-level command (ods status-json), not ods status --json; ods doctor --json uses the opposite convention (completions/ods-cli.bash:44-46), so JSON output is inconsistent across the CLI's own surface. cmd_preset snapshots .env + per-extension state + a meta.txt into presets/<name>/ and validates compatibility before load. Notably, "hermes" is not a subcommand tree — it is a magic string special-cased inside the generic cmd_restart/cmd_stop bodies (ods-cli:1583, :1787); cmd_agent (:4756-4900+) is unrelated — it controls ods-host-agent.py, a local Wi-Fi/setup helper daemon.

ODS's own docs/ODS_CLI_DECOMPOSITION.md acknowledges the monolith as a liability — "the file is large enough that future changes need a staged decomposition plan" (:1-8) — with a six-stage extraction order (read-only helpers → status/logs → diagnostics → model/mode → extensions → lifecycle/backup last, :26-92). The plan is written but unexecuted; ods-cli remains one file. docs/MODE-SWITCH.md documents the one-env-var (LLM_API_URL) mode design cleanly. docs/PROFILES.md is misleadingly named: it documents that Docker Compose profiles (not CLI presets) were removed — "removed in favor of the current all-core architecture for simplicity" (:39-41) — so all 16 core services always start today, and disabling one means hand-editing a docker-compose.override.yml. completions/ods-cli.bash is a single static Bash completion file mirroring the verb/alias list by hand.

A.5 Runtime images

ODS builds llama-server per backend: NVIDIA and CPU pull upstream ghcr.io/ggml-org/llama.cpp build tags directly (docker-compose.nvidia.yml:8, docker-compose.cpu.yml:8); AMD builds a custom image (ods-lemonade-server:latest) from Dockerfile.amd with build args AMDGPU_TARGET/LLAMA_CPP_REF/LEMONADE_SERVER_IMAGE (docker-compose.amd.yml:10-17); Intel Arc builds SYCL from source (docker-compose.arc.yml:29-33). None of these are digest-pinned — every image reference in every compose file is a tag (:server-b9014, :v0.7.2, :v10.2.0), never @sha256:....

B. hal0 today (verified)

hal0 has no Docker Compose file anywhere in the repository (confirmed: find /home/user/hal0 -maxdepth 1 -iname "docker-compose*" is empty, and packaging/ holds only systemd/toolbox/runner/proxmox/sudoers). The entire runtime is Podman, orchestrated by systemd, one container per slot under hal0-slot@<name>.service (ARCHITECTURE.md:27-51). ContainerProvider (src/hal0/providers/container.py:1-40) is "the sole slot-lifecycle backend" for every provider type — GPU llama-server, FLM NPU, Kokoro/Qwen3-TTS, and ComfyUI all render through one function, _render_quadlet_from_plan (:803-990+), which emits a Podman Quadlet .container unit rather than a hand-built podman run string: typed Image=/AddDevice=/Volume=/PublishPort=/Environment= keys, with GroupAdd=/SecurityOpt= deliberately routed through PodmanArgs= instead of native Quadlet keys because "4.x generators HARD-FAIL the conversion on those keys" on real hardware (:899-909, citing live evidence from two named test boxes, halo150/halo143). Restart policy is systemd-native and far more expressive than Compose's restart: unless-stopped: Restart=always, RestartSec=5, RestartSteps=6, RestartMaxDelaySec=300 (a 5s→10→20→39→ 77→152→300s geometric ramp), capped by StartLimitIntervalSec=1800/ StartLimitBurst=10 at the [Unit] level (:854-891) — and the code comments record a real regression (#1424) from an earlier, too-tight 300s/5-burst pairing that "LATCHED the unit in failed" for hours. There is, however, no resource ceiling: a repo-wide grep for MemoryMax=, CPUQuota=, or any memory/CPU limit key in container.py, base.py, or config/schema.py returns nothing — every slot container runs unconstrained on host resources, unlike every ODS Compose service's deploy.resources.limits.

Podman images are always fully-qualified (ghcr.io/hal0ai/...), so hal0 never hits the "podman has no implicit docker.io" short-name problem ODS works around in podman-registries.sh. Every runner image is pinned by both tag and sha256 digest in manifest.json's toolbox_images block, each with a dated provenance note (e.g. the cpu entry records the exact llama-server --version output used to verify the pinned digest). /home/user/hal0-runner-images is the dedicated aggregator repo: "owned" Dockerfiles (cpu/flm/kokoro/moonshine/comfyui) plus "referenced" images pinned by digest but built upstream (vulkan/rocm → Hal0ai/amd-strix-halo-toolboxes, a hal0-maintained fork of kyuz0/amd-strix-halo-toolboxes adding *-server entrypoint variants for systemd) — CI resolves published ghcr digests and opens an automatic manifest-bump PR against hal0 (hal0-runner-images/README.md:8-21). The OpenWebUI companion is likewise a hand-rendered, digest-pinned systemd/Podman unit (packaging/systemd/hal0-openwebui.service:34,68, same Restart=always backoff pattern, ExecStartPre=-podman pull ...@sha256:...).

Config validation runs through Pydantic v2 models (src/hal0/config/schema.py:1-20): "All TOML files under /etc/hal0/ are validated against these models at startup. Typos like backend = "vukan" raise a ValidationError with the field path." This is a stronger mechanism than ODS's bash/jq validator (native typed enums/unions, validated on every process start rather than only when an operator remembers to run ods config validate), though hal0 has no equivalent of ODS's CI matrix that materializes and validates a synthetic config per hardware tier.

hal0's CLI (src/hal0/cli/) is Typer-based and modular: 35 files, 18,245 lines total, the largest single file (doctor_commands.py, 2,272 lines) still well under half of ods-cli's size. main.py:61-127 builds one root Typer app and mounts ~17 sub-apps via app.add_typer(...) — slot, model, memory, config, doctor, upstream, capabilities, comfyui, agent, app, migrate, registry, runner-images, profile, mcp, auth, board — plus flat commands (bench, chat, system-info, ports, status, update, serve, uninstall). Every mount point carries an inline comment naming the GitHub issue that motivated it (e.g. :88-91 for runner-images), and a deprecated command (probe, :234-244) is kept as a hidden, redirecting shim rather than silently removed. The CLI calls the same /api/* HTTP surface the dashboard uses (status() at :198-230 hits /api/status, /api/slots, /api/upstreams) rather than shelling out to systemctl/podman directly — one behavior surface for both consumers.

Two concrete, source-confirmed CLI defects match the owner's characterization:

  • Long-blocking lifecycle verbs with no progress feedback. src/hal0/slot_lifecycle_budget.py:1-192 (written in response to #1832, a prior timeout mismatch) derives client-side HTTP timeouts from the server's own worst case: HEALTH_TIMEOUT_S=180, TERMINATE_TIMEOUT_S=30, EVICTION_UNLOAD_ALLOWANCE=3, plus an output-sanity gate up to 360s on a CPU-backed slot (:29-136). The derived timeouts — SLOT_LOAD_TIMEOUT_S ≈ 1587s, SLOT_UNLOAD_TIMEOUT_S ≈ 828s, SLOT_LIFECYCLE_TIMEOUT_S (restart/swap) ≈ 2415s, via slot_lifecycle_timeout_s() (:165-192) — are consumed directly by slot_load/slot_unload/slot_restart in src/hal0/cli/slot_commands.py:311-362, each one blocking api_post(...) call: no spinner, no incremental state (the server does transition starting→warming→ready internally per ARCHITECTURE.md:231-266, but the CLI never polls or streams it) — just the final state once the HTTP response returns or the timeout fires. The module itself flags the remaining gap as open: "bounding it server-side is the real fix (#1869)" (:107). This is the mechanism behind a lifecycle verb sitting silent for minutes; the owner-cited 966s figure falls inside this documented envelope.
  • hal0 update does not refresh /usr/local/bin/hal0 (#1844). installer/install.sh:1074-1099 creates the PATH symlink exactly once, at install time. The self-update path (src/hal0/updater/updater.py, activate_release()) atomically swaps /usr/lib/hal0/current and pip install --force-reinstalls into the same venv (_reinstall_into_venv, :2338-2341: "apply() swaps the current symlink but the venv imports hal0 from its own site-packages... until the code is reinstalled") — but nothing re-runs the ln -sfn step, so a box whose wrapper predates a venv/path change keeps resolving a stale binary. This caused a real, separately-numbered regression (#2092, fixed) where hal0 agent reprovision's privilege-drop re-exec resolved via shutil.which("hal0") and silently re-exec'd an old release as root (src/hal0/cli/agent_commands.py:610-625). CHANGELOG documents the operator-facing symptom and workaround across three releases: "check with hal0 --version and re-run the installer if it disagrees" (CHANGELOG.md:2299-2306, echoed at :1085-1088, :2397-2399).

hal0 has no WSL2 path at all — WSL appears only in host-classification code (src/hal0/hardware/probe.py:644,664) used to label a detected Microsoft kernel, never in an install flow or doc.

C. Better / worse / equivalent

ODS is clearly better at: (1) documented AMD/Strix Halo host tuning — sysctl, GTT/TTM sizing, GRUB cmdline, tuned profile — all automated, RAM-scaled, and independently corroborated by the same upstream toolbox project hal0 forks from; (2) breadth of documented, tested platform combinations (NVIDIA, AMD, Apple, Intel Arc, CPU, WSL2, Docker Desktop) against a named compatibility contract; (3) a written, CI-enforced env-schema contract with cross-service checks run as a 5-tier matrix on every PR; (4) a real Docker-Engine-version regression workaround (29.3.x /dev/dri) reflecting production field experience.

hal0 is clearly better at: (1) the runtime mechanism itself — Podman Quadlets + native systemd restart/backoff/rate-limiting is a cleaner, more debuggable primitive than Compose's coarser restart: unless-stopped, and needs no daemon; (2) boot-persistent GPU device-node convergence (hal0-gpu-perms.service) versus ODS's one-shot, login-shell-dependent usermod; (3) supply-chain pinning — every hal0 runner/companion image is digest-pinned with provenance notes, whereas ODS pins by tag only and has a written ADR accepting :latest for three services; (4) config validation as a load-time, typed contract (Pydantic) rather than an optional, separately-invoked script; (5) CLI architecture — a modular, issue-annotated Typer tree is a healthier long-term surface than one 270 KB Bash file, even though ods-cli is more feature-complete in places (masked config show, preset export/import).

Roughly equivalent: rootless-ownership handling (ODS's busybox-helper chown dance vs. hal0's boot-converge service — both exist because container runtimes and host device nodes disagree on ownership); mode/provider routing (ods mode {local,cloud,hybrid} vs. hal0's upstream/dispatcher model).

Podman support specifically is not equivalent: ODS runs on Podman only incidentally and never documents it as supported; hal0 is Podman-native by design. ODS's edge is specifically the tuning and platform-breadth axes named above — its actual container-runtime mechanism (Compose + a real Docker Engine dependency, needing its own version-regression workaround) is the less robust of the two once Podman/rootless edge cases are in play.

D. Port candidates

  1. AMD/Strix Halo host tuning phase (highest value, most direct port). Source: installers/phases/10-amd-tuning.sh (304 lines, logic reusable near-verbatim, e.g. the GTT sizing at :146-157 — gtt_pct 90/80/65 by RAM tier, pages_limit = gtt_size_MB*1024*1024/4096, page_pool_size = pages_limit/2). Target: a new installer/lib/amd- tuning.sh sourced once from installer/install.sh (hal0's installer is one 4,002-line script with no phase directory, so this is a function block called after GPU detection, not a new phase file). Size: ~150-180 lines of new Bash. Risk: medium — edits /etc/default/grub and /etc/modprobe.d/, needs a reboot, and should follow hal0's existing "never fail the boot" posture (gpu_perms.py:1-27, hal0-gpu-perms.service's SuccessExitStatus=0 1). Low-controversy on the merits: hal0's own upstream image source already publishes the amdgpu.gttsize=... ttm.pages_limit=... half of this recommendation (amd-strix-halo-toolboxes/README.md:186); open questions are packaging (default-on vs. flag) and reboot messaging. Scope (2026-10-04) — GTT/TTM, sysctl and tuned only. The port must not write ODS's amd_iommu=off (10-amd-tuning.sh:186-229): it removes /dev/accel (the XDNA NPU) and contradicts hal0's documented iommu=pt amd_iommu=on (drivers.mdx:102, ADR-0003:96-99). If the host already carries hal0's IOMMU line, leave it; if it carries amd_iommu=off, warn and leave it — the operator chose it knowingly. Default (2026-10-04) — opt-in, never default-on. drivers.mdx:86-97: on kernels ≥ 6.14 amdgpu grows the GTT limit dynamically, hal0 reads the live mem_info_gtt_total on every request, and pinning is documented as an operator choice. So the lib applies fixed sizing only on an explicit request (flag or --apply), the doctor reports the live pool and offers the recipe, and on a kernel ≥ 6.14 the default is to change nothing — above all on a multi-tenant Proxmox host, where a 65–90 % reservation starves every other guest and adds a boot-critical GRUB/initramfs edit for no measured gain. Correction (2026-10-04) — a container guard is mandatory. hal0's common deployment is a Proxmox LXC. Inside a container /etc/default/grub, /etc/modprobe.d/, the initramfs and amdgpu module parameters belong to the host kernel; editing the guest's copies and asking for a reboot applies nothing (and the IOMMU parameter is out of the port's scope regardless — see above). ODS's phase 10 has no such guard (10-amd-tuning.sh only remarks that containers need /dev/kfd, :33-55), so a literal port would report "tuning applied" on every LXC. The hal0 port must branch on systemd-detect-virt --container (the installer already distinguishes LXC in preflight_gpu, install.sh:363-380,500-509): on bare metal apply as above; in a container do not derive the tier from the guest's memory — an LXC sees its cgroup allocation, not host RAM (docs/getting-started/proxmox.mdx:171-173; the recipe's 32 GB guest on a 128 GB host would pick the wrong tier and undersize the host GTT; correction 2026-10-04) — so emit a host-side script that reads the host's /proc/meminfo and computes the tier there, or take --host-ram-gb explicitly, and print the host-side recipe (the GTT/TTM /etc/default/grub additions, update-grub, the /etc/modprobe.d/ drop-in, the sysctls) for the operator to run on the Proxmox host, and record the phase as skipped: container, never as applied. The hal0 doctor arm (item 3) should check live values (/sys/module/amdgpu/parameters/gttsize, /proc/cmdline) rather than the files it would have written.

  2. HSA_OVERRIDE_GFX_VERSION auto-detection for gfx1151. Source: .env.example:477 + ODS's gfx1151-conditional phase-06 logic. Target: src/hal0/install/profile_derive.py or the ROCm slot default [server].env near src/hal0/config/schema.py:298-306. Size: ~15-20 lines. Risk: low — one conditional env default into an injection point that already exists; only care needed is not clobbering an operator-set value.

  3. Kernel/firmware known-bad-version checks in hal0 doctor. Source: the same upstream README's warnings (kernel <6.18.4 unstable on gfx1151; linux-firmware-20251125 "breaks ROCm support," amd-strix-halo- toolboxes/README.md:53) plus ODS's version-gated remediation pattern (installers/phases/05-docker.sh:194-244). Target: a new check in src/hal0/cli/doctor_commands.py reading uname -r and rpm -q/dpkg -l for linux-firmware. Correction (2026-10-04): inside an LXC the kernel half still holds (uname -r is the host kernel) but the firmware half does not — the GPU runs firmware the host kernel loaded from the Proxmox host's linux-firmware, so the guest package is irrelevant. Under systemd-detect-virt --container the check must say "firmware is host-side", print the host command (dpkg -l linux-firmware on the Proxmox host) and accept host-sourced evidence, rather than passing or warning on the guest's package. Size: ~40-60 lines. Risk: low — read-only.

  4. Resource limits on slot Quadlet units. Source: ODS's deploy.resources.limits.{cpus,memory} convention (docker-compose.base.yml:173-180). Target: _render_quadlet_from_plan in src/hal0/providers/container.py:803-990+ — Quadlet has native [Service] keys (MemoryMax=, CPUQuota=) that drop in next to the existing Restart=/RestartSec= block, sourced from a new optional profile/slot field. Size: ~25-40 lines plus one schema field. Risk: low-medium — additive when unset, but should default off since hal0 currently assumes an unconstrained, single-tenant appliance box.

  5. CI matrix that validates a materialized config per bundle tier. Source: .github/workflows/validate-env.yml's 5-tier matrix. Target: a GitHub Action rendering each installer/manifests/omni/<tier>.json bundle's default slots.toml/capabilities.toml and asserting it loads under hal0.config.schema.Hal0Config without a ValidationError — closes the CI-coverage gap ODS's matrix has and hal0's (already-stronger) Pydantic validation does not. Size: ~50 lines. Risk: low.

  6. AMD multi-GPU topology detection (installers/lib/amd-topo.sh) is lower priority: Strix Halo is single-iGPU, so this only matters if hal0 ever supports discrete multi-GPU AMD boxes.

E. Do-not-copy

  • The 270 KB / 6,595-line ods-cli monolith. ODS's own decomposition doc concedes the problem; hal0's modular Typer tree (35 files, per-group ownership, issue-annotated mount points) is already the healthier architecture. Do not regress toward one giant dispatch file.
  • Compose "profiles removed, everything is core" as a configurability model (docs/PROFILES.md:39-41). ODS's own history is a cautionary tale: it had opt-in profiles and removed them, ending up with "disable a service by hand-editing an override file." hal0's per-slot, per-capability TOML enable/disable is strictly more granular already.
  • Silently downgrading a system package as an installer side effect (Docker 29.3.x→29.2.1, installers/phases/05-docker.sh:205-223). Reasonable for ODS's Docker-Engine-dependent world, but "installer reaches into apt/dnf and force-downgrades a system daemon" is not a pattern for hal0's podman/systemd model — detect-and-warn is the safer analog.
  • The written acceptance of unpinned :latest tags (docs/ADR-IMAGE-TAG-PINNING.md) is strictly looser than hal0's current digest-pin-everything discipline; don't use it as license to relax hal0's bar.
  • Multi-platform installer sprawl (installers/macos/, installers/windows/, docs/WINDOWS-*, docs/WSL2-*) is out of scope unless hal0 commits to non-Linux support.

F. Owner decisions

  1. Adopt AMD host tuning (D.1) as default-on or opt-in? (Answered 2026-10-04: opt-in — see the Default note in D.1.) It touches GRUB and needs a reboot; hal0 would want a "reboot recommended" banner (ODS's pattern, installers/phases/10-amd-tuning.sh:289-302) and a call on whether a single-target-hardware product should just always apply it.
  2. Close the #1870-class long-blocking lifecycle calls via the event bus hal0 already has (src/hal0/events/, /api/journal) instead of a bare blocking api_post + multi-thousand-second timeout — is streaming slot-state transitions into the CLI (and eventually bounding the server-side pieces named open at slot_lifecycle_budget.py:107, #1869) worth prioritizing now, or does it wait on #1869 first?
  3. Fix #1844 properly: should activate_release() (src/hal0/updater/updater.py) re-run the /usr/local/bin/hal0 symlink step idempotently on every update, or is /usr/local/bin/* deliberately outside the updater's managed tree, better enforced by re-invoking install.sh unattended at the end of hal0 update?
  4. Extend hal0's digest-pinning discipline into a published "known-good kernel/firmware" doc, mirroring docs/KNOWN-GOOD-VERSIONS.md and the amd-strix-halo-toolboxes warnings — worth wiring into hal0 doctor (D.3) rather than documentation alone.
  5. Resource limits on slot containers (D.4): is "unconstrained, single-tenant appliance" intentional for a Strix-Halo-only product, or a gap worth closing before a multi-slot box (chat + code + embedding resident at once, slot_lifecycle_budget.py:92-94) can starve itself?
09-issue-map.md Open-issue map

ODS → hal0 issue map

Scope. All 106 open hal0 issues (snapshot 2026-09-05) triaged against the ODS reference tree. Every "Direct port" / "Pattern adoption" row below was verified by opening the cited ODS code and the cited hal0 code; "Informs only" rows were verified on the ODS side and spot-checked on the hal0 side; "Unrelated" rows were checked only far enough to establish that no ODS shape applies.

Path conventions. ODS paths are relative to /home/user/ods/ods/ unless they start with .github/ (those are /home/user/ods/.github/). hal0 paths are relative to /home/user/hal0/.

Verdict definitions.

verdict means
Direct port An ODS file/function transliterates into hal0 with only naming/runtime substitution and closes the issue outright.
Pattern adoption ODS's structure solves the issue; the code must be rewritten for hal0's runtime (podman/quadlet/FastAPI) but the design decision is settled.
Informs only ODS's handling is a useful reference or precedent — it narrows the design space but does not itself close the issue.
Unrelated No ODS shape applies (polish sweeps, test hygiene, upstream tracking, docs, hal0-only seams).

1. Summary

verdict count share
Direct port 4 3.8%
Pattern adoption 34 32.1%
Informs only 29 27.4%
Unrelated 39 36.8%
total 106

The honest read: just over a third of the backlog is untouchable from ODS — 31 polish/test/docs/upstream issues plus 8 that ODS simply does not model (Hindsight memory internals, the runner image build, a11y). But the high-severity operational half of the backlog maps very well: 18 of the 20 issues I'd call "an operator hits this on a real box" land in Direct port or Pattern adoption.


2. Full triage table

# title (short) verdict ODS shape (file:line) what to do in hal0 eff imp
1974 _flm_image_present caches False forever Direct port extensions/services/dashboard-api/helpers.py:26-56 (_DirSizeCache TTL) Give the negative probe a TTL (or cache only the positive result) S M
1869 bound systemd/podman subprocesses Direct port ods-uninstall.sh:321; scripts/bootstrap-upgrade.sh:939; ods-cli:2166-2172 Pass an explicit timeout= on every seam call in the load path (container.py:2697/2708/2900) S H
1521 retention prune runs only at boot Direct port memory-shepherd/install.sh:150-226 (timer+service pairs); memory-shepherd.sh:329 Run prune on an interval task/systemd timer instead of once in the API lifespan S M
1512 stacks import never verifies checksum Direct port ods-restore.sh:273-310 Verify the envelope digest on the commit path; fail closed; keep --skip-verify as the explicit opt-out S H
2234 default_images[family] only on llama path Pattern adoption installers/lib/compose-images.sh:40-71 Resolve every slot image through one plan-derived seam so no provider path can bypass default_images[family] M M
2221 last_crash_line None on exit 64 Pattern adoption installers/lib/compose-failure-report.sh:92-186; ods-cli:1080-1128 Replace the single crash-line regex with a bounded evidence report: exit code + last N unit-log lines + likely image M M
2216 LXC /dev/kfd seeds gpu-vulkan Pattern adoption installers/lib/detection.sh:187-233; installers/phases/02-detection.sh:159-168 Make device-node presence (kfd_present() AND render_node_present()) sufficient for the ROCm lane; never gate a lane on a userspace tool being installed S H
2212 verify-files / re-pull for model + mmproj Pattern adoption ods-restore.sh:273-310; config/model-library.json:5-14 (gguf_sha256) Carry per-file sha256 in the registry row and verify/re-pull on demand for model + mmproj M M
2203 seed-profile route emits changed_fields always Pattern adoption ods-cli:6261-6324 (_template_preview delta) Compute the real delta once and emit changed_fields only when it is non-empty S M
2201 runs_on reads hardware per row per poll Pattern adoption extensions/services/dashboard-api/helpers.py:26-56, 585-619 Resolve hardware once per request from a TTL-cached snapshot written by a poller, not per row S M
2195 apply preview omits removed flags Pattern adoption ods-cli:6261-6324 Enumerate the complete delta (added/changed/removed) from one diff function shared with apply S M
2192 Omni never sees tts/stt/embed modality Pattern adoption extensions/services/comfyui/manifest.yaml:19,38; extensions/services/dashboard-api/config.py:297-431 Declare modality/capability facts in the slot profile manifest and derive Omni eligibility from that table M H
2180 migrate-flags aborts whole run Pattern adoption ods-cli:2059-2113 (per-service accounting, continue then report) Make the sweep fault-isolated per slot and report a summary instead of aborting the run S M
2164 pi install fails as daemon uid 996 Pattern adoption installers/phases/07-devtools.sh:66-73, 386-395 Give the daemon identity its own npm prefix ($HOME/.npm-global) or run the install as the target user with HOME set M M
2108 tool_model has no UI path Pattern adoption extensions/services/dashboard-api/settings.py:194-241; .env.schema.json Render settings fields from the config schema so a new key gets a UI path automatically M M
2096 update should restart slots on image roll Pattern adoption ods-cli:2049 (up -d --force-recreate); ods-cli:1601 (--pull never) Make recreate-on-image-roll the default and require an explicit opt-out S H
2028 services/health static three-branch Pattern adoption extensions/services/dashboard-api/config.py:297-431; helpers.py:624-700 Build the health list from a manifest table and fan out with gather(return_exceptions=True) M H
2019 dev-deploy leaves wrappers stale Pattern adoption installers/phases/13-summary.sh:283; ods-cli:2049 Have deploy.sh run the same wrapper/PATH refresh step hal0 update runs (refresh_privileged_wrappers) S H
1990 _VALID_FACT_TYPES hardcoded allowlist Pattern adoption extensions/schema/service-manifest.v1.json; scripts/validate-manifest-schema.sh Move the fact-type set into a shared schema both sides validate against, with a CI drift check M M
1983 gpu-perms Before=hal0.target loses race Pattern adoption docker-compose.base.yml:463-465 (depends_on: condition: service_healthy) Add an explicit per-unit ordering edge on the slot template (After=/Requires=), not a target-level Before= S H
1966 ComfyUI has no lane on kfd-less AMD Pattern adoption extensions/services/comfyui/manifest.yaml:19,38 (gpu_backends) Declare each runtime's supported lanes in the manifest; the picker offers only those and explains the gap M H
1936 segfault with zero GPU devices Pattern adoption installers/lib/detection.sh:187-233, 235-257 (apply_cpu_gpu_fallback) Refuse to spawn a GPU lane when no device nodes are mapped; fall back to CPU with a named reason S H
1870 lifecycle verbs wait 966s silently Pattern adoption installers/phases/12-health.sh:66-106; installers/lib/background-tasks.sh:119-148; readiness-summary.sh:52-127 Stream per-phase progress while waiting and bound the wedged call with an elapsed/deadline loop M H
1868 static seeds warm brain at 65536 Pattern adoption installers/lib/tier-map.sh:200-330; scripts/select-model.py:170-182 Derive seed context_size from the detected memory envelope, exactly as derive_device already derives the lane M H
1867 anchor window below Hermes floor Pattern adoption installers/phases/11-services.sh:1059 (_hermes_context=${MAX_CONTEXT}) Derive the agent window from the slot's served window and raise the ceiling at provision time M H
1862 stale hardware.json books VRAM as RAM Pattern adoption installers/lib/constants.sh:30; installers/lib/detection.sh:26-46; scripts/ods-doctor.sh:81-85 Rebuild the hardware fact from a live probe when the cache is absent/stale instead of degrading to 'no GPU' M H
1859 llama-only ctx resolver on FLM/Kokoro Pattern adoption extensions/schema/service-manifest.v1.json; scripts/load-backend-contract.sh Gate the context resolver on the slot's declared runtime family S M
1845 remediation panel cannot work Pattern adoption scripts/ods-doctor.sh:1110-1163 (issue id -> next_steps table) Emit runnable next_steps as data from the server, including --stop-services S H
1844 update never refreshes /usr/local/bin/hal0 Pattern adoption installers/phases/13-summary.sh:270-300 (ln -sf into the install dir) Re-assert the PATH symlink from the activated release next to refresh_privileged_wrappers S H
1834 extraction queue never drains on CPU Pattern adoption installers/lib/tier-map.sh:200-330; ods-cli:400-480 (ensure_llama_cpu_budget) Size background extraction from the detected tier: concurrency cap, max_tokens, and a retry ladder floor M H
1833 memory status green on empty store Pattern adoption installers/lib/readiness-summary.sh:47-96 (ready requires a real signal) Require a positive landed-fact signal for 'Writes landing', not absence of failures S M
1825 build_roster collapses model.gguf models Pattern adoption config/model-library.json:5-14 (id + gguf_sha256); scripts/select-model.py:76-115 Canonicalise the roster by content digest / registry id, never by gguf basename S M
1823 capability labels in hand-curated web file Pattern adoption config/model-library.json:1-40; scripts/select-model.py:76-115 Move capability labels into the versioned model registry and have the leaderboard read the registry M M
1822 LAN bind with auth off unsurfaced Pattern adoption scripts/ods-doctor.sh:1074-1089; .env.example:50-70; installers/phases/13-summary.sh:419-427 Add a doctor finding + install-summary line when the bind is non-loopback and auth is off S H
1820 netavark black hole has no repair owner Pattern adoption ods-cli:975-1060 (detect known-recoverable failure, repair, bounded retry); ods-cli:6115-6204 (cmd_repair) Call the existing prune_dnat seam from the reconciler behind a bounded retry, and expose a repair verb M H
1550 deploy reports success on old code Pattern adoption ods-update.sh:378-406 (wait_for_healthy); 647-742 (verify, then record version) Assert the served build identity after restart before printing success; roll back on failure S H
1519 uninstall prints 'Uninstalled' on 207 Pattern adoption ods-uninstall.sh:362, 394-402 (INSTALL_DIR_CLEANED + residual remedy) Track per-step outcomes, gate the final line on them, and print residual paths with the remedy command S M
1511 stack apply unloads untouched slots Pattern adoption ods-cli:6261-6324 (_template_preview: already / will-enable / nothing to change) Show the full delta including the unloads in the confirm dialog; make the unload sweep opt-in M H
2200 comfyui self-heal on list not single GET Informs only extensions/services/dashboard-api/helpers.py:624-680 Share one enrichment/self-heal function between the list and single-GET routes S M
2191 generate_image self-deadlocks single GPU Informs only installers/lib/model-lifecycle-lock.sh:15-60 Serialize GPU-exclusive ops behind a named lock with a bounded wait instead of a 503 mid-loop M H
2190 _runtime_family ignores runner aliases Informs only lib/service-registry.sh:29-31 (SERVICE_ALIASES) Fold runner aliases in one registry seam every classifier consults S M
2184 curated slots reference demoted profiles Informs only extensions/schema/service-manifest.v1.json; scripts/validate-manifests.sh Add a CI validator asserting every curated slot reference resolves to a shipped seed profile S M
2178 backfill Model.architecture Informs only ods-update.sh:678-690 (migrations dir); config/model-library.json Ship an idempotent data migration executed by the updater's migration step S M
2154 bank hygiene surface Informs only memory-shepherd/memory-shepherd.sh:110-148,329; memory-shepherd/install.sh:150-226 Age/size thresholds + a retention sweep on a timer, with a guarded 'suspiciously small → skip' rule M M
2118 retire upstream runner-image variant Informs only installers/lib/compose-images.sh:40-71 Derive the image set from the resolved plan so retiring a variant is a one-line data change S L
2101 GA delivery checklist Informs only ods-update.sh:456-505 Resolve 'latest' from the release API itself so tagging IS delivery — no separate pointer to forget M H
1969 nothing pins llama-server LOADING Informs only extensions/services/dashboard-api/helpers.py:661-672 (degraded != down) Model 'up but loading' as a distinct state and never feed it to the output-sanity gate S M
1967 save_slot_config drops keys on round-trip Informs only extensions/services/dashboard-api/settings.py:194-241 Round-trip through a schema-derived field table so unknown keys survive the write M M
1947 support CIRU vLLM distributions Informs only extensions/services/*/manifest.yaml; docker-compose.*.yml overlays Add a runtime family as manifest + overlay data rather than code L M
1932 read_slot_ceiling assumes .toml Informs only lib/service-registry.sh:29-48 Resolve the ceiling through the naming seam rather than assuming <name>.toml S M
1931 extraction preflight trusts config Informs only extensions/services/dashboard-api/helpers.py:624-680 Preflight the LIVE drop-in/endpoint, not the configured intent S M
1930 MCP memory_add has no ctx preflight Informs only extensions/services/dashboard-api/helpers.py:624-680 Route both HTTP and MCP add paths through one preflight helper S M
1929 memory-map GTT fallback reads dead keys Informs only installers/lib/llama-memory-budget.sh:6-31 Same single-source memory envelope; fail loudly rather than silently substituting system RAM S M
1928 memory ruler header/bar different bases Informs only installers/lib/llama-memory-budget.sh:18-31 Derive header and bar from ONE effective-memory function so they cannot disagree S M
1873 self-call clients bare-float timeouts Informs only ods-cli:3851-3880 (explicit per-call timeout argument) Import slot_lifecycle_budget in the brain self-call clients instead of bare floats S M
1858 by-name/by-id unenriched payload Informs only extensions/services/dashboard-api/helpers.py:624-700 Share one enrichment function across /api/slots, /by-name and /by-id S M
1545 every PR conflicts on CHANGELOG.md Informs only .github/pull_request_template.md; CHANGELOG.md:1-30 ODS has no fragments — but its PR template requires no CHANGELOG edit; notes are authored at cut time M M
1537 Ubuntu 26.04 py3.14 --ignore-requires-python Informs only installers/lib/python-runtime.sh:36-72,127-171 Provision/select a supported interpreter and refuse loudly with a named remedy instead of --ignore-requires-python M M
1530 stable channel pointer still 0.9.8 Informs only ods-update.sh:456-505 Same as #2101 — derive the update target from the release API rather than a hand-maintained channel pointer S H
1522 journald follow has no keep-alive Informs only extensions/services/dashboard-api/routers/talk.py:393-397,455-495 Wrap journalctl_sse in the keepalive + disconnect loop hal0 already ships in journal.py / slots/logs.py S M
1436 artefact names still derived from slot NAME Informs only lib/service-registry.sh:29-48 Route the four remaining artefact names through the naming seam S M
1429 quadlets hard-code LogDriver=none Informs only docker-compose.base.yml:12-16 (single logging anchor + rotation) Already switched to LogDriver=passthrough (container.py:890); adopt one log-policy constant plus rotation S M
1428 slot_sample UNIQUE constraint floods log Informs only extensions/services/token-spy/db.py:37 (surrogate PK, no natural-key UNIQUE) Drop the (ts, slot_id) natural key for a surrogate PK, or use INSERT ... ON CONFLICT DO UPDATE S M
1426 hal0-systemctl has no unmask verb Informs only ods-cli:6115-6204 (cmd_repair dispatcher) Add unmask to _UNIT_VERBS + the wrapper allow-list and expose it as a repair verb S M
1422 /api/slots duplicate entries Informs only lib/service-registry.sh:29-48 (id-keyed maps + alias table) Key the roster by durable id with names as an alias map, and dedupe at load M M
1421 POST /api/slots writes name-keyed Informs only lib/service-registry.sh:29-48 Route slot creation through the naming seam so an id-keyed box never gets a name-keyed artefact M H
1319 integrate llama-ai / CachyLLama Informs only installers/lib/tier-map.sh:200-330; docker-compose.*.yml overlays Add a runtime as a data-declared backend + compose/profile overlay rather than new code paths L L
2228 comfyui phase4 tests patch shared isfile Unrelated — Inject the filesystem seam per provider instead of patching os.path.isfile process-wide S L
2202 promote crud app fixtures to conftest Unrelated — Test-fixture consolidation S L
2169 cut PR wall clock Unrelated — ODS CI has no path gating/xdist to copy (single lint+smoke workflows) M M
2168 execute v1.1.0 sunset backlog Unrelated — Repo chore; no ODS analogue M L
2155 memory migrate unify HTTP 410 Unrelated — Upstream engine API version pin; no ODS analogue M M
2153 guide for external coding agents Unrelated — Documentation M L
2111 docs-sync fragment anchors Unrelated — Docs-sync anchor rewriting S L
2017 memory plugin one uuid4 doc per turn Unrelated — Hermes plugin metadata; no ODS analogue M M
2016 no document-ingest endpoint Unrelated — ODS delegates RAG ingest to Open WebUI's own Knowledge surface — no endpoint to port M M
2011 cross-backend profile switching dead code Unrelated — Dead-code removal S L
2003 docs-discourse-sync reconcile Unrelated — Forum sync bookkeeping M L
2002 flaky warming-slot test Unrelated — CI-only state race; fix the test's readiness barrier S L
1997 no e2e for slab-truncation notice Unrelated — Missing e2e coverage S L
1996 tag-chip dimming keys dead field Unrelated — UI field-name fix S L
1995 memory v2 mock graph dialect Unrelated — Test mock dialect fix S L
1994 no toast for queued agent delete Unrelated — UI toast S L
1993 no operation-retry affordance Unrelated — UI retry affordance S L
1989 list_memories comma-joins multi-type Unrelated — Client query encoding bug S M
1984 _vulkan_lane_is_loadable stale pin Unrelated — Cache/pin invalidation bug internal to the retag pass S M
1948 restore Vulkan backend / refresh upstreams Unrelated — Runner image build work L H
1925 A/B non-AMD Vulkan lanes Unrelated — Validation campaign L M
1841 polish(ui) rc.5 sweep Unrelated — UI polish sweep S L
1840 polish(cli/api) rc.5 sweep Unrelated — 13-item CLI/API polish sweep M L
1829 board tools 401 on fresh install Unrelated — Upstream wheel packaging gap M H
1821 nothing asserts slot-owned flag reaches argv Unrelated — Test coverage for argv rendering M M
1783 converge dashboard tables on .dtable Unrelated — UI table convergence M L
1756 quadlet allow-list follow-ups Unrelated — hal0-specific sudo wrapper allow-list; ODS has no equivalent seam S M
1552 Menu primitive has no keyboard support Unrelated — Frontend a11y work M M
1536 polish(board/palette) GA sweep Unrelated — Polish sweep S L
1529 polish(journal/logs) GA sweep Unrelated — Polish sweep S L
1528 polish(agent-cli) GA sweep Unrelated — CLI polish sweep S L
1525 polish(upstreams) GA sweep Unrelated — Polish sweep S L
1524 polish(stacks) GA sweep Unrelated — Polish sweep S L
1502 scar ratchet counts prose Unrelated — Repo CI ratchet tuning S L
1477 low-priority polish sweep Unrelated — 31-item polish sweep L L
1445 typed request bodies audit Unrelated — Typed-body audit follow-through M M
1349 Strix Halo field results Unrelated — Field-report tracking issue L L
1249 track hermes-agent py3.14 Unrelated — Upstream tracking issue S M
1070 validate/promote ROCmFPX variant Unrelated — Quantization validation work L M

3. Top 15 wins

Ranked by (impact ÷ effort). Each is an ODS shape I opened and a hal0 defect I located; the pointers are exact.

1. #1869 — bound the systemd/podman lifecycle subprocesses (S / H, Direct port)

ContainerProvider._run (src/hal0/providers/container.py:2549-2566) defaults timeout=None, and the three calls on the load path pass nothing: container.py:2697 (daemon-reload), container.py:2708 (systemctl restart <unit>), container.py:2900 (daemon-reload in the re-render sweep). SystemCtlSeam.systemctl (src/hal0/system/seam.py:415-489) forwards the bound faithfully on both routes — the plumbing already exists, nobody passes a number.

ODS never leaves a hang-prone privileged call unbounded: ods-uninstall.sh:321 — timeout 20s sudo -n -- systemctl disable --now …; scripts/bootstrap-upgrade.sh:939 — ps_env_cmd+=(timeout --foreground --kill-after=5s "${restart_timeout}s"); ods-cli:2166-2172 — a perl -e 'alarm 3; exec "docker","info"' guard, with the comment explaining that docker info hangs 20+s on a half-booted daemon.

Fix: give _run a non-None default (the existing _UNIT_STOP_TIMEOUT_S is already imported at container.py:2943) and let subprocess.TimeoutExpired become a typed slot.spawn_failed the same way CalledProcessError already does at container.py:2709-2716. That also shrinks the client budget in slot_lifecycle_budget.py — see #1870.

2. #1822 — LAN bind with auth off is never surfaced (S / H)

ODS surfaces this in three places, all data-driven: scripts/ods-doctor.sh:1074-1089 raises a warn finding (ODS-RUNTIME-EXTERNAL-LEMONADE-UNAUTHENTICATED-HOST-ROUTE) whose detail says exactly "if the daemon is bound beyond loopback so Docker can reach it, that same daemon may also be reachable from the LAN"; scripts/ods-doctor.sh:1150-1155 carries its next_steps; installers/phases/13-summary.sh:419-427 prints the LAN URL only when BIND_ADDRESS=0.0.0.0, and .env.example:50-58 states the danger inline on the OPENCLAW_DANGEROUSLY_DISABLE_DEVICE_AUTH key.

Fix: add one doctor check (src/hal0/cli/doctor_all.py, alongside check_hermes_anchor_window at doctor_all.py:1046) that reads the bind address and the auth toggle together and emits a warn with a remedy; echo the same line in the installer summary.

3. #1845 — the "Convergence incomplete" panel prints a remediation that cannot work (S / H)

src/hal0/cli/update_commands.py:466-492 renders entry.get('command') verbatim under prose that says "Stop hal0, then run:" — but the command the server hands over does not carry --stop-services, so a literal copy-paste refuses.

ODS keeps remediation as a table keyed by a stable issue id, never prose glued to a value: scripts/ods-doctor.sh:1110-1122 maps id → title, :1124-1163 maps id → next_steps (a list of runnable lines), and :1165-1175 assembles the diagnosis.

Fix: have /api/updates/convergence return next_steps: [str] per pending migration (built server-side, where --stop-services is known), and make the panel print exactly those lines. Same change fixes the class, not just this instance.

4. #1844 — hal0 update never refreshes /usr/local/bin/hal0 (S / H)

installer/install.sh:1077-1082 links /usr/local/bin/hal0 → ${VENV_DIR}/bin/hal0 and the comment claims it "survives upgrades because it points at the venv shim, not a copy" — but install.sh is the only writer, and hal0 update never re-asserts it.

hal0 already solved the identical class for the sudo wrappers: src/hal0/updater/updater.py:3963-4096 refresh_privileged_wrappers(), called from updater.py:4159-4163. ODS's equivalent is installers/phases/13-summary.sh:270-300, which ln -sfs the CLI into the install dir on every install run and falls back to ~/.local/bin with a PATH warning when sudo is unavailable.

Fix: add refresh_path_links(target) next to refresh_privileged_wrappers and call it from the same activation step. ~30 lines.

5. #1550 — deploy.sh reports success while the backend serves old code (S / H)

scripts/deploy.sh:213-231 polls /api/status for HTTP 200 and then prints deploy complete @ <sha> — the sha comes from git rev-parse, i.e. from the tree, never from the process.

ODS gates the success line on a health verdict and rolls back otherwise: ods-update.sh:378-406 wait_for_healthy() (deadline loop, per-attempt remaining-time log, dumps the final failing status), ods-update.sh:717-724 calls _update_rollback when it fails, and only :725-740 records the new version. ods-cli:2059-2113 does the per-service variant: enumerate the stack, check each container's State.Status/ExitCode, and exit 1 with "Run 'ods rollback'".

Fix: expose the running build's version/commit on /api/status (or read hal0.__version__ from the live process) and compare it to git rev-parse HEAD before printing success.

6. #2216 — LXC with /dev/kfd forwarded seeds every slot gpu-vulkan (S / H)

src/hal0/install/profile_derive.py:110-113:

rocm_ok = any(g.compute_capable for g in hw.gpus) or kfd_present()
if rocm_ok and (hw.platform == "strix-halo" or any(g.compute_capable for g in hw.gpus)):
    return "gpu-rocm"

kfd_present() can satisfy the first clause but never the second, and compute_capable is set from a rocm-smi/rocminfo exit code (src/hal0/hardware/probe.py:483-484). So a container with /dev/kfd forwarded but no ROCm userspace installed falls through to Vulkan.

ODS decides the AMD lane from device nodes only — installers/lib/detection.sh:187-210 (amd_gpu_missing_runtime_devices checks /dev/kfd is a char device and /dev/dri/renderD* exists) and installers/phases/02-detection.sh:159-168 gates the whole lane on that, with show_amd_gpu_device_guidance (detection.sh:218-233) printing the LXC passthrough commands when it fails.

Fix: make both device-node predicates together sufficient for the ROCm branch — kfd_present() and render_node_present() — and drop the rocm-smi dependency. Correction (2026-10-04): the first draft said kfd_present() alone; that loses the second half of the ODS check (renderD* must exist, detection.sh:200-204), and a Strix Halo LXC with only /dev/kfd forwarded would then seed every slot gpu-rocm and launch it without a render device. hal0 already has both helpers (src/hal0/providers/_gpu.py:251-280 kfd_present, :397-420 render_node_present), each stricter than ODS's because they check openability by the slot runner uid, so this is still a one-expression predicate change.

7. #2028 — /api/services/health is a static three-branch construction (M / H)

src/hal0/api/routes/services_health.py:198-307 hand-writes comfyui, hermes and openwebui as three literal blocks; the docstring at :1-15 admits the list is fixed. The dashboard footer's readiness count therefore cannot move on a hindsight-api outage.

ODS's dashboard-api is manifest-driven end to end: extensions/services/dashboard-api/config.py:297-424 load_extension_manifests() reads every extensions/services/*/manifest.yaml (schema-pinned to ods.services.v1) into SERVICES/FEATURES at config.py:430-448; helpers.py:624-679 check_service_health() probes one service from its manifest port + health path with per-service timeouts and a healthy/degraded/down/not_deployed verdict; and helpers.py:682-700 get_all_services() fans out with asyncio.gather(..., return_exceptions=True) so one misbehaving service can't take the response down.

Fix: add a services.yaml-style table (or extend the existing slot/profile manifests) carrying id, name, unit, probe_url, health_path, timeout, and replace the three blocks with one loop. New services then appear by adding a row.

8. #1868 — static slot seeds are copied verbatim with no hardware budget (M / H)

installer/etc-hal0/slots/brain.toml ends with a literal [model] context_size = 65536, copied unchanged onto every box by src/hal0/install/static_seeds.py:195-260. That module already re-derives device per host through LLAMA_SEED_CAPABILITIES (static_seeds.py:74-82) → derive_device; the context window is the one field it doesn't touch.

ODS never lets a context window escape the hardware envelope: installers/lib/tier-map.sh:200-330 binds MAX_CONTEXT to the tier (which is derived from detected VRAM/RAM), and scripts/select-model.py:170-182 (estimated_context_kv_gb / selector_required_memory_gb) charges the KV cache against the model's memory requirement, with usable_memory_gb (select-model.py:134-144) giving the unified-memory box only 55% of RAM.

Fix: extend static_seeds.py's derivation pass with a _CONTEXT_LINE rewrite that clamps context_size to a tier-derived ceiling — the same shape, the same file, one more regex.

9. #1862 — a stale or missing hardware.json books VRAM as RAM (M / H)

src/hal0/slots/capacity.py:499-522 _host_has_capable_gpu() reads the cached /etc/hal0/hardware.json and returns False on any error, and build_per_slot (capacity.py:570-585) uses that bool to decide whether resident memory is charged to VRAM or RAM.

ODS has no durable hardware cache to go stale. CAPABILITY_PROFILE_FILE defaults to ${TMPDIR:-/tmp}/ods-capabilities.json (installers/lib/constants.sh:30), and it is rebuilt on every run by scripts/build-capability-profile.sh — from the installer (installers/lib/detection.sh:26-46) and from the doctor (scripts/ods-doctor.sh:81-85).

Fix: when the cache is missing or older than N hours, fall through to hal0.hardware.probe's live light probe rather than to False. hal0 already has that probe; only the fallback branch is wrong.

10. #1936 + #1966 — no GPU devices / no CPU lane (S–M / H)

#1936 is a segfault at model load when the container has zero GPU devices mapped; #1966 is ComfyUI offering only gpu-vulkan on a kfd-less AMD box.

Two ODS shapes cover both. installers/lib/detection.sh:235-257 apply_cpu_gpu_fallback() is a single function that rewrites the whole hardware verdict (GPU_BACKEND=cpu, GPU_VRAM=0, HAS_NPU=false, overlays cleared) and prints the reason; it is invoked from four guard sites (installers/phases/02-detection.sh:132,168,181, 11-services.sh:311). And extensions/services/comfyui/manifest.yaml:19 declares gpu_backends: [amd, nvidia], repeated on the feature at :38 — so the dashboard can only offer a lane the service declares.

Fix: (a) refuse to spawn a GPU-lane container when the device nodes are absent, with a named reason instead of a SIGSEGV; (b) put supported_backends on the ComfyUI runner in the profile manifest and have the device picker render "requires /dev/kfd" instead of an unusable row.

11. #1870 — lifecycle verbs wait 966s with no progress (M / H)

src/hal0/cli/slot_commands.py:51-54 derives the client budget from slot_lifecycle_budget.slot_lifecycle_timeout_s() — correct, but the CLI then blocks on one api_post for the whole window with nothing on stdout.

ODS's two halves: installers/phases/12-health.sh:66-106 _check_container_health() prints ... Waiting for <name> and rewrites the line in place with \r each poll, short-circuiting on exited|dead|missing; and installers/lib/background-tasks.sh:119-148 bg_task_wait() is the bounded elapsed/timeout loop with three distinct exits (ok / failed / timeout). installers/lib/readiness-summary.sh:52-127 is the closing report: Ready now: N/total, then Ready: / Needs attention: / Next: with the log paths.

Fix: stream the server's phase transitions (the state machine already stamps them) over the existing SSE seam and render a per-phase progress line; keep the derived budget as the hard deadline.

12. #2096 — update should restart slots when the runner image rolls (S / H)

hal0 makes this opt-in: src/hal0/cli/update_commands.py:516-531 explicitly says "we never bounce automatically" and tells the operator to run hal0 update --restart-slots.

ODS makes recreate the default and pinning the opt-out: ods-cli:2049 runs up -d --force-recreate on every update, while ods-cli:1601 uses --force-recreate --no-build --pull never for a plain restart so a restart can never silently change the image. The image set itself is derived from the resolved plan, not per-code-path (installers/lib/compose-images.sh:40-71).

Fix: invert the default when — and only when — /api/updates/slot-drift reports an image change; keep --no-restart-slots for the operator who wants the old behaviour.

13. #1512 — stack import commit never verifies the envelope checksum (S / H, Direct port)

ods-restore.sh:273-310 is nearly transliterable: it looks for checksums.sha256, picks sha256sum -c or shasum -a 256 -c, returns 1 on mismatch with "This backup may be corrupted or tampered with", warns loudly when the file is absent (older format), and only skips when --skip-verify was passed explicitly (:308-310). It also warns when the archive is missing expected data paths (:313-320).

Fix: move the digest check hal0 currently runs only in dry_run onto the commit path, fail closed, and add an explicit --skip-verify.

14. #1974 — _flm_image_present caches False and never re-probes (S / M, Direct port)

src/hal0/capabilities/catalog.py:178-235 caches the probe at module scope and clears it only after a successful FLM pull (reset_flm_image_present_cache) — so a transiently broken podman permanently drops the NPU backend from /api/capabilities.

ODS's _DirSizeCache (extensions/services/dashboard-api/helpers.py:26-56) is the 30-line answer: per-key TTL, expiry sweep on write, bounded size, explicit invalidate.

Fix: cache the positive result indefinitely and the negative one for ~60s, or don't cache the "no runtime / probe failed" branch at all — it is a different answer from "image absent".

15. #1820 — the netavark port black hole has no repair owner (M / H)

hal0 has both halves already and no wiring: detection in src/hal0/system/netavark.py:58-170 (parses nft -a list table inet netavark) and the privileged repair in src/hal0/system/seam.py:363-413 prune_dnat() (with a dry_run/check-dnat mode). The reconciler parks the slot in ERROR instead of calling it.

ODS's _compose_run_with_summary (ods-cli:960-1140) is the template for "known-recoverable failure → repair → bounded retry": it classifies the compose log (ods-cli:1006 network errors, :1021 port-already-allocated, :1042 stale container name conflict), performs the matching repair (_compose_remove_conflicting_ods_containers, ods-cli:978-996 — which refuses to remove a non-ODS container), and retries under per-class attempt counters and backoff (:996-1060). ods-cli:6115-6204 cmd_repair is the operator-facing companion verb.

Fix: on the ERROR transition, run prune_dnat(dry_run=True); if it identifies a stale rule, prune and retry the start once, then park with the evidence.


4. Clusters — one change, many issues

A. A progress / timeout / evidence seam

Closes or de-risks: #1870, #1869, #1873, #2221, #1550, #1820, #2180.

ODS treats "a long privileged call" as one shape with four obligations: a hard bound (ods-uninstall.sh:321, bootstrap-upgrade.sh:939, ods-cli:2166-2172), visible progress while waiting (12-health.sh:66-106, background-tasks.sh:119-148), a classified retry for known-recoverable failures (ods-cli:996-1060), and a persisted evidence report when it finally fails (compose-failure-report.sh:92-186 — redacted env, port occupancy, compose ps, log tail — plus readiness-summary.sh:52-127).

In hal0 that is one module: a bounded_call() helper every seam invocation goes through (container.py:2549, seam.py:415), a phase-progress event on the existing SSE bus, a retry classifier for the netavark/name-conflict/port classes, and a write_slot_failure_report() that the crash-line extractor feeds instead of replacing. Doing this once removes the need to hand-tune slot_lifecycle_budget.py's EVICTION_UNLOAD_ALLOWANCE caveat (src/hal0/slot_lifecycle_budget.py:95-107 explicitly names #1869 as the real fix).

B. Manifest-driven service + capability registry

Closes or de-risks: #2028, #2192, #1966, #1859, #2190, #2184, #1990, #1823, #2234, #1947, #1319.

Every one of these is "a fact about a service/runtime is hard-coded in one code path and the other paths can't see it." ODS answers all of them with one mechanism: extensions/services/<id>/manifest.yaml validated by extensions/schema/service-manifest.v1.json, loaded once by lib/service-registry.sh:29-48 (bash) and extensions/services/dashboard-api/config.py:297-431 (Python), and consumed by health (helpers.py:624-700), features (routers/features.py:19-113) and the CLI. Backend compatibility is a manifest field (comfyui/manifest.yaml:19); model capability is a registry field (config/model-library.json); the image set is derived from the resolved plan (compose-images.sh:40-71).

hal0 has the pieces scattered — RUNNER_IMAGES, seed profiles, _SLOT_META, services_health.py's three literals, capabilities/catalog.py. One hal0.services.registry loaded from a schema-validated table, read by the health route, the Omni eligibility check, the device picker and the context resolver, would retire the whole cluster.

C. Hardware re-detect + honest CPU fallback

Closes or de-risks: #1862, #2216, #1936, #1966, #1868, #1834, #1867, #2201, #1928, #1929.

ODS makes detection cheap and always fresh (tmp-file capability profile rebuilt every run, constants.sh:30 + detection.sh:26-46 + ods-doctor.sh:81-85), decides GPU lanes from device nodes rather than userspace tools (detection.sh:187-210), has exactly one function that demotes a box to CPU with a stated reason (detection.sh:235-257), and derives every memory/context number from the resulting envelope (tier-map.sh:200-330, select-model.py:134-182, llama-memory-budget.sh:6-31).

In hal0 this is: (1) live-probe fallback when hardware.json is stale/absent; (2) kfd_present() and render_node_present() together sufficient for the ROCm lane; (3) one apply_cpu_fallback() that rewrites the whole verdict; (4) one memory-envelope function that seed context sizes, the memory ruler, the extraction sizing and the anchor-window floor all read.

D. Update / deploy delivery integrity

Closes or de-risks: #1844, #2019, #1550, #2096, #1845, #1519, #1512, #1530, #2101.

ODS's update is a transaction: snapshot → change → verify → record version, with auto-rollback at every step (ods-update.sh:647-742, _update_rollback at :412-449, wait_for_healthy at :378-406), the installed artefacts re-asserted on every run (13-summary.sh:270-300), verification of the running stack before success (ods-cli:2059-2113), and "latest" resolved from the release API so tagging is delivery (ods-update.sh:456-505). Uninstall tracks per-step residue and names the remedy (ods-uninstall.sh:362,394-402).

hal0 already has the hardest parts (cosign, staged trees, atomic symlink swap, refresh_privileged_wrappers). What's missing is the cheap discipline: re-assert the PATH link, verify the served build, verify digests on commit, gate the success line on per-step outcomes, and let scripts/deploy.sh share the update path's refresh steps.

E. Schema-driven settings and previews

Closes or de-risks: #2108, #1967, #2195, #2203, #1511, #1445.

extensions/services/dashboard-api/settings.py:194-241 _build_env_fields() renders a UI field for every key in .env.schema.json — label, type, enum, secret flag, read-only reason, default — and falls through to a generic string field for keys the schema doesn't describe. A new config key gets a UI path by existing, not by someone hand-writing a control. The apply side is a declared allow-list per service (settings.py:33-58). The preview side is ods-cli:6261-6324 _template_preview: compute the delta, print "Already enabled / Will enable / Nothing to change", and keep apply additive.

For hal0 that means one ChangeSet diff function shared by preview and apply (so removals can't go unreported), and a schema-derived settings renderer so tool_model and its siblings stop needing bespoke UI.


5. Where ODS has nothing to offer

Called out so nobody goes looking:

  • Memory / Hindsight internals (#2016, #2017, #1989, #1990-partial, #1993-#1997, #2155): ODS's RAG is Qdrant + TEI behind Open WebUI's own Knowledge surface, and its "memory-shepherd" is a MEMORY.md file-reset tool on a systemd timer — a different problem entirely. Only the hygiene/retention shape transfers (#2154, #1521).
  • CI throughput (#2169, #1502): ODS's workflows have no path gating, no concurrency groups and no test tiering (/home/user/ods/.github/workflows/lint-python.yml:3-8). hal0's CI is the more mature of the two.
  • Changelog fragments (#1545): ODS uses a plain Keep-a-Changelog file too. The only transferable observation is that its PR template (/home/user/ods/.github/pull_request_template.md) has no changelog checkbox — notes are written at cut time, so PRs never touch the file.
  • Polish sweeps (#1477, #1524-#1529, #1536, #1840, #1841, #1783, #1552): item-level UI/CLI work with no structural analogue.
  • Upstream tracking (#1249, #1948, #1829, #2118, #1070, #1349, #1925): pins, builds and third-party bugs.