Verdict
The premise was half right, and the half that was wrong matters more. ODS is ahead where the product meets the operator: installer structure and failure forensics, the two-file extension model, auth as a product feature, a phone-first agent portal, persona craft, AMD host tuning, consequence-first copy, bootstrap-then-hot-swap, and a fleet-verified model catalog. hal0 is ahead in the engine room: update integrity, agent provisioning rigor, MCP, memory architecture, the container runtime, diagnosis modelling, the multi-slot inference model, the CLI's architecture, and, surprisingly, mDNS.
That split is good news. Almost everything worth taking from ODS is additive: shapes and discipline layered onto mechanisms hal0 already has. Very little of it is a rewrite, and the few rewrites on offer (compose, the monolithic CLI, the git-pull updater) would be regressions.
Three premise corrections
ODS ships no MCP at all.
No mcp_servers block in the Hermes template, no MCP for OpenCode or Open WebUI, no MCP UI or CLI verb. The n8n "RAG pipelines" are sticky-note stubs. There is nothing to port on this axis. Meanwhile hal0 already has most of the "user-added MCP servers" feature you asked for: a per-server TOML registry at /etc/hal0/mcp-servers/, a 945-line REST router, hal0 mcp list|install|uninstall, and an SSRF-guarded manifest resolver. Three gaps remain: no process supervisor (the route literally says "pending ADR-0015"), no join between installed servers and the Hermes config, and no install drawer in the dashboard. Update 2026-10-04: two of the three closed on main — #2253 (ADR-0015) shipped the exposure join, [secrets]/[tools]/[exposure] and the CLI/REST verbs for http/sse servers; the stdio supervisor is deferred and the drawer is #2245. Review also found that the shipped join mirrors the tool policy without enforcing it on Hermes's call path — see report 06 §E.3 and hal0 issue #2303.
hal0 already has mDNS.
src/hal0/services/mdns.py writes avahi service-group XML, is wired to a dashboard card, and is tested. ODS runs its own zeroconf daemon plus a Caddy host-router on :80 to get chat.ods.local. hal0's approach is lighter for what it does. What hal0 lacks is not mDNS, it is the single friendly URL and a safe LAN posture, an address publisher if subdomain URLs are wanted (avahi service files advertise _http._tcp records on the machine's hostname; they do not create chat.<host>.local A-records, which is exactly what ODS's bin/ods-mdns.py exists for), and one dead path: the base hal0.service avahi file the dashboard promises is never written.
ODS's Hermes is packaging; the product around it is the prize.
ODS runs the pinned upstream container with five files of glue and a regex YAML patcher for upgrades. hal0's provisioner is more rigorous (hermes config migrate + config set, converging writes, a backup, an operator overrides file), its context-floor logic reads the floor from the installed Hermes instead of hard-coding 65536, and its systemd sandbox is tighter than the container. What ODS does better is everything an operator touches: magic-link auth with owner cards, a mobile chat portal whose pooled WebSocket bridge keeps the prompt cache warm, agent-driven OAuth setup, and a persona written from live testing. Keep pip-installed Hermes. Copy the product layer.
Two findings that should not wait for a plan
A. Approval endpoints execute gated tools with auth removed.
/api/agent/approvals/{id}/approve runs model_pull, slot_delete and config_write and its module explicitly notes auth was removed (src/hal0/api/routes/approvals.py:24-27), on an API that binds 0.0.0.0:8080 by default. ODS's session_signer.py (171 lines, stdlib only) plus its verify-session/admin-session routes are a near-verbatim fix, and hal0 already has half the machinery in api/agents/_auth.py. About 250 lines. Do this before anything else in this document. Update 2026-10-04: addressed on main for the LAN-bound case by #2246 (946c512) — ADMIN routes, approvals included, now require an admin session/key from any non-loopback caller when the box is LAN-bound, even with require_auth off. The loopback dev-open path remains by design.
B. hal0-Max recommends a model ODS found broken on unified memory.
hal0's top first-run bundle for the 100 GB Strix Halo class sets chat.coder = Qwen3-Coder-Next-80B-A3B (curated.py:260-275). ODS's fleet testing found that exact family produces all-? output on unified-memory backends, reproduced on Strix Halo and on Grace-Blackwell, and now hard-routes every unified-memory host away from it (installers/lib/tier-map.sh:100-138, scripts/select-model.py:326-372). ODS tested against llama.cpp b9014 and its own Lemonade path, so this may not reproduce on hal0's ROCm runner. It costs one afternoon on a real 395 to find out and it decides whether the flagship pick is a correctness bug.
A third, smaller item: the sandbox install surfaced a real ODS bug (the host agent inherits the installer's lock file descriptor in the non-systemd fallback path, so later model operations block forever). Details in the install section. Since permission flows both ways, it is worth sending upstream.
Where hal0 falls short
The honest list, in the order I would fix it. Each item names the evidence; each maps to a row in the port ledger.
- Mutation endpoints run unauthenticated on a LAN-open bind. Approvals execute gated tools; auth is off by default and the doctor's own posture check prints "PASS (dev/loopback)" without reading the bind address (
cli/doctor_all.py:62-83, issue #1822). - The installer is one 4,002-line script with no persisted log. No step files, no tee'd transcript, no failure report, no summary JSON, no dry run, no per-step contracts. ODS has all six; that is why it can unit-test detection and tier mapping and hal0 cannot.
- There is no extension model. The services registry is a hard-coded four-entry tuple,
hal0 appsupports exactlyopenwebui, and the UI already lists n8n with an icon and no backend. ODS gets a working service from two files and no code edits. - Remediation is designed but empty.
Diagnosis.next_stepsand a panel that renders them exist, yethealth_report.to_diagnosis()hard-codesnext_steps=[]for every check. Issues #1845, #1870, #1994 and #2108 are all this gap wearing different hats. - Long privileged calls are unbounded and silent.
ContainerProvider._rundefaultstimeout=Noneand the load path passes nothing; the CLI blocks up to ~2,400 s with no output. ODS bounds every hang-prone call and prints while it waits. - Hardware truth goes stale. A missing or old
hardware.jsonbooks VRAM as RAM; the ROCm lane depends onrocm-smibeing installed rather than on/dev/kfdexisting; seed slots copycontext_size = 65536onto every box. ODS rebuilds its capability profile every run and derives every number from it. - No AMD host tuning. GTT/TTM sizing, swappiness, and a tuned profile are absent from hal0's installer even though hal0's own upstream toolbox README documents them as required for Strix Halo. Update 2026-10-04: the IOMMU half of that README (
amd_iommu=off) is not for hal0 to port — it removes/dev/accel(the XDNA NPU) and hal0's own guidance isiommu=pt amd_iommu=on(drivers.mdx:102, ADR-0003). The gap is GTT/TTM, sysctl and tuned — and on kernels ≥ 6.14 even GTT pinning is an opt-in floor, not a default: amdgpu sizes the pool dynamically and hal0 reads the live value per request (drivers.mdx:86-97), see D7. - Open WebUI is half wired. hal0 sets chat and voice, but not RAG embeddings, web search or image generation, so Open WebUI's own upload and image buttons point at nothing. ODS wires all of it, including a baked ComfyUI workflow.
- The copy explains internals, not consequences. Tooltips read like
hal0.toml [memory] · requires restart. ODS's status vocabulary and settings hints lead with what happens to you. This is an audience choice, but it should be a choice. - Own docs have drifted. The "15-phase, checkpointed" provisioner is 12 steps and deliberately uncheckpointed;
BUNDLED_AGENTSis misquoted; a unit'sDocumentation=path does not exist; two byte-identical 629-line memory providers have no sync;CLAUDE.mdsaysdocs/adr/holds the decision records whileARCHITECTURE.mdsays hal0 keeps no ADR tree, and the ADR numbers the code cites (0012, 0015, 0023) live in the gitignored internal tree perCHANGELOG.md, so a contributor readingroutes/mcp.pycannot find the decision it defers to. Fix these before anyone ports against them. (ADR bookkeeping reconciled on main by #2244:docs/adr/now holds 0004, 0012, 0013, 0015, 0020 and 0023.)
How ODS does it, domain by domain
Mechanism first, then what to copy and what to skip. The full reports with every citation are in the appendices; this is the distilled version.
ODS runs nousresearch/hermes-agent:v2026.6.5 as a container that exposes no host port. A Caddy sidecar (hermes-proxy) fronts it with forward_auth to dashboard-api/api/auth/verify-session, which HMAC-verifies a stateless cookie of the form <id>.<expiry>.<sig>. Cookies come from magic links (single-use guest, revoke-only owner cards, hashed at rest, rate-limited, QR-rendered) or from POST /api/auth/admin-session so the owner is never locked out. Config is a bind-mounted cli-config.yaml template that encodes hard-won settings: a schema-level disabled_toolsets: [terminal, browser] because small models reach for a shell on weather questions; max_tokens: 1024 so a looping model cannot pin the only slot; compression tuned to 0.75 / 0.50 / 40 after a single web-search result kept triggering it. The persona (SOUL.md) gets a generated "About this installation" block built from docker ps and a live model probe. A per-cookie WebSocket pool in hermes_bridge.py keeps one Hermes session per phone so llama-server's KV cache stays warm across messages, with exactly one retry, allowed only before prompt.submit lands.
hal0 runs Hermes as a pip-installed wheel under a sandboxed hal0-agent@.service (Type=notify, watchdog, ProtectSystem=strict), provisions it with converging writes through Hermes's own config migrate and config set, reads the 64K context floor from the installed package instead of hard-coding it, injects a lean STATE.md refreshed on slot swaps, and ships real Hermes plugins for provider discovery and Hindsight memory. Its policy layer is on the call path; ODS's APE policy engine is attached to nothing.
Copy
session_signer.py+routers/auth.py→ generalizeapi/agents/_auth.py; gate approvals first.- The three non-obvious Caddyfile lines: explicit
route {}wrapper, stripSec-Websocket-*/Upgrade/Connectionfrom the auth sub-request,redir * /auth/required 303. oauth_passthrough.py+oauth-providers.json+ the SOUL script that teaches the agent to drive OAuth setup.- The pooled-bridge discipline (same socket for create and submit; retry only before submit) if hal0 ever adds a REST/SSE chat surface.
- Tool-name → spinner caption table with the honest fallback
Using `<name>`…. - The generated-config-writers matrix (
docs/INSTALLER-ARCHITECTURE.md:118-135) as a section inARCHITECTURE.md. - The persona voice rules as a persona option, not a replacement for the admin SOUL.
Skip
- APE as a service: wired to nothing; its allowed paths still name an OpenClaw workspace. Take its algorithms only.
agents/templates/*.yaml: OpenClaw-era, unreferenced.memory-shepherdas shipped: a MEMORY.md reset timer pointed at a retired agent.- The regex YAML patcher as the primary config path (keep it only for removing stale blocks).
- Hard-coded
HERMES_MIN_CONTEXTand auto-raising the box's context; hal0's read-it-from-Hermes posture is correct. HERMES_DASHBOARD_INSECURE=1and scraping the session token out of HTML.
Your description is accurate. A working ODS extension is extensions/services/<id>/manifest.yaml (20 to 30 lines) plus compose.yaml (30 to 50 lines) and zero edits anywhere else; searxng ships as 22 + 33 lines. From those two files, fourteen consumers pick it up: the bash service registry, the compose resolver, ods list|enable|disable|logs|shell, the generic health check, the dashboard's services and features loaders, sidebar quicklinks, feature cards, setup hooks, CI audits, the catalog generator, presets, and the host agent. Four independent walkers each re-read the directory, all gated on schema_version: ods.services.v1; that redundancy is why it feels magical. GPU and mode overlays (compose.{nvidia,amd,local}.yaml) are picked up by filename probe, not by manifest declaration. Pre-wiring is four conventions: Docker DNS equals the compose service name, the LLM is the gateway alias ods/current, ports bind loopback with an env override, data lives under ./data/<id>/.
Two honest seams: a hand-maintained 21-line ENABLE_* table in phase 03 (the file's own comment says omission is a bug), and enable/disable implemented as renaming compose.yaml to .disabled, which makes the content tree the state store. hal0 has every ingredient (a typed launch plan, a validated quadlet writer, a services registry with the right fields, a portable envelope format, a port authority, a registry-driven Services page) and none of the wiring. The hard constraint is the root-side allow-list in installer/wrappers/hal0-systemctl:389-516, which pins units to the hal0-slot@ prefix by design.
Copy (translated to podman/systemd)
- Manifest schema → Pydantic
manifest.tomlwithschema_version = "hal0.ext.v1"; keep field names that carry meaning so ODS manifests convert mechanically. - The registry loader (
service-registry.shis already a Python program embedded in bash) →hal0/extensions/registry.py. calculate_feature_statusand its vocabularyenabled | available | insufficient_vram | services_needed.audit-extensions.pychecks →hal0 app doctor.- The heavily commented template and the swap-safe doc; they are a large part of why the model is usable.
- Render:
manifest.toml → RuntimeLaunchPlan → _render_quadlet_from_plan → hal0-ext@<id>.container; env written to/etc/hal0/extensions/<id>.envwithOPENAI_BASE_URL,OPENAI_MODEL=hal0/brain, andslot:<name>references resolved through the port authority.
Skip
- Rename-the-file as state; use a state file with
SlotConfigStore's commit/revert shape. - The hand-maintained
ENABLE_*table; derive toggles from category plus state. - Two parallel status vocabularies and a committed generated catalog JSON.
config/ports.json; hal0's port authority is strictly better.- Accepting any author-written unit body; render from a typed manifest, always.
install-core.sh is a 363-line orchestrator over 14 phase files and 19 pure libraries, every file carrying a Purpose / Expects / Provides / Modder notes header, with a contracts document giving each phase Owns / Inputs / Outputs / Idempotency / Failure modes and a written reinstall contract. The narrator helpers (ai, ai_ok, ai_warn, signal) all tee to a log file; on compose failure a redacted install-report-*.txt captures the failed image, port occupancy with owning PID, and the compose config, and ods doctor mines it later. Image pulls retry with a backoff table, validate after pull, and bail immediately on non-retryable errors. A fixture-driven preflight engine emits JSON ({id, status, message, action}) and six named scenarios pin its verdicts in CI. The signature move is bootstrap mode: a 1.2 GiB pinned model boots the stack in about two minutes while a detached upgrader downloads the real model with script-owned resume, snapshots every file it will touch (recording absence with a .missing sentinel), force-recreates, asserts the container's --model argument actually changed, and rolls back through a proven completion on any of four failure paths, deleting the small model only after HOT_SWAP_VERIFIED=true.
hal0 is ahead on the parts that protect a fleet: a cosign-verified, staged, atomic-symlink updater with venv rollback (ODS does git pull), typed Diagnosis/Evidence/NextStep with a frozen ID taxonomy, and a preflight that classifies LXC privilege and GPU gid problems ODS's threshold checker never sees. hal0 already has a support bundle, a QR code and a summary card; those are not gaps.
Copy
- The module header convention, first, because it makes the decomposition reviewable.
- Tee every narrator line to
${VAR_DIR}/logs/install-<ts>.log; write a failure report from the ERR trap; havehal0 doctor bundleglob both. - The 25-line progress protocol (
HAL0_PROGRESS:<pct>:<step>:<msg>, no-op unless a flag is set). - Pull retry: backoff table, post-pull
podman image inspect, non-retryable classifier. hal0's only pull today is(podman pull … || true) &. - Preflight JSON using
Diagnosisfield names plus a fixture corpus; reserveHAL0-PREFLIGHT-*IDs. --summary-json, per-step time estimates, and later--dry-runplus a simulate harness.- Bootstrap-then-hot-swap as five ideas (status file, snapshot with
.missing, verified-before-cleanup, lock, transientsystemd-rununit), not as a file.
Skip
ods-update.sh's mechanism: unsignedgit pullinto the live tree.- The CRT theme, lore lines, and
type_line's per-character sleep. bootstrap-upgrade.shas an artifact (3,412 lines, three platforms interleaved, noset -e).- The Tauri GUI and per-platform phase duplication.
- The ADR that retains
:latestfor three images; keep digest pins.
config/model-library.json (56 entries) carries per-consumer fleet-test provenance: app_compatibility.{openai_chat, hermes_talk, perplexica, …} with status, reason, evidence path, host scope and product SHA. select-model.py is a deterministic offline selector that charges an estimated KV footprint on top of file size against a memory envelope (55 % of RAM on unified memory), and encodes the unified-memory exclusion above. Activation goes through the host agent's transaction: snapshot every file it touches, swap, then require the previous model to re-prove identity and a completion before a rollback counts as successful. A small model-router service gives every app one stable alias, ods/current, that survives a swap; it exists alongside LiteLLM because LiteLLM's alias map is static YAML.
hal0's shape is ahead: N independent slots, a real registry mid-cutover to SQLite, GGUF header parsing for capabilities, a live amdgpu sysfs GTT-fit check at request time, first-class mmproj two-file pulls, and HF update checks. ODS's local models are text-only.
Copy
- The unified-memory exclusion as knowledge (verify first; see finding B).
- The
app_compatibilityschema idea as an optional field; populating it is a separate effort. - Install-time re-verify of an existing file against its pinned SHA, with auto re-pull.
- The rollback bar: previous model re-proves before rollback is "successful".
- The stable-alias idea, generalizing
hal0/braintohal0/chatonly after a design pass on the dispatcher.
Skip
pre-download.sh(stale tier vocabulary, no SHA).- The Switchboard doc as a description of current truth.
- The single-process, swap-the-whole-model runtime shape.
- Static memory heuristics where hal0 has live telemetry.
Two layers. bin/ods-mdns.py is a zeroconf daemon under systemd that publishes <device>.local plus chat., dashboard., auth., api., hermes. and talk. A-records and re-reads .env every 30 s; a static AST test pins that subdomain set to the Caddyfile routes and the magic-link redirect targets so the three cannot drift. ods-proxy is Caddy on :80 doing host-based routing (path-based broke Open WebUI's assets, websockets and OAuth callbacks). It is the one service allowed to bind 0.0.0.0; everything else stays on loopback unless --lan. The cookie is Domain=<device>.local-scoped so one redemption authenticates every subdomain. Onboarding is appliance-grade: a printable 4×6 setup card with a Wi-Fi QR and a setup-URL QR, a terminal QR fallback, and a disabled-by-default AP mode that ODS itself calls destructive.
hal0's control plane and Open WebUI bind 0.0.0.0 by default with auth off; inference slots stay loopback. A full three-tier auth layer exists (api/auth.py, security/exposure.py) but ships off after an earlier auto-enable locked operators out of a dashboard that had no login UI. The doctor's posture check never reads the bind address (issue #1822, pinned to doctor_all.py:75).
Copy
- The doctor cross-check (bind non-loopback and auth off → WARN with the remedy). About 20 lines. Highest leverage per line in this study.
- Write the base avahi file the dashboard already claims exists.
- Optionally a single-port Caddy gateway from the
chat.<device>.localblock, if the LAN decision goes that way. - A
hal0 setup-cardreusing the existing LAN-IP detection and QR primitive.
Skip
- The zeroconf daemon; avahi XML is lighter for hal0's shape.
- The cookie-domain trick without subdomain routing.
- AP mode unless hal0 becomes a sealed appliance.
remote-provider-*as a remote-access model; they are outbound egress boundaries for cloud LLM keys.
ODS has no MCP. Its memory is upstream Hermes's own files plus memory-shepherd, a 338-line timer that archives everything below a --- separator in a markdown file every three hours. Qdrant and TEI are deployed and the README says vectors land in Qdrant; nothing writes to it. The one genuinely excellent pattern here is generated-config discipline: patch-hermes-config.py is a block-scoped, comment-preserving patcher that distinguishes "operator wins" keys (fill only if missing) from "ODS converges" keys (re-assert unconditionally), returns changed: bool, and writes only on diff. APE's approval grants are bound to an args fingerprint and are strictly one-shot; that idea is worth stealing even though APE itself is not.
The user-added-MCP design in the report is concrete and reuses hal0's own pieces: extend the existing per-server TOML with [secrets] (indirection into /etc/hal0/api.env), [tools] (the existing ToolPolicy, so a new server starts with zero callable tools), and [exposure] (hermes, brain now; openwebui/opencode return 501 until built); regenerate Hermes's mcp_servers block on every registry mutation, deleting only entries hal0 itself wrote (main's #2253 keeps that ownership list in a side-car manifest rather than sentinel comments, since YAML comments do not survive a re-dump); add hal0 mcp test <id> that probes the server and prints each tool with its verdict; run stdio servers behind a client-attached bridge unit (hal0-mcp@<id> executing a hal0-owned stdio↔streamable-http adapter), never as a free-standing process with nobody on its pipe. Write ADR-0015 first; the code already promises it. Corrected 2026-10-04 (report 06 §E.2–E.3): the builtin loop's admin bearer must not be reused for third-party URLs; the [tools] mirror only describes policy — Hermes talks to the server directly and nothing enforces allow/gated/blocked until hal0 fronts user-installed servers with a policy proxy mount (issue #2303); and a stdio server needs a client-attached bridge, not a free-standing unit.
Copy
- The block-scoped patcher for one job: removing stale hal0-owned
mcp_serversblocks, whichhermes config setcannot express. - The writers table (see agents).
- Windowed caps plus a circuit breaker for gated tools, keyed on
(server, tool)not on guessed intent. - Persisted, args-fingerprinted one-shot approval grants; the queue is in-memory today and a restart drops pending approvals.
Skip
- APE as a deployed service and its advisory-by-default mode.
- Intent classification by verb tokens in the tool name.
- A vector DB with no consumer, documented as if it had one.
- A periodic destructive memory reset; Hindsight is the durable record.
ODS's nav is seven items and outcome-named. GET /api/features turns manifest data plus live health into first-person sentences ("Your hardware can run Document Q&A. Enable it?"), and GET /api/features/{id}/enable returns numbered steps with deep links that a modal renders with working buttons. Every extension status word has a one-sentence description; every settings save states in advance which containers it will recreate; the phone-first FirstBoot wizard takes four taps; the docs include a "friendly guide" that explains the mesh as an office building and an "if you know Ollama, ODS adds…" table. The installer, when it auto-disables something, prints the exact command to reverse that in the same breath.
hal0 is more rigorous where it counts (one classifier drives CLI and API; RestartApiPanel is a complete confirm → restart → poll → toast loop; ApplyBadge gives per-field restart guidance) and weaker where the user is stuck: next_steps is hard-coded empty, waits are silent, one screen shows two different "free memory" numbers, and the nav teaches the architecture ("Slots", "Runner Images", "Stacks") before it teaches what you can do.
Copy
- Populate
health_report.to_diagnosis()with realNextSteps (the CLI already knows the commands) and make the chips copy or run. Highest leverage in this domain. - The enable-steps modal shape for slot-load failures and convergence panels.
- Status word → one sentence, alongside the precise phase vocabulary.
- "Auto-derived X; reverse with Y" wherever hal0 picks a default silently.
- Full Open WebUI env wiring (RAG, search, image), gated on live slot state.
- The README "at a glance" tables and a post-install checklist page.
Skip
- Typing-effect theatrics and ideological lore copy.
- The flat 60-file ALL-CAPS docs directory; hal0's Starlight split is better.
- Client-side status re-derivation; keep the shared server classifier.
- The invites model unless multi-user becomes a feature.
ODS is Docker Compose with a formal layering contract (base + hardware overlay + mode overlay + extension fragments), resource limits and healthchecks on every service, an env schema validated in a five-tier CI matrix, and tag-only image pins with a written ADR accepting :latest for three images. Podman is tolerated only when it wears the Docker CLI mask. The ods-cli is one 6,595-line Bash file with a written but unexecuted decomposition plan and inconsistent JSON flags. Where ODS is clearly ahead is 10-amd-tuning.sh: render/video groups, amdkfd modprobe, swappiness and cache pressure, amdgpu ppfeaturemask, RAM-scaled GTT/TTM sizing (90 % of RAM at ≥ 96 GB → about 115 GB on a 128 GB box), amd_iommu=off on single-GPU APUs, a tuned accelerator-performance profile, and a reboot banner. hal0's own upstream toolbox README documents the same GTT/TTM GRUB parameters as required and hal0's installer applies none of them. (The README's amd_iommu=off is the one line hal0 should keep ignoring: it drops the NPU's /dev/accel and hal0 documents iommu=pt amd_iommu=on — see D7.)
hal0's runtime is the stronger design: Podman Quadlets with systemd's geometric restart backoff, a boot-time GPU device-group converge service, digest pins on every image, and Pydantic validation at load time. Its CLI is a modular Typer tree with issue-annotated mount points. Two source-confirmed defects match your complaints: unbounded silent lifecycle verbs (#1870/#1869) and hal0 update never re-linking /usr/local/bin/hal0 (#1844, which already caused #2092).
Copy
- The AMD tuning phase as
installer/lib/amd-tuning.sh, opt-in only (kernels ≥ 6.14 size GTT dynamically; see D7), GTT/TTM + sysctl + tuned, with the reboot banner. - gfx1151
HSA_OVERRIDE_GFX_VERSIONauto-detection without clobbering an operator value. - Known-bad kernel and firmware checks in
hal0 doctor(kernel < 6.18.4, the broken linux-firmware release), from the toolbox README. - Optional
MemoryMax=/CPUQuota=on slot quadlets, off by default. - A CI job that materializes each bundle tier's config and asserts it loads.
Skip
- The CLI monolith and the "everything is core" configurability model.
- Silently downgrading a system package from the installer; detect and warn instead.
- Tag-only pins and the
:latestacceptance. - Multi-platform installer sprawl.
Side-by-side matrix
| Concern | ODS | hal0 | Lead | Why |
|---|---|---|---|---|
| Agent runtime | Pinned upstream container, internal-only, Caddy forward_auth | pip wheel under sandboxed systemd unit, loopback, HMAC WS proxy | hal0 | Tighter sandbox; ODS needed HERMES_DASHBOARD_INSECURE=1 |
| Agent config writes | Template copied on first start + regex patcher on upgrade | hermes config migrate/set, converging, backup, operator overrides | hal0 | Works with upstream's config surface instead of around it |
| Agent context floor | Hard-coded 65536; installer raises box context | Read from installed Hermes; effective = min(model, slot); read-only diagnosis | hal0 | Correct by construction |
| Agent auth & sharing | Signed cookie, magic links, owner cards, admin-session | One HMAC seam on the chat proxy; approvals were unauthenticated (gated on LAN-bound boxes since #2246) | ODS | Largest single gap in hal0 |
| Mobile chat surface | ODS Talk: pooled WS bridge, SSE keepalive, tool captions, TTS | None | ODS | Best engineering in the ODS repo |
| Tool policy | APE engine, wired to nothing, advisory by default | Default-deny server+tool axes, load-time disjointness, approval inbox | hal0 | hal0's is on the call path |
| Extension model | manifest + compose fragment; 14 auto-attaching consumers | Hard-coded 4-entry registry; hal0 app supports one app | ODS | hal0 has ingredients, no wiring |
| Installer structure | 363-line orchestrator, 14 phases, 19 pure libs, contracts doc | One 4,002-line script, 16 steps, no lib/step split | ODS | Enables unit tests, dry-run, summary JSON |
| Install forensics | Tee'd log, failure report, preflight JSON, summary JSON | Terminal only; good ERR trap; support bundle exists | ODS | Transcript lost when the terminal scrolls |
| Preflight substance | Threshold checker, fixture-tested | LXC privilege classes, GPU gid remediation, soft/hard gates | hal0 | More capable, less provable |
| Updater | git pull + migrations + snapshot/rollback | cosign keyless, sha256, staged tree, atomic symlink, venv rollback | hal0 | Different class of integrity |
| First token after install | ~2 min via bootstrap model, hot-swap later with rollback | Blocking 15 to 31 GB decision; brain model pull never fatal | ODS | Best reliability idea in the repo |
| Model catalog | 56 entries with per-app fleet-test provenance | 41 curated, code-shipped, tool_calling flag + notes | ODS | Evidence vs. assertion |
| Inference model | One llama-server, swap the whole model | N slots, capability cards, NPU trio, GPU arbiter | hal0 | Architecturally ahead |
| Memory fit | Static envelope at install | Live amdgpu sysfs GTT verdict per request | hal0 | Truth over estimate |
| Swap rollback bar | Previous model must re-prove before rollback counts | State-machine transition back | ODS | Adopt the bar |
| Name → IP | Own zeroconf daemon, subdomain A-records | avahi service-group XML, system daemon | hal0 | Lighter, same outcome for hal0's shape |
| Port → service | Caddy :80 host router, one URL | Each service its own port; links follow request Host | ODS | UX; hal0 needs no proxy for raw use |
| Default bind | Loopback; proxy is the sole LAN surface; --lan opt-in | Control plane and chat UI 0.0.0.0; slots loopback | ODS | Safer default |
| Headless onboarding | AP mode, setup card, terminal QR, FirstBoot wizard | Terminal QR at install | ODS | Appliance framing vs. operator framing |
| MCP | None | 2 hosted servers, registry, REST, CLI, resolver, policy | hal0 | Finish it, don't start it |
| Memory | Hermes files + markdown reset timer + orphaned Qdrant | Provider ABC, Hindsight, closed namespace grammar, fail-closed reads | hal0 | Reliability on CPU boxes is hal0's open problem (#1834) |
| Remediation UX | Numbered steps + deep links from the API; status sentences | Typed next_steps model, rendered, but empty | ODS | Plumbing exists; fill it |
| Open WebUI wiring | Chat, RAG, search, STT, TTS, image with baked workflow | Chat, STT, TTS | ODS | Native buttons point at nothing |
| Container runtime | Compose, tag pins, :latest ADR, Docker 29.3 workaround | Quadlets, systemd backoff, digest pins, boot GPU converge | hal0 | Cleaner primitive, no daemon |
| AMD host tuning | GTT/TTM, iommu, sysctl, tuned, reboot banner | Diagnosis only; none applied | ODS | Corroborated by hal0's own toolbox README for GTT/TTM; amd_iommu=off is NPU-hostile and stays out (D7) |
| CLI | 6,595-line Bash file; presets; masked config show | 35-file Typer tree, issue-annotated; silent long waits | hal0 | Architecture; fix the waits |
| Config validation | Bash/jq schema validator, CI matrix per tier | Pydantic at load; no materialized-config CI | hal0 | Add the CI matrix |
| Doc hygiene | Dense operator docs; several stale claims | Good taxonomy; several stale claims and an ADR gap | tie | Both need a drift pass |
The install, observed
You asked for a fresh privileged LXC with the model directories mounted. I could not do that from this session: it runs in a remote sandbox with no GPU, 4 vCPU, 15 GB RAM, about 30 GB of disk and a Docker daemon I started by hand. So the install below is the closest honest equivalent: a rootless install user, CPU tier 0, non-interactive, with voice, workflows, RAG and ComfyUI disabled to fit the disk. Commands to reproduce it properly on the 395 box are at the end of this section. The full journal is appendix 00.
| Step | Wall clock | Outcome |
|---|---|---|
| Run 1: preflight → detection → config → dev tools → image pulls → model download | 07:25:36 → 07:31 | Rootless posture chosen up front; CPU fallback; tier 0 forced; 7.7 GB of images pulled largest-first; 1.2 GB GGUF SHA-verified; Hermes template patched with the served model id |
| Run 1: local image builds | 07:31 → 07:33:59 | All builds failed on TLS inside BuildKit (sandbox CA). Three attempts each, per-service build log, then the installer refused to start stale images and aborted. Correct behaviour. |
| Run 2 blocked on the model-lifecycle lock | 07:36:23 → 07:37 | Run 1's host agent had inherited the installer's flock descriptor. Killed it; the lock released. A real ODS bug (below). |
| Run 2: builds → compose validate → up → health → summary | → 07:41:07 | 4 min 44 s end to end with caches warm. 13 of 15 containers healthy; readiness summary Ready now: 5/6; summary JSON written |
| Post-install fixes | 07:42 → 07:58 | ods-dashboard and ods-searxng crash-looped because this kernel has no IPv6 (listen [::]:3001, GRANIAN_HOST=::). Patched the install copy; both healthy afterwards. |
What the installer taught me about ours
- Privilege posture is decided once, early. With no passwordless sudo the installer printed "continuing rootless", and every later privileged step was skipped and logged rather than failing under
set -e. hal0's installer issudo bash; the pattern still applies to its optional steps. - Six user-facing phases over thirteen internal ones, each with an estimated time. The internal structure is compressed into a storyline the operator can follow.
- Hardware is probed once and serialized (
/tmp/ods-capabilities.json); every later phase and the doctor read the file. The preflight verdict is also JSON, with anactionstring per check. - The model selector explains itself in one sentence: "Qwen 3.5 2B needs about 3GB including context/KV, fits 5.2GB system RAM on cpu, and gives 64K context." hal0's picker has the data to say this and does not.
- Every disabled feature says why ("Qdrant compose disabled (RAG not enabled or unsupported on this host)"), and every sub-step of the config phase is named, so a failure names the step.
- Image tags are validated before the pull, largest image first, with a counter and a human label per image. The GGUF is SHA-verified after download and the Hermes template is patched with the model id llama-server will actually advertise, then the substitution is checked.
- Compose config is validated before
up, and manifestdepends_onis checked, so a broken fragment fails before a container starts. - It found a real ODS bug. In the non-systemd fallback the host agent is started as a background child while the installer holds its model-lifecycle
flock; the child inherits the descriptor (/proc/<pid>/fd/10 → …/ods-model-lifecycle-….lock) and holds the lock after the installer exits, so the next run blocks on "Waiting for another model lifecycle operation". ODS guards the bootstrap upgrader against exactly this with_phase11_close_inherited_fds_for_daemonbut not the host agent. Under systemd the unit starts fresh and nobody notices. Worth an upstream issue, and a reminder for hal0's ownsystemd-runplan: never inherit a lock into a daemon.
Two sandbox-only adjustments, disclosed
The sandbox re-terminates outbound TLS, so image builds inside BuildKit could not verify certificates; I injected the sandbox CA into the eight locally built Dockerfiles in my install-user copy only (never the reference clone the reports were written against). And because the installer requires data/hermes to be mode 700 when sudo is unavailable, I pre-created it. Neither changes ODS behaviour.
The surfaces, as installed
Screenshots from the sandbox stack at 1440×900, dark scheme. The dashboard shots were taken after the IPv6 patch; no data beyond the fresh install exists on these pages.
Reproduce it properly on the Strix Halo box
Corrected 2026-10-04: the first draft used lxc launch / lxc config, which is LXD's client, not Proxmox's. On a Proxmox VE host the container CLI is pct and device passthrough lives in /etc/pve/lxc/<CTID>.conf; the recipe below mirrors hal0's own docs/getting-started/proxmox.mdx. The LXD form follows for anyone on a plain LXD host.
# ── Proxmox VE host ──────────────────────────────────────────────────────────
# pick a template; ODS's tested list includes Ubuntu 24.04. Resolve the exact file name the way
# hal0's scripts/proxmox-ve/hal0.sh:278 does rather than hard-coding a build number
pveam update
TPL=$(pveam available | awk '$1=="system" && $2 ~ /^ubuntu-24.04-standard.*amd64.tar.zst$/ {print $2; exit}')
[ -n "$TPL" ] || { echo "no ubuntu-24.04 template offered by pveam"; exit 1; }
pveam download local "$TPL"
CTID=9105
pct create $CTID local:vztmpl/$TPL \
--hostname ods-lab --cores 8 --memory 32768 --swap 0 --rootfs local-lvm:64 \
--net0 name=eth0,bridge=vmbr0,ip=dhcp \
--unprivileged 0 --features nesting=1,keyctl=1,fuse=1,mknod=1
# first boot WITHOUT GPU lines: the gid= value must be the render group's id INSIDE the guest,
# which cannot be known before the guest exists (docs/getting-started/proxmox.mdx §3 does the same dance)
pct start $CTID
RG=$(pct exec $CTID -- sh -c 'getent group render >/dev/null || groupadd -r render; getent group render' | cut -d: -f3)
echo "guest render gid: $RG"
pct stop $CTID
# GPU nodes + model stores go in the CT config (PVE 8.2+ dev passthrough). BOTH nodes carry gid=$RG:
# hal0's installer realigns /dev/kfd to the render gid itself (install.sh:487-498; hal0.sh:253,411-446
# does the same on the host side) but ODS's does not, so a kfd left at the host's gid is unreadable by ods.
# A privileged CT with GPU nodes also needs an unconfined AppArmor profile, or the container
# runtime inside the guest cannot launch (docs/getting-started/proxmox.mdx §AppArmor; hal0.sh:387-391).
# hal0's store is mounted read-only; ODS gets its own writable model dir on host storage (mp1) so a
# 15–31 GB download does not land on the 64 GB rootfs.
mkdir -p /var/lib/ods-lab/models
# derive the device majors the way hal0.sh:199-204 does (stat prints hex; cgroup rules want decimal) —
# never assume 226/234
DRI_MAJ=$(printf '%d' 0x$(stat -c '%t' /dev/dri/renderD128)); KFD_MAJ=$(printf '%d' 0x$(stat -c '%t' /dev/kfd))
cat >> /etc/pve/lxc/$CTID.conf <<EOF
dev0: /dev/dri/renderD128,gid=$RG
dev1: /dev/kfd,gid=$RG
mp0: /var/lib/hal0/models,mp=/mnt/hal0-models,ro=1
mp1: /var/lib/ods-lab/models,mp=/mnt/ods-models
lxc.cgroup2.devices.allow: c $DRI_MAJ:* rwm
lxc.cgroup2.devices.allow: c $KFD_MAJ:* rwm
lxc.prlimit.memlock: unlimited
lxc.apparmor.profile: unconfined
EOF
pct start $CTID
pct exec $CTID -- stat -c '%g %n' /dev/dri/renderD128 /dev/kfd # BOTH must show $RG before installing (docs/getting-started/drivers.mdx)
pct exec $CTID -- bash -lc 'apt-get update && apt-get install -y curl jq git rsync docker.io docker-compose-v2 && \
useradd -m -s /bin/bash ods && usermod -aG docker,render,video ods && \
echo "ods ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ods'
# 21f4b3a is the ODS revision every citation and result in this study is pinned to
pct exec $CTID -- su - ods -c 'git clone https://github.com/Osmantic/ODS.git && cd ODS && git checkout 21f4b3a'
# ODS reads ./data/models/*.gguf relative to ~/ODS/ods (docker-compose.base.yml:29; 11-services.sh:668,697) and
# writes its bootstrap download there (bootstrap-model.sh:44), so point that path at the writable host dir
# before install.sh runs. Seed it with any hal0 GGUF you want ODS to reuse — the file name must match ODS's
# catalog entry or the installer downloads its own copy regardless.
pct exec $CTID -- bash -lc 'chown ods:ods /mnt/ods-models && mkdir -p /home/ods/ODS/ods/data && ln -sfn /mnt/ods-models /home/ods/ODS/ods/data/models'
pct exec $CTID -- su - ods -c 'cp /mnt/hal0-models/<path>/<file>.gguf /mnt/ods-models/' # optional, once per model
pct exec $CTID -- su - ods -c 'cd ~/ODS/ods && ./install.sh --all --summary-json ~/ods-summary.json'
# ODS's phase 10 edits grub/modprobe INSIDE the CT; those belong to the Proxmox host kernel, and the
# GTT/TTM numbers it printed are WRONG for the host: phase 10 sizes them from the CT's own MemTotal
# (10-amd-tuning.sh:140-157) — 32 GB here, not the host's 128 GB. If you want fixed sizing at all (it is
# opt-in; kernels ≥ 6.14 size GTT dynamically, drivers.mdx:86-97), recompute ON THE HOST from its own
# /proc/meminfo — e.g. hal0's documented 128 GB line, drivers.mdx:100-104:
# amdgpu.gttsize=118784 ttm.pages_limit=30408704 ttm.page_pool_size=30408704
# then reboot the host (report 08 §D.1). Do NOT copy ODS's amd_iommu=off: it removes /dev/accel (the
# XDNA NPU) and contradicts hal0's own iommu=pt amd_iommu=on (drivers.mdx:102; ADR-0003:96-99).
# afterwards: ods status; ods list; ods doctor --json; cat /tmp/ods-preflight-report.json
# ── plain LXD host (not Proxmox) ────────────────────────────────────────────────
lxc launch images:ubuntu/24.04 ods-lab -c security.privileged=true -c security.nesting=true
RG=$(lxc exec ods-lab -- sh -c 'getent group render >/dev/null || groupadd -r render; getent group render' | cut -d: -f3)
lxc config device add ods-lab kfd unix-char path=/dev/kfd gid=$RG
lxc config device add ods-lab dri unix-char path=/dev/dri/renderD128 gid=$RG
mkdir -p /var/lib/ods-lab/models
lxc config device add ods-lab hal0models disk source=/var/lib/hal0/models path=/mnt/hal0-models readonly=true
lxc config device add ods-lab odsmodels disk source=/var/lib/ods-lab/models path=/mnt/ods-models
lxc exec ods-lab -- bash -lc 'apt-get update && apt-get install -y curl jq git rsync docker.io docker-compose-v2 && \
useradd -m -s /bin/bash ods && usermod -aG docker,render,video ods && \
echo "ods ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/ods'
lxc exec ods-lab -- su - ods -c 'git clone https://github.com/Osmantic/ODS.git && cd ODS && git checkout 21f4b3a'
lxc exec ods-lab -- bash -lc 'chown ods:ods /mnt/ods-models && mkdir -p /home/ods/ODS/ods/data && ln -sfn /mnt/ods-models /home/ods/ODS/ods/data/models'
lxc exec ods-lab -- su - ods -c 'cd ~/ODS/ods && ./install.sh --all --summary-json ~/ods-summary.json'
# same host-side tuning caveat as the Proxmox branch: recompute on the LXD host from host RAM; never amd_iommu=off
Adoption plan
Principles
- Copy shapes, not files. The ODS artifacts that are literally droppable are small (the session signer, the patcher, the progress protocol, the AMD tuning math). Everything else is a design decision hal0 re-implements in Python on podman and systemd.
- Extensions are companion services, never slots. A separate unit family,
hal0-ext@<id>, rendered from a typed manifest, never from an author-written unit body. Explicitly rootful in v1 — it goes through the same system-scope seam as slots — with compensating controls (digest-pinned images, volumes confined to/var/lib/hal0/ext/<id>/, no devices, capabilities dropped) and third-party import disabled until a user-scope seam makes rootless real (report 02, Decision 3, corrected 2026-10-04). - Keep what is better. Quadlets, digest pins, the cosign updater, pip Hermes, the Typer CLI, Hindsight, the shared health classifier. None of these are on the table.
- Every installed thing stays pre-wired to the slots. Pre-wiring in hal0 is rendered env, computed at unit-render time from live truth:
OPENAI_BASE_URLto hal0-api,OPENAI_MODEL=hal0/brain, andslot:<name>references resolved to hal0-api's gateway route for that modality — slots are loopback-only on the host, so a bridge-network extension cannot reach a slot port directly, and a value rendered once does not follow a later port change, whereas the gateway's port is stable and the dispatcher follows the slot. Direct127.0.0.1:<slot port>wiring is reserved for host-network extensions and needs a slot-change convergence hook that does not exist yet (report 02 §D.5, corrected 2026-10-04). Still stronger than ODS's static DNS names, for a narrower claim. - Finish before starting. Wave 0 closes completely before Wave 1 opens. The backlog already has enough 80 % threads.
Port ledger
Ranked within waves. Size: S under a day, M under a week, L multi-week. Risk is integration risk in hal0. "Issues" are the open hal0 issues the row closes or de-risks.
| # | Port | From (ODS) | To (hal0) | Size | Risk | Impact | Issues |
|---|---|---|---|---|---|---|---|
| Wave 0 · stop the bleeding | |||||||
| 0.1 | Signed session cookie + verify-session + admin-session; gate approvals and other mutation routes (approvals gated on LAN-bound boxes by #2246) | dashboard-api/session_signer.py, routers/auth.py | src/hal0/api/auth/, generalize api/agents/_auth.py | S | low | high | — |
| 0.2 | Reproduce Qwen3-Coder-Next on a real 395; demote from hal0-Max and add a unified-memory exclusion if it reproduces | tier-map.sh:100-138, select-model.py:326-372 | registry/curated.py:260-275, model_fit.py, bundle manifest | S | low | high | — |
| 0.3 | Doctor cross-check: bind non-loopback and auth off → WARN with remedy; same line in the install summary | ods-doctor.sh:1074-1089, 13-summary.sh:419-427 | cli/doctor_all.py:62-83, installer/install.sh:3934 | S | low | high | #1822 |
| 0.4 | Bound every seam subprocess call; typed timeout error | ods-uninstall.sh:321, bootstrap-upgrade.sh:939 | providers/container.py:2549, :2697, :2708, :2900 | S | low | high | #1869, #1870 |
| 0.5 | Re-assert /usr/local/bin/hal0 in the updater next to refresh_privileged_wrappers; have deploy.sh call the same refresh | 13-summary.sh:270-300 | updater/updater.py:4159, scripts/deploy.sh | S | low | high | #1844, #2019 |
| 0.6 | Server-built next_steps for convergence; the panel prints exactly those lines | ods-doctor.sh:1110-1163 | /api/updates/convergence, cli/update_commands.py:466-492 | S | low | high | #1845 |
| 0.7 | Deploy verifies the served build identity before printing success | ods-update.sh:378-406, :647-742 | scripts/deploy.sh:213-231, expose version on /api/status | S | low | high | #1550 |
| 0.8 | Doc drift pass: provisioner description, BUNDLED_AGENTS, Documentation= path, duplicate memory provider, ADR bookkeeping | — | ARCHITECTURE.md, CLAUDE.md, docs/adr/, installer/agents/hermes/plugins | S | none | med | — |
| Wave 1 · cheap shapes with big returns | |||||||
| 1.1 | Module header convention on every installer lib; tee'd install log; failure report from the ERR trap; --summary-json; per-step time estimates | ui.sh:97-103, compose-failure-report.sh, 13-summary.sh:435-502 | installer/lib/ui.sh, new installer/lib/failure-report.sh, hal0 doctor bundle | M | low | high | — |
| 1.2 | Pull retry discipline: backoff table, post-pull inspect, non-retryable classifier | ui.sh:269-370 | installer/install.sh:2177, components/runner_images_arm.py | S | low | high | — |
| 1.3 | Preflight JSON report using Diagnosis fields + fixture corpus with named scenarios | scripts/preflight-engine.sh, tests/contracts/test-preflight-fixtures.sh | installer/lib/preflight.sh, tests/installer/ | M | low | high | — |
| 1.4 | Populate next_steps for the seven doctor checks; chips copy or run; enable-steps modal shape | routers/features.py:130-144, :188-250, FeatureDiscovery.jsx | health_report.py:246-283, DiagnosisPanel.jsx | M | low | high | #1845, #1994 |
| 1.5 | Hardware re-detect cluster: live-probe fallback for stale hardware.json; kfd_present() and render_node_present() sufficient for ROCm; one apply_cpu_fallback(); clamp seed context_size to the envelope | detection.sh:26-46, :187-257, select-model.py:134-182 | slots/capacity.py:499-585, install/profile_derive.py:110, install/static_seeds.py | M | med | high | #1862, #2216, #1936, #1966, #1868 |
| 1.6 | Manifest-driven services health with gather(return_exceptions=True); first row of the extension registry | dashboard-api/config.py:297-431, helpers.py:624-700 | api/routes/services_health.py:198-307, services/registry.py | M | low | high | #2028, #2192, #1859 |
| 1.7 | AMD host tuning lib (opt-in only, no default-on: kernels ≥ 6.14 size GTT dynamically and hal0 reads the live pool, drivers.mdx:86-97; GTT/TTM + sysctl + tuned only — never amd_iommu=off, which drops the NPU's /dev/accel and contradicts drivers.mdx:102; container-guarded — prints the host-side recipe inside an LXC, never claims applied); gfx1151 HSA auto; kernel/firmware known-bad checks in doctor | 10-amd-tuning.sh:100-302, .env.example:477 | new installer/lib/amd-tuning.sh, install/profile_derive.py, cli/doctor_commands.py | M | med | high | — |
| 1.8 | Full Open WebUI wiring (RAG embeddings, SearXNG-style search if present, image gen with baked workflow), gated on bound slots | docker-compose.base.yml:100-168 | openwebui/env_writer.py:106-131 | M | med | med | — |
| 1.9 | Update recreates slots on image roll by default (opt-out flag); stack import verifies checksum on commit; uninstall gates its final line on per-step outcomes; TTL on negative FLM probe; retention prune on a timer | ods-cli:2049, ods-restore.sh:273-310, ods-uninstall.sh:362-402, helpers.py:26-56 | cli/update_commands.py:516, stacks/portable.py, agent uninstall, capabilities/catalog.py:178, activity prune | M | low | med | #2096, #1512, #1519, #1974, #1521 |
| 1.10 | Docs: generated-config-writers table; area → required-validation table; reinstall contract; known-good versions; README "at a glance"; post-install checklist | INSTALLER-ARCHITECTURE.md:118-135, HIGH_RISK_CHANGE_MAP.md, KNOWN-GOOD-VERSIONS.md | ARCHITECTURE.md, CONTRIBUTING.md, docs/reference/, README.md | S | none | med | — |
| Wave 2 · structural | |||||||
| 2.1 | Decompose install.sh into installer/steps/, one step per PR, UI_STEP_TOTAL test green throughout; then --dry-run and a simulate harness | install-core.sh, installers/phases/, simulate-installers.sh | installer/install.sh → orchestrator + steps | L | med | high | — |
| 2.2 | Finish user-added MCP — partly shipped on main (#2253, ADR-0015): schema, exposure join, hal0 mcp test|allow|gate|block|expose, REST. Remaining: policy proxy mount so [tools] is enforced on Hermes's call path (#2303); client-attached stdio bridge (not a free-standing unit); install drawer (#2245) | patch-hermes-config.py:17-59,114-115,215-218 | mcp/installed.py, routes/mcp.py, agents/hermes_provision.py:2012, ui/src/dash/connections.jsx | L | med | high | #1930 first |
| 2.3 | Extension model v1: manifest.toml → hal0-ext@<id> quadlet; registry walker; capability status vocabulary; hal0 app verbs; allow-list +2 keys; convert OpenWebUI as proof; n8n as the first new extension | manifest schema, service-registry.sh, features.py:19-110, audit-extensions.py, templates | new src/hal0/extensions/, system/seam.py:283, installer/wrappers/hal0-systemctl:389-516, cli/app_commands.py | L | high | high | #1947, #1319 |
| 2.4 | Progress/timeout/evidence seam: stream slot phases over SSE into the CLI; failure report per slot; classified retry for netavark/name-conflict | 12-health.sh:66-106, background-tasks.sh:119-148, ods-cli:960-1140 | cli/slot_commands.py:311-362, slots/, system/netavark.py + seam.prune_dnat | M | med | high | #1870, #2221, #1820, #2180 |
| 2.5 | Persist the approval queue; args-fingerprinted one-shot grants; windowed caps + circuit breaker keyed on (server, tool) | ape/main.py:482-612, :954-1010 | mcp/approval_queue.py, new mcp/limits.py | M | low | med | — |
| 2.6 | Schema-driven settings renderer; one ChangeSet diff shared by preview and apply | settings.py:194-241, ods-cli:6261-6324 | settings pages, slot_config | M | low | med | #2108, #1967, #2195, #2203, #1511 |
| Wave 3 · product surfaces (each gated on a decision below) | |||||||
| 3.1 | Single-URL LAN gateway: Caddy forward_auth on :80, <host>.local subdomains, owner card + guest invite | ods-proxy/Caddyfile, hermes-proxy/Caddyfile:54-129, routers/magic_link.py | new packaging/hal0-proxy/, api/routes/services.py:38-73 (X-Forwarded-Host); prerequisites: take the direct ports off the LAN first (install.sh:157,1567 seed 0.0.0.0 today) and keep that converged on upgrade, or forward_auth is bypassable on :8080/:3001 — but not by binding hal0-api to loopback alone: Open WebUI and every bridge-network extension reach it via host.docker.internal, the podman bridge gateway, not 127.0.0.1 (openwebui/env_writer.py:102-128), so hal0-api needs a second listener on the bridge gateway address or an nftables rule that drops LAN ingress to :8080/:3001 while exempting the bridge subnet; plus an mDNS address publisher for the subdomains (services/mdns.py writes _http._tcp service records only) | L | med | med | — |
| 3.2 | Bootstrap-then-hot-swap for first-run tiers via systemd-run, status JSON, snapshot/rollback, verified-before-cleanup | bootstrap-model.sh, 11-services.sh:673-690, bootstrap-upgrade.sh:305-452 | install/agent_model.py, new install/background_pull.py, hal0-model-pull@.service | L | med | high | — |
| 3.3 | Agent-driven OAuth passthrough + provider registry + persona addendum | routers/oauth_passthrough.py, hermes/oauth-providers.json | new api/routes/oauth.py, installer/agents/hermes/ | M | low | med | — |
| 3.4 | Phone chat portal on the pooled-bridge pattern (only if decided) | hermes_bridge.py:162-528, routers/talk.py | new REST/SSE surface beside api/agents/chat_proxy.py | L | med | med | — |
| 3.5 | Setup card + consequence-first copy pass on status words and tooltips | generate-setup-card.py, Extensions.jsx:60-84 | new hal0 setup-card, slot-status.js, settings tooltips | M | low | med | — |
| Wave 4 · optional, design-pass first | |||||||
| 4.1 | Stable alias generalization (hal0/chat) through the dispatcher | model-router/app/main.py | src/hal0/dispatcher/ | L | high | med | — |
| 4.2 | Fleet-verified compatibility ledger on curated entries, populated by hal0's own release-validation kit | model-library.json:9-63 | registry/curated.py, tests/release-validation/ | L | med | med | #1823 |
| 4.3 | Optional resource limits on slot quadlets; CI job materializing each bundle tier's config | deploy.resources, validate-env.yml | providers/container.py:803, .github/workflows/ | S | low | low | — |
Waves
Numbered because the order carries information: each wave makes the next one reviewable.
Stop the bleeding 8 items, all S
Close the auth hole, settle the flagship model question, land the five one-afternoon fixes the issue map ranks highest, and fix the documentation the later waves will be built against.
- Done when: approvals require a session; the 395 verdict on Qwen3-Coder-Next is recorded in an issue;
hal0 doctor allwarns on a LAN-open, auth-off box; no seam call is unbounded;hal0 updatere-links the CLI; #1845's panel prints a command that works;deploy.shcannot lie;ARCHITECTURE.mdmatches the code.
Cheap shapes, big returns 10 items, S–M
Instrument the installer, make hardware truth fresh, give the doctor real next steps, wire Open WebUI fully, and offer the AMD host tuning hal0's own upstream documents (opt-in; see D7).
- Done when: a failed install leaves a log and a report; preflight verdicts are pinned by fixtures; every doctor row has a runnable step; the ROCm lane follows
/dev/kfdand arenderD*node together (kfd_present()andrender_node_present()); an image roll restarts slots; a 128 GB box's live GTT verdict is in the doctor and fixed sizing is offered, not applied; Open WebUI's upload and image buttons work.
Structural 6 items, M–L
Decompose the installer, finish MCP, ship the extension model with OpenWebUI converted and n8n added, and build the progress seam the CLI needs.
- Done when:
installer/steps/exists and--dry-runruns in CI;hal0 mcp addreaches Hermes with default-deny tools;hal0 app install n8nrenders a unit, claims a port, wires env, probes health and shows a card with no code edit;hal0 slot loadprints phases.
Product surfaces 5 items, gated on decisions D1, D2, D8
Only after the LAN-posture and phone-surface decisions: the single-URL gateway with owner cards, bootstrap-then-hot-swap, OAuth passthrough, an optional phone portal, and the copy pass.
Design-pass items 3 items
The stable alias, the fleet ledger, resource limits. Each needs its own design note before code.
Issue clusters: one change, many issues
Of 106 open issues, 4 are direct ports, 34 are pattern adoptions, 29 are informed by ODS, and 39 are unrelated (polish sweeps, test hygiene, upstream pins, Hindsight internals). Eighteen of the twenty an operator actually hits on a real box fall into five clusters. Full table in appendix 09.
| Cluster | hal0 change | Closes or de-risks | Ledger |
|---|---|---|---|
| A. Progress / timeout / evidence | A bounded_call() every seam invocation goes through; phase events on the existing SSE bus; a classified retry for known-recoverable failures; a per-slot failure report | #1870 #1869 #1873 #2221 #1550 #1820 #2180 | 0.4, 0.7, 2.4 |
| B. Manifest-driven service and capability registry | One schema-validated table read by the health route, Omni eligibility, the device picker and the context resolver | #2028 #2192 #1966 #1859 #2190 #2184 #1990 #1823 #2234 #1947 #1319 | 1.6, 2.3 |
| C. Hardware re-detect and honest CPU fallback | Live-probe fallback for a stale cache; device nodes decide the lane; one apply_cpu_fallback(); one memory-envelope function read by seeds, ruler, extraction sizing and the anchor floor | #1862 #2216 #1936 #1966 #1868 #1834 #1867 #2201 #1928 #1929 | 1.5 |
| D. Update and deploy delivery integrity | Re-assert PATH links; verify the served build; verify digests on commit; gate success on per-step outcomes; recreate on image roll | #1844 #2019 #1550 #2096 #1845 #1519 #1512 #1530 #2101 | 0.5, 0.6, 0.7, 1.9 |
| E. Schema-driven settings and previews | Render a settings field for every schema key; one diff function shared by preview and apply | #2108 #1967 #2195 #2203 #1511 #1445 | 2.6 |
Decisions only you can make
Ordered by conviction: what I would do, the safe version, and the cheapest experiment. Reversible ones can move fast; the two that touch boot and trust deserve a day of thought.
D1. LAN exposure model
HAL0_BIND_HOST to loopback (the library fallback already is) together with a bridge-reachable second listener or a LAN-ingress firewall cut — loopback alone cuts off Open WebUI and every bridge extension, which reach hal0-api via host.docker.internal (report 02 §D.5 (iii)) — add an explicit --lan like ODS, and gate mutation routes with the signed session regardless. Revisit the KB-1 objection: it no longer holds now that a login page and a Security settings page exist.0.0.0.0, ship 0.1 (session on mutation routes) and 0.3 (doctor warning). Nobody gets locked out; the worst hole closes.Why it matters: this decision gates the whole of Wave 3.1. Reversible: a bind default is one env var.
D2. Do you want a phone surface?
Why it matters: 3.1, 3.3 and 3.4 stand or fall together. Reversible.
D3. Container Hermes or pip Hermes
ProtectSystem=strict set is a stronger boundary than ODS's container, which needed HERMES_DASHBOARD_INSECURE=1 punched through it, and hal0's plugins depend on the venv.ARCHITECTURE.md so the question stops being reopened.D4. What an extension is allowed to be
slot:<name>.slot: refs). The eligibility half of the design collapses and v1 gets meaningfully cheaper.manifest.toml by hand before writing any loader. If it cannot be expressed, the schema is wrong.Why it matters: the seam allow-list is a root-boundary change; decide its width once. Partly irreversible: a published manifest schema is a contract.
D5. MCP supervisor shape (ADR-0015)
hal0 mcp test for http/sse servers first (done on main, #2253 / ADR-0015), then the policy proxy mount so allow/gated/blocked actually bind on Hermes's call path (#2303), then a client-attached stdio bridge behind that same mount. Not a free-standing hal0-mcp@<id>.service: a stdio server with nobody on its pipe is dead.Also decide: credentials in /etc/hal0/api.env (reuses the secrets API) versus a separate file. I would reuse.
D6. Decompose the installer
make harness-install between moves. Re-affirm "the installer is the whole setup" before starting, because decomposition is exactly when a post-install wizard gets proposed.D7. AMD host tuning: default-on or opt-in
drivers.mdx:86-97), so a fixed 65–90 % reservation is an operator's floor, not a product default — and on a shared Proxmox host it takes that RAM from every other guest. Ship it as an explicit --apply with a reboot banner; the doctor reports the live GTT and offers the pinned recipe only when the kernel is older than 6.14 or the operator asks. It edits GRUB, so keep it reversible. Scope it to GTT/TTM, sysctl and tuned: never write ODS's amd_iommu=off, which removes /dev/accel (XDNA NPU) — hal0 documents iommu=pt amd_iommu=on (drivers.mdx:102, ADR-0003:96-99) and the port must leave that line as hal0 documents it.iommu=pt amd_iommu=on, not the README's amd_iommu=off) on one box and confirm the fit: a GGUF that previously failed the live GTT check now loads. Do not chase the README's 5 to 12 % or ODS's ~6 %; both are the IOMMU-off delta hal0 forgoes to keep the NPU.Slow down here: this is the one item in the plan that can leave a box unbootable if the initramfs step fails.
D8. Who is the copy for?
slot-status.js).D9. Bootstrap-then-hot-swap
systemd-run with journald logging, a status file for the dashboard, and the snapshot / verified-before-cleanup rules. It strengthens "the installer is the whole setup" by removing the one question big enough to make people defer.Do not copy · do not do
Do not copy from ODS
- Docker Compose, or any layering engine. Quadlets plus rendered units are the better primitive.
ods-cli, or any single-file dispatcher.ods-update.sh'sgit pullmechanism, or the:latestADR.- APE, the agent templates, memory-shepherd, the agents metrics router, the regex config patcher as a primary path.
- The zeroconf daemon, the cookie-domain trick alone, AP mode, the mobile installer stub.
- The CRT theme, the lore copy, the typing animation, the Tauri GUI, per-platform phase copies.
- A hard-coded Hermes context floor,
HERMES_DASHBOARD_INSECURE, HTML token scraping. - Rename-the-file as enable state, the hand-maintained
ENABLE_*table, a committed generated catalog,ports.json. - A vector database with no consumer.
Do not do next
- Do not fork ODS or vendor its tree. Take shapes; the files are wrong-shaped for podman/systemd.
- Do not start Wave 1 while any Wave 0 item is open.
- Do not begin the extension loader before the allow-list width (D4) is written down.
- Do not build the Caddy gateway before D1 is decided; it changes the trust model — and never enable
forward_authon :80 while hal0-api (:8080) and Open WebUI (:3001) still answer the LAN directly, or the gateway is decorative — and closing them must keep the container path open (Open WebUI reaches hal0-api throughhost.docker.internal, not loopback). - Do not add a first-run wizard. Strengthen the end of the installer instead.
- Do not ship hal0-Max's coder pick until 0.2 has a recorded verdict.
- Do not port ODS's memory pieces to fix #1834 or #2016; those are Hindsight problems.
- Do not widen the MCP exposure targets past
hermesandbrainin v1.
Appendices
The full reports, unedited. Every claim in the sections above traces to a file:line in one of these. Reports 01, 02, 03, 06 and 09 were written by Opus agents; 04, 05, 07 and 08 by Sonnet agents; 00 is my own install journal.
00-install-journal.md Install journal (sandbox run)
ODS install journal (sandbox run, 2026-09-05)
Environment: Claude Code remote sandbox (Ubuntu 24.04, 4 vCPU, 15 GB RAM, ~30 GB writable disk, no GPU, Docker 29.3.1 daemon started by hand with overlay2, egress only via an HTTPS proxy on the host). This is NOT the requested privileged LXC with the AI-model directories mounted — that must be done on the operator's Strix Halo box. Commands to reproduce there are at the end.
0. Pre-install decisions forced by the sandbox
| Constraint | ODS behaviour observed | Notes for hal0 |
|---|---|---|
Installer refuses root (installers/phases/01-preflight.sh:22) |
Hard error with a one-line reason | hal0's installer is sudo bash; ODS is rootless-first and escalates only for "extras" |
| No passwordless sudo for the install user | ods_prepare_sudo (installers/lib/sudo.sh) detects it up front, prints "continuing rootless", every later ods_sudo call is skipped and logged instead of hanging or failing under set -e |
Clean pattern: decide privilege posture once, early, then make every privileged step optional and visibly skipped |
data/hermes must be uid:gid:700 on a rootful runtime or the phase errors (phase 06) |
Pre-created it with mode 700 before running | Upstream Hermes image demands 0700 on HERMES_HOME |
| Containers have no direct internet | Image pulls are done by the daemon (which has the proxy), model download is done by the host (curl/huggingface_hub), so this mostly works |
Same split hal0 already has (host downloads, container serves) |
Flags used: --non-interactive --skip-docker --tier 0 --no-voice --no-workflows --no-rag --no-comfyui --summary-json …
1. Phase-by-phase observations
Boot / arg parsing (install.sh → installers/dispatch.sh → install-core.sh)
install.shis a 40-line dispatcher: resolves OS → Linux/macOS/Windows target andexecs it.install-core.shparses ~40 flags; defaults live in one block (voice/workflows/RAG/recommended/Hermes on; OpenClaw/Langfuse/proxy/Tailscale off) with a comment on every non-obvious default.- Distro detection prints
Detected distro: ubuntu (like: debian, pkg: apt); PyYAML is auto-provisioned for the system python because the service registry reads YAML manifests.
Phase 1/6 "Pre-flight Checks" (internal phases 01)
- A themed banner + narrator voice:
Signal acquired. I will guide the installation. Stay with me.; every phase header carries an estimated time (EST. TIME: ~30 seconds). - The UI shows 6 user-facing phases while the code has 13 internal phases — a deliberate compression of internal structure into a human storyline.
- Checks: OS, curl, jq (auto-installs), optional tools (rsync → prints the exact apt/dnf/pacman command), active UFW/firewalld warning, filesystem POSIX-permission check (exFAT/NTFS/9p refuse
chmod 600 .env), related-install detection (scans$HOMEfor other compose stacks with open-webui + dashboard-api + llama-server/litellm, and running compose projects) so two stacks never fight over ports silently.
"System detection" (internal phase 02 + 02b)
- Writes a capability profile JSON (
/tmp/ods-capabilities.json: platform, gpu vendor/name/memory_type/count/vram, runtime backend + health URL + port, compose overlays, recommended tier, hardware class). This one file is what later phases and the dashboard read — hardware is probed once and serialised. - No GPU →
apply_cpu_gpu_fallback→GPU_BACKEND=cpu, backend contract loaded fromconfig/backends/cpu.json. - Prints the full compose selection (base + cpu overlay + every enabled extension's compose fragment +
docker-compose.tier0.ymlmemory-limit overlay), then prints one line per disabled feature saying why it is disabled ("Qdrant compose disabled (RAG not enabled or unsupported on this host)"). - Catalog model selector output is a sentence a human can act on:
Qwen 3.5 2B needs about 3GB including context/KV, fits 5.2GB system RAM on cpu, and gives 64K context. Throughput requires a local benchmark after first launch. Bounded by --tier 0's model size ceiling (1221MB); use ODS_DISABLE_CATALOG_MODEL_SELECTOR=true to bypass.
"Requirements check" (internal phase 04)
- Delegates to a fixture-driven preflight engine that emits
/tmp/ods-preflight-report.json:{summary:{checks,blockers,warnings,can_proceed}, checks:[{id,status,message,action}]}. Each check has anactionstring — the remediation is data, not prose in a log. - Disk requirement is computed from the selected model size + 15 GB images, not a static table.
"Directories / config" (internal phase 06)
- Every sub-step is named and logged (
Phase 06 step: prune-retired-services,copy-extensions-library,configure-legacy-openclaw,prepare-service-permissions,generate-env,render-model-router-config,validate-env,generate-searxng-config). A failure names the step, not just the phase. - Docker rootless state is detected once (
lib/rootless-ownership.sh) and drives every ownership decision;ODS_UID/ODS_GIDare persisted into.envso re-runs and containers agree on who ownsdata/. - A hard-coded
chown … 1000:1000for token-spy failed under uid 1001 and was reported as non-fatal with the consequence spelled out ("container may crash if installer ran as a different uid"). .envis generated with secrets (chmod 600), then validated against.env.schema.jsonbefore anything starts. The extension library (optional catalog) is copied intodata/extensions-library/so the dashboard can browse it offline.
"Dev tools" (internal phase 07)
- Installs Claude Code, Codex CLI and OpenCode into
~/.npm-global(no sudo needed); OpenCode failed and the installer printed the exact one-liner to retry later. Non-fatal by design. - Starts the ODS host agent (
bin/ods-host-agent.py): a host-side helper for model downloads and mDNS. No systemd in the sandbox → "starting the host agent without a system service … Runods agent startafter reboot". Graceful degradation, with the recovery command printed. - Installs
zeroconfviapip --userwhen the distro package cannot be installed: the mDNS announcer is a Python script (bin/ods-mdns.py), not an avahi service file. It publishes<device>.localpluschat/dashboard/auth/api/hermes/talk.<device>.localA-records and re-reads.envevery 30 s so renames need no restart. A static AST test (tests/test_mdns_subdomains.py) pins the subdomain set to the Caddyfile routes and the magic-link redirect targets.
Phase 4/6 "Downloading Modules" (internal phase 08)
- Validates that each pinned image tag exists in the registry before pulling (llama-server, Hermes), so a bad pin fails in seconds, not after a 4 GB download.
- Pull order is largest first, with a spinner, elapsed time,
[n/N]counter and a human label per image ("LLAMA-SERVER — downloading the brain (CPU)", "HERMES PROXY — magic-link auth gate (Caddy)"). Rotating one-line taglines fill the wait. Images pulled: llama.cpp server-b8248 (113 MB), open-webui v0.7.2 (4.39 GB), perplexica, hermes-agent v2026.6.5 (3.13 GB), caddy 2.11.3-alpine (63 MB).
Phase 5/6 "Starting Services" (internal phase 11)
- Compose flags are recomputed from the install directory and saved to
.compose-flags(the CLI reuses them; no re-detection at everyods start). - GGUF download (host-side,
huggingface_hub), then SHA-256 verification against the pinned digest ininstallers/lib/bootstrap-model.sh. - Generates
models.inifor llama-server, then patches the Hermes template with the actually-served model id (model.default=Qwen3.5-2B-Q4_K_M.gguf, context=65536) and verifies the substitution landed. - Validates service dependencies (manifest
depends_on) and runsdocker compose configbeforeup— a broken fragment fails before any container starts. - Local builds are skipped per disabled service ("Skipping local image build for disabled service: brave-search").
Run 1 outcome: local image builds failed, install aborted (by design)
- Every locally built service (
dashboard,dashboard-api,model-router,ape,privacy-shield,token-spy, …) failed withcertificate verify failedinside BuildKit: the sandbox re-terminates outbound TLS and build containers do not trust its CA. Not an ODS defect. - ODS behaviour worth copying: each build gets three attempts with a 5 s delay, a per-service build log (
/tmp/ods-install.log.<svc>.build.log), a cross-check that the build actually produced the tagged image, and then a refusal to start an image left by an earlier install ("Fix the build error and rerun the installer"). It never silently ran stale code. - Sandbox fix: the sandbox CA was appended to the eight locally built Dockerfiles in the install user's private copy only.
A real ODS bug found while re-running: the host agent inherits the installer's lock
- Run 2 blocked at "Waiting for another model lifecycle operation before Linux installer model configuration…".
- Cause: in the no-systemd fallback, phase 07 starts
bin/ods-host-agent.pyas a background child after the installer has taken its model-lifecycleflock. The child inherits the descriptor (/proc/<agent-pid>/fd/10 → /tmp/ods-model-lifecycle-<uid>/ods-model-lifecycle-<cksum>.lock) and keeps the lock for its whole lifetime, so any later installer run orods model swapwaits forever. ODS closes inherited FDs for the bootstrap upgrader (_phase11_close_inherited_fds_for_daemon) but not for the host agent. Under systemd the unit starts fresh and nobody notices. Killing the agent released the lock; run 2 then started its own agent, which inherited the lock again. - Lesson for hal0's planned
systemd-runbackground pulls: never let a daemon inherit a lock descriptor; take locks inside the daemon.
Run 2 outcome: stack up in 4 min 44 s (images and model already cached)
| Phase | Wall clock | Notes |
|---|---|---|
| Run 1 start → killed at builds | 07:25:36 → 07:33:59 (8 m 23 s) | ~7.7 GB of images pulled in ~3 min; 1.2 GB model downloaded and SHA-verified |
| Run 2 start → "YOUR ODS IS LIVE" | 07:36:23 → 07:41:07 (4 m 44 s) | includes 6 local image builds; ~1 min was the inherited-lock wait |
| Containers healthy at the end of run 2 | 13 of 15 | ods-dashboard and ods-searxng restart-looping |
- Readiness summary:
Ready now: 5/6with a per-URL table, a "Needs attention" block, and a "Next:" block naming the exact compose logs command. This is the shape hal0's install summary should have. --summary-jsonwrote a versioned payload: installer version, tier id and name,runtime.{gpu_backend, backend_service, llm_model, compose_flags, dry_run}, hardware class, preflight report path.- Extension manifest validation ran against the installed ODS version: 27 manifests, 0 incompatible, then an extension runtime check probing each enabled non-core service's health URL from the manifest.
- The generated
.envhas ~150 keys;.compose-flagscaches the resolved stack;config/llama-server/models.iniis four lines;data/has one directory per service.
Two IPv6 findings (matter for Proxmox LXCs with IPv6 disabled)
ods-dashboard: nginx config haslisten [::]:3001;unconditionally ("IPv6 support for Docker healthcheck"); on a kernel without IPv6, nginx fails withsocket() [::]:3001 failed (97: Address family not supported by protocol)and the container crash-loops. Sandbox fix: comment out the IPv6 listen line in the image.ods-searxng: the upstream image defaultsGRANIAN_HOST=::; same failure. Sandbox fix:GRANIAN_HOST=0.0.0.0in the service env.- Both are one-line hardenings ODS could ship (
listen [::]:3001 ipv6only=on;guarded, or IPv4 default plus an env toggle). hal0 already binds IPv4 explicitly in its unit templates; keep it that way.
Live surfaces after the fixes
llama-server /v1/models→Qwen3.5-2B-Q4_K_M.gguf; a direct chat completion answers in ~20 tokens on 4 vCPU.- LiteLLM exposes only the aliases
defaultand*; a completion toods/currentis answered by the router and reports the real model id in the response. hermes-proxy /→303 → /auth/required("Owner card required") with no cookie: the forward_auth gate is real.dashboard-api /api/statusreturns per-servicestatus/state/severity/countsAsIssueplus the manifest'sllmcontract (Hermes:route: gateway,min_context: 65536,swap_safe: true, with a one-sentenceswap_safe_reason)./api/features→ 4 feature cards (hermes-agent: services_needed,hermes-sso: services_needed,coding: insufficient_vram,agent-governance: enabled) and two plain-English recommendations ("Limited GPU memory — chat will work with small models", "Consider cloud hybrid mode for better quality")./api/features/hermes-agent/enablereturns numbered steps plus an "Open Hermes" link./api/extensions/catalog→ 33 catalog entries; 24incompatibleon the CPU backend, 9not_installed./api/external-linkslists 7 sidebar quicklinks derived from manifests.ods listprints 27 services with category (core/recommended/optional) and state (always-on/enabled/disabled);ods statusprints the container table, then per-service health checks.- The rendered persona (
data/persona/SOUL.md) opens with "About this ODS install — read this BEFORE answering questions about your environment": host, backend, serving model and context window, the dashboard URL, the running-services list, and a "you can call these / you can only point the operator at these" split. Two inaccuracies in the generated block: it advertiseshttp://vm.localandchat.vm.localalthoughods-proxyis disabled in this install, and it expands APE as "agentic prompt engineering surface" (it is the Agent Policy Engine). A generated context is only as good as its description table.
2. What hal0 should take from the install experience (short list)
- One capability-profile JSON written once per run, read by everything after it.
- Preflight and doctor verdicts as JSON rows with an
actionstring; fixture-tested. - A tee'd install log plus a redacted failure report written from the ERR trap.
- Per-step time estimates, named sub-steps, and one line per disabled feature saying why.
- Validate image tags before pulling; pull largest first; retry with backoff; validate after pull.
- A readiness summary with Ready / Needs attention / Next, and
--summary-json. - Bind IPv4 explicitly (hal0 already does); never inherit a lock into a daemon (hal0 must, when it adds background pulls).
01-agents.md Agents and Hermes
Agents end-to-end: ODS vs hal0
Source-level comparison. Every claim below is cited to file:line in the two
read-only checkouts (/home/user/ods, /home/user/hal0). Where a doc asserts
something the code does not do, it is called out as drift rather than repeated.
A. How ODS does it
A.1 Hermes is a pinned upstream container, not a fork
ODS runs nousresearch/hermes-agent:v2026.6.5
(ods/extensions/services/hermes/compose.yaml:8), version-pinned with an
HERMES_AGENT_IMAGE escape hatch and a documented bump ritual
(ods/docs/HERMES.md:142-173 + a bump-history table at :284-290). No upstream
code is forked; the whole integration is a packaging layer of five files.
The container runs gateway run (compose.yaml:153-155) rather than
dashboard, because the cron scheduler tick lives in the gateway main loop —
an earlier draft ran only the dashboard and cron jobs silently never fired
(compose.yaml:139-152). HERMES_DASHBOARD=1 embeds the React SPA in the same
process (compose.yaml:50-52), and HERMES_DASHBOARD_TUI=1 (:87) turns on
the PTY-backed chat tab that upstream defaults off.
Network posture is the load-bearing decision: Hermes declares expose: "9119"
and no ports: (compose.yaml:137-138), and the manifest pins
external_port_default: 0 specifically so dashboard-api cannot advertise a
non-existent direct URL (manifest.yaml:19-23). Health polls /api/status
because /api/health does not exist in the pinned image and everything else is
session-gated (manifest.yaml:24-29, mirrored in the container healthcheck at
compose.yaml:156-164). Resource caps, no-new-privileges, log rotation and
UID remapping to the host user round it out (compose.yaml:18-19,167-181).
A.2 Config: a template that gets surgically patched
cli-config.yaml.template is bind-mounted over the in-image
cli-config.yaml.example (compose.yaml:97), so Hermes's own entrypoint copies
ODS's defaults into /opt/data/config.yaml on first start. The template is
unusually well-reasoned:
model.base_urlwins over theOPENAI_BASE_URLenv var — empirically verified on macOS where the env var was ignored (cli-config.yaml.template:34-42).model.context_length: 131072andauxiliary.compression.context_lengthboth set, because Hermes checks the compression model's window against the 64K floor independently (:74-84).model.max_tokens: 1024to stop a looping local model monopolising the only inference slot (:51-57).providers.custom.request_timeout_seconds: 180for slow local prefill (:59-70).agent.disabled_toolsets: [terminal, browser]— a schema-level subtraction, chosen over prompt-only guidance because removing tools from the schema is the only reliable fix for wrong-tool selection on small models (:86-104).- Compression tuned to
threshold 0.75 / target_ratio 0.50 / protect_last_n 40after live testing showed a single 10-15 kB web-search result spiking past a 0.50 threshold and compressing away the conversation (:153-186). - WhatsApp pre-seeded disabled with
bridge_port: 3010purely to dodge upstream's port-3000 bridge colliding with Open WebUI (:121-151).
Because Hermes only copies the template on first start, upgrades need a
second writer: scripts/patch-hermes-config.py is a line-oriented YAML patcher
that converges only ODS-owned keys and leaves operator sections untouched
(patch-hermes-config.py:1-8). It has real nuance — _ensure_provider_timeout
only overwrites ODS's own shipped 180s default, never an operator's value
(:120-156); _ensure_whatsapp_bridge never flips an existing enabled state
(:220-262); _ensure_compression does converge every install to the tuned
values so existing boxes migrate on the next bootstrap-upgrade (:186-215).
Installer phase 11 drives it: it picks the model name per mode (switchboard
alias ods/current, Lemonade, external, or extra.$GGUF_FILE), picks
base_url (litellm on AMD/switchboard, llama-server otherwise), passes
--api-key when routing through LiteLLM, then verifies the substitution
took with a grep -Fqx and warns loudly if it did not
(installers/phases/11-services.sh:992-1094).
A.3 The 64K context floor
HERMES_MIN_CONTEXT = 65536 is a shared constant
(extensions/services/dashboard-api/context_policy.py:3), declared per-service
in the manifest (hermes/manifest.yaml:37-45 — llm.min_context: 65536,
route: gateway, plus a custom probe against /api/talk/message/stream with
auth: cookie:dream-session), consumed by the dashboard readiness surfaces
(dashboard-api/main.py:537-587, performance_oracle.py:1499), and — crucially
— enforced by raising the whole box's context at install:
installers/phases/03-features.sh:93-102 bumps MAX_CONTEXT to 65536 when
Hermes is enabled and appends the reason to the model recommendation. macOS has
the same logic (installers/macos/install-macos.sh:1534-1539).
A.4 SOUL.md and the generated "About this installation" block
SOUL.md.template is a consumer-grade persona written for text-to-speech
delivery — length budgets, "speak in sentences not lists", literal-output
honouring, and hard tool-selection rules that exist because operators watched
models stall chat for minutes reaching for a shell on weather questions
(SOUL.md.template:103-120). It contains an eight-step script teaching the
agent to drive OAuth skill setup end-to-end (:26-60).
The single line <!-- INSTALLATION_CONTEXT --> (:3) is replaced at build time
by scripts/build-installation-context.py. That script builds a
{container_name: service_id} index by regexing every extension manifest
(:86-127), runs docker ps --filter name=ods- and collapses compound
containers (ods-langfuse-clickhouse → langfuse) to parent service ids
(:130-168), best-effort asks llama-server/Lemonade what model is actually
loaded (:171-188), and renders a bounded Markdown block that explicitly tells
the model not to run tool calls to second-guess it (:277-297). It also
distinguishes the ODS dashboard from Hermes's own :9119 because the model has
a strong prior to name the wrong one (:270-287). A local-lemonade profile
emits a much shorter variant for prompt-constrained backends (:328-377), and
build_soul self-heals the pathological case where Docker auto-created
data/persona/SOUL.md as a directory (:406-422).
Refresh path: ods-cli's _ods_cli_refresh_soul regenerates the file and then
docker exec ods-hermes cp /opt/hermes/docker/SOUL.md /opt/data/SOUL.md
(ods-cli:1691-1719) — a container-side copy chosen because macOS Docker
Desktop's virtiofs refuses nested bind mounts (compose.yaml:98-127). It runs
both before compose (to repair a directory-shaped SOUL.md that would fail the
mount) and after (so docker ps sees real state) — ods-cli:1582-1612,
mirrored at installers/phases/11-services.sh:1108-1123 and :1293-1300.
A.5 Auth: magic links, forward_auth, admin-session
hermes-proxy is a 50 MB Caddy sidecar (hermes-proxy/compose.yaml:6). Its
Caddyfile keeps /health, /healthz, /favicon.ico and /auth/required*
public inside an explicit route {} block (order matters — global directive
order would otherwise run forward_auth first), then forward_auths everything
else to dashboard-api/api/auth/verify-session
(hermes-proxy/Caddyfile:54-129). Two hard-won details: an earlier draft only
checked that the cookie header existed, trivially bypassable (:8-12); and
the WebSocket upgrade headers must be stripped from the auth sub-request or
FastAPI 403s a valid session (:104-119). Denials 303-redirect (not 302) so a
no-cookie POST does not replay its body (:96), and the redirect needs an
explicit * matcher or Caddy misparses it (:123-127).
The credential itself is a stateless HMAC cookie:
<random-id>.<expiry-epoch>.<base64url-HMAC-SHA256>
(dashboard-api/session_signer.py:1-45), verified with hmac.compare_digest
and an independent server-side expiry check (:125-171). An unset
ODS_SESSION_SECRET disables issuing entirely rather than silently signing with
an empty key (:60-66,109-114). verify-session is deliberately not
API-key-gated (the cookie is the credential) and returns a byte-identical 401
for every failure mode (routers/auth.py:65-107).
POST /api/auth/admin-session trades DASHBOARD_API_KEY for an identical
cookie so the box owner is not locked out of their own services
(routers/auth.py:110-180).
Cookies are minted by magic-link redemption (routers/magic_link.py:1-46):
token_urlsafe(32), only the SHA-256 hash persisted, single-use by default,
60-minute guest expiry, per-IP rate limit on redemption, and identical 404s for
invalid/expired/redeemed so a holder cannot fingerprint state. The dashboard's
/invites page exposes owner cards and scoped guest invites (chat vs
hermes) with QR printing (dashboard/src/pages/Invites.jsx:23-31).
A.6 ODS Talk: the server-side bridge that is the best code in the repo
Phones never see Hermes. dashboard-api/hermes_bridge.py scrapes the dashboard
token out of Hermes's HTML (TOKEN_RE, :43, _fetch_hermes_token:99-113),
opens the JSON-RPC WebSocket on the Docker network, and returns only simplified
results. Two architectural findings are baked in as comments:
- Hermes scopes streaming events to the WS that owns the session, so
session.createandprompt.submitmust happen on the same socket (hermes_bridge.py:9-15). - Therefore a per-cookie connection pool keeps one long-lived WS + session
per phone, so llama-server's KV cache stays warm for the ~16 k-token agent
system prompt — the second message costs ~1 k tokens of prefill, not 17 k
(
:17-25,_HermesConnection:162-176).
The pool is genuinely careful: _POOL_GUARD is held only for dict bookkeeping
while the network open happens under a per-key lock, so one cold open cannot
head-of-line-block every other phone (:236-270); an idle sweeper evicts after
ODS_TALK_IDLE_EXPIRY (300 s) but skips connections whose lock is held
(:279-311); and exactly one failure mode retries transparently —
HermesConnectionStale, raised only when send_str fails before Hermes
accepted the prompt, so a retry can never duplicate tool calls (:63-73,
:471-528).
routers/talk.py wraps that in SSE with a 5 s keepalive comment frame (iOS
Safari closes idle streams during 30-60 s cold prefill) and client-disconnect
cancellation so an abandoned tab does not pin an inference slot (:444-460). A
_TOOL_LABELS table maps Hermes tool names to spinner captions with an honest
Using `<name>`… fallback (:404-442). /api/talk/status composes Hermes,
whisper and TTS health plus model compatibility into a capabilities dict
including live_mic_requires_secure_context (:560-590).
A.7 OAuth passthrough
routers/oauth_passthrough.py closes the "copy the code from your URL bar"
gap. POST /api/oauth/init mints a nonce bound to {skill_id, return_url};
the provider redirect lands on the unauthenticated /api/oauth/callback, which
validates and consumes the nonce before writing {code, state, ts} to
data/persona/oauth_callback.json for the agent to pick up (:1-66). The
return_url is bound at init, eliminating the open-redirect surface, and
skill_id is resolved server-side so an attacker cannot steer which skill the
code is exchanged against (:52-62). oauth-providers.json is the operator-
facing registry of flows, credential filenames and acceptable redirect URIs,
with an explicit "prefer PKCE / device flow so local appliances need no shared
secret" stance (oauth-providers.json:22-51).
A.8 The rest of the agent surface
dashboard-api/routers/agents.pyis a red herring — 77 lines of GPU/ session/throughput metrics (:14-77) backed byagent_monitor.py, which itself admitstokens_per_secondandqueue_depthhave no data source (agent_monitor.py:27-30). Real Hermes control lives intalk.py,auth.py,magic_link.pyand the Hermes SPA itself.- APE (
extensions/services/ape/main.py) is a competent standalone policy gateway: intent classification, allowlists, path guards, deny regexes, sliding windows (5 m/1 h/1 d), a circuit breaker, warmup grace, persisted state under a file lock, andrequire_approvalescalation that deliberately does not raise so a framework can route to a human and retry via/approve(main.py:1-56,187-200,793-1072). It is not wired to Hermes (see E). ods/agents/templates/is five OpenClaw-era YAML agent templates with a README dated 2026-02-11 (agents/templates/README.md:1-24); nothing outside the files themselves references them.memory-shepherd/is a systemd-timer baseline-reset tool with a genuinely good idea (operator-owned baseline above a---separator, scratch below, archived and reset every 3 h —README.md:14-51) but its shipped config still points atconfig/openclaw/workspace/*.md(memory-shepherd.conf:19-35). It is not wired to Hermes.- OpenCode runs as a host user unit on loopback 3003, deliberately
UnsetEnvironment=OPENCODE_SERVER_PASSWORD(opencode/opencode-web.service:16-19), configured by phase 07 which writesopencode.jsonpointing at LiteLLM (withLITELLM_KEY) on Lemonade or llama-server direct (no-key) otherwise, with a jq-rewrite path and a deterministic fresh-write fallback (installers/phases/07-devtools.sh:140-219). The generated-config writer matrix indocs/INSTALLER-ARCHITECTURE.md:118-135lists every place each surface is written across Linux/macOS/Windows/upgrade — the single most transferable process artifact in the repo.
A.9 ODS doc drift found
docs/HERMES.md:110states compressionthreshold 0.50 / target_ratio 0.20. The shipped template is0.75 / 0.50 / 40(cli-config.yaml.template:171-186) and the patcher converges to the same (patch-hermes-config.py:207-212).docs/HERMES.md:115claims "4 CPUs / 4GB RAM hard limit". The compose default iscpus: ${HERMES_CPU_LIMIT:-1.0}(compose.yaml:170).docs/MIGRATION-OPENCLAW-TO-HERMES.md:28claims an "APE policy plugin (pre_tool_call hook) routes every tool call through ODS's policy engine".pre_tool_callappears nowhere else in the repository — that doc line is its only occurrence — anddocs/HERMES.md:140says the opposite ("No APE policy enforcement yet"). The migration table is marketing, not code.- Same table,
:27, claims voice is "wired through ODS's whisper + kokoro out of the box". True at the ODS Talk portal layer (routers/talk.py:279-368), not inside Hermes;docs/HERMES.md:182still lists it as unverified roadmap. cli-config.yaml.template:20-27admits the model-name substitution is an "open follow-up" requiring hand-edits on AMD/Apple — butinstallers/phases/11-services.sh:1068-1094has shipped it. The comment is stale.
B. How hal0 does it today
hal0 runs Hermes as a pip-installed wheel in a hal0-owned venv under a
sandboxed systemd template unit, not a container. BUNDLED_AGENTS is
("hermes", "pi") with single-pick enforced in the manager, not the API or CLI
(src/hal0/agents/manager.py:117, :28-31).
Unit. installer/systemd/hal0-agent@.service is Type=notify +
WatchdogSec=60 (:21-26,64), User=hal0, NoNewPrivileges,
ProtectSystem=strict, ProtectHome, PrivateTmp, the full Protect*/
Restrict* set (:69-77), RuntimeDirectory/LogsDirectory created before
the namespace is built (:79-87), and StartLimitBurst=5/120s (:16-19).
The ReadWritePaths= comment is a model of honesty: it spells out exactly what
the boundary does not buy — /etc/hal0 is hal0-owned 2775, so the agent can
rewrite api.env and upstreams.toml, and %i.env was left 0644 on every
upgraded box by #1876 (:89-130). The hermes drop-in sets HERMES_HOME,
HERMES_DASHBOARD_TUI=1, HAL0_INFERENCE_BASE, and an
ExecStartPre=-/usr/local/bin/hal0-agent %i render-context so every restart
refreshes live context (override.conf:11-49). HERMES_WEB_DIST is resolved at
runtime from the venv's real site-packages after a hardcoded python3.12 path
crash-looped the unit on 3.14 hosts (override.conf:26-35).
Shim. src/hal0/cli/agent_shim.py is pure-stdlib by design so it cannot be
broken by hal0 wheel import drift, translates serve into
hermes dashboard --tui --skip-build --no-open --host 127.0.0.1, and owns the
sd_notify handshake (:1-33).
Provisioner. hermes_provision.py is 6 769 lines. It is explicitly not
checkpointed any more — idempotency comes from every write being a converging
write, and provision.json is a last-run snapshot (:1-21). The pipeline is 12
steps (_INSTALL_STEPS:6583-6595). config_write is the standout: it runs
hermes config migrate (Hermes owns and schema-migrates its own file), then
layers hal0's keys via hermes config set, then deep-merges only the two
irreducible list keys plus an operator overrides.yaml — never a wholesale
rewrite (:2290-2308). It snapshots to config.yaml.bak first (:2318-2323)
and derives the terminal-tool posture from pre-migrate content so a fresh box
cannot be mistaken for an existing opt-in, failing closed on an unrecordable
fresh opt-in (:2324-2371).
Context files. _phase_context_link renders SOUL.md, /etc/hal0/AGENTS.md,
MCP-CLIENTS.md from Jinja templates with observable fallbacks (:3247-3330).
Live state is separated from stable identity: STATE.md.j2 is a deliberately
lean volatile snapshot with an _as_of: stamp (STATE.md.j2:1-29), injected
every session by inject-system-state.sh, which cats the pre-rendered file and
only kicks a detached background refresh when it is older than 300 s — never
blocking the session (inject-system-state.sh:20-30). hermes_refresh.spawn_context_refresh
fires the same refresh from the daemon on slot swap / capability apply without
blocking the event loop (hermes_refresh.py:1-31).
Context floor. anchor_window.py reads Hermes's own
MINIMUM_CONTEXT_LENGTH out of the installed venv rather than hard-coding it,
falling back to HERMES_MINIMUM_CONTEXT_LENGTH = 64_000 with a drift test
(:64-84). It computes effective = min(model window, slot ceiling), asks the
by-id route because hal0/agent is a virtual name that may resolve elsewhere,
treats "unknown" as not a pass, and is deliberately read-only — it names the
slot, its ceiling and the repair command instead of silently rewriting operator
config (:1-50).
Discovery. hal0 ships two real Hermes plugins rather than pointing a base
URL: hal0-provider is a ProviderProfile advertising hal0-api with live
/v1/models discovery (X-hal0-Model-Filter: hal0, alias-stripping, no cache)
and default_aux_model = "hal0/agent" so retargeting a role hot-swaps with no
gateway restart (installer/agents/hermes/plugins/hal0-provider/profile.py:1-27);
hal0-memory is an exclusive memory provider wiring Hermes durable memory to
Hindsight through /api/memory/* (plugins/hal0-memory/plugin.yaml).
Chat surface. chat_proxy.py bridges the browser to loopback Hermes over
JSON-RPC WS; WS upgrades are gated by an Origin allowlist + HMAC session cookie,
the outbound hop carries the runtime.json embed token as a Bearer header
(never a query string), tool.progress is coalesced at 100 ms while
message.delta passes through untouched, and reconnect is the browser's job
(:1-40). _auth.py documents why: an unauthenticated 0.0.0.0:8080
JSON-RPC bridge into Hermes's tool surface is LAN-RCE (:1-33); the cookie is
HMAC-SHA256 over a JSON payload with an 8 h TTL and a 0600 secret at
/var/lib/hal0/agents/secret.bin (:60-80).
Policy. mcp_client.py is a transport-agnostic policy layer with a
three-tier verdict (allow/gated/blocked, plus unknown_server/
unknown_tool) that the wire client consults (:1-45). Gated calls land in
ApprovalQueue, surfaced by /api/agent/approvals with an SSE stream that
replays the pending set on subscribe (api/routes/approvals.py:1-27).
personas.py owns a TOML store with active.txt, conservative default
auto-approve/require-approval globs, and a distinct brain-profile memory
identity (:1-55).
Operator surfaces. api/routes/agents.py gives list / persona-enums /
skills / install / uninstall / activity (:68-347); restart.py, and
memory_stats.py back the sidebar chips; api/plugins/manifest_proxy.py
reverse-proxies Hermes plugin manifests while stripping Authorization/Cookie
inbound, injecting X-hal0-Agent outbound, enforcing SRI hashes and rejecting
traversal (:1-36). The v3 dashboard renders agents as trading-card style
agent-cards with live health, throughput and restart wired to real endpoints
(ui/src/dash/agents/agents-overview.jsx:1-20). Test coverage is substantial:
30 files under tests/agents/, 56 agent-related test modules overall.
B.1 hal0 doc drift found
ARCHITECTURE.md:322-329describes a "15-phase" pipeline includingpersona_seed,namespace_register,model_automapandself_report._INSTALL_STEPShas 12 entries;model_automapis gone and the other three relocated to the hal0-api lifespan (hermes_provision.py:6570-6595).ARCHITECTURE.md:325says "Idempotent + checkpointed via …provision.json". The module says the opposite — checkpoints are gone, the file is a snapshot (hermes_provision.py:9-21).ARCHITECTURE.md:272-274saysBUNDLED_AGENTS = ("pi-coder", "hermes"); code says("hermes", "pi")(manager.py:117).- The unit and its drop-in both point
Documentation=atdocs/agents/hermes/SERVICE.md(hal0-agent@.service:8,override.conf:9); that path does not exist in the tree. installer/agents/hermes/plugins/hal0-memory/provider.pyandsrc/hal0/agents/hermes/plugins/memory_hindsight/provider.pyare byte-identical 629-line copies with no visible sync mechanism.hermes_provision.py:976-979names the installer tree as canonical.
C. Honest scorecard
ODS is genuinely better at:
- Auth as a product feature. Signed stateless cookie +
forward_auth+ magic links + owner cards + admin-session bootstrap is a complete, proxy-agnostic story (session_signer.py,routers/auth.py,routers/magic_link.py,hermes-proxy/Caddyfile). hal0 has one HMAC seam on the chat proxy and explicitly unauthenticated approval endpoints (approvals.py:24-27,46-48) that executemodel_pull/slot_delete/config_write. That is the single largest gap. (Update 2026-10-04: #2246 on main,946c512, closes the LAN case — ADMIN routes, approvals included, now require an admin session/key from any non-loopback caller when the box is LAN-bound, even withrequire_authoff; the loopback dev-open path remains by design. The product-feature gap — magic links, owner cards, admin-session — stands.) - A mobile chat portal that never exposes the agent.
hermes_bridge.py+talk.pyis better engineering than anything on either side: per-cookie WS pooling for prompt-cache warmth, one narrowly-scoped transparent retry, SSE keepalive, disconnect cancellation, tool-name spinner captions. - Persona craft.
SOUL.md.templateencodes hard-won behavioural rules (TTS-shaped output, literal-output honouring, "web_search first, never curl") that hal0's admin-flavouredSOUL.md.j2does not attempt. - Agent-driven OAuth.
oauth_passthrough.py+oauth-providers.json+ the SOUL.md script is a complete capability hal0 lacks entirely. - Cross-platform config-writer discipline. The generated-config-writers
matrix (
docs/INSTALLER-ARCHITECTURE.md:118-135) is a process artifact hal0 would benefit from copying verbatim. - Doc density for operators — bump history, troubleshooting per symptom,
explicit "known limitations" tables (
HERMES-SSO.md:129-146).
hal0 is genuinely better at:
- Provisioning rigour. Converging writes over checkpoints,
config.yaml.bak,hermes config migrate+config set(working with upstream's own config surface instead of regexing YAML), operatoroverrides.yaml, and repair-mode ownership reconciliation.patch-hermes-config.pyis a clever hack; hal0'sconfig_writeis the right answer. - Context-floor correctness. Reading the floor from the installed Hermes,
resolving the effective window through the by-id route, and refusing to
treat unknown as a pass (
anchor_window.py:1-50) is strictly better than ODS's hard-coded 65536 and auto-bump ofMAX_CONTEXT. - Live-state injection. A lean
STATE.mdrefreshed on restart, on slot swap, and lazily on a 300 s TTL beats regenerating a large SOUL.md anddocker exec cp-ing it. - Sandboxing. The systemd unit's
Protect*set plus the brutally honestReadWritePathscomment beats a container withno-new-privilegesand full bridge-network egress (docs/HERMES.md:139). - Deep integration. Real Hermes plugins (provider + memory) with
restart-free role aliases and live model discovery, versus ODS's static
model.defaultstring that needs an installer patcher. - Policy that is actually wired. The MCP allow-list → approval-queue path is real code; APE is a fine engine attached to nothing.
- Test coverage. 30 test modules under
tests/agents/, including idempotency, upgrade, ownership and security-deliverable suites.
Equivalent: image/wheel pinning with human-gated bumps (HERMES.md:142-173
vs [tool.hal0.upstream-hermes] + the weekly drift action); dashboard agent
cards; per-agent restart; upstream-owns-runtime posture.
D. Port candidates
D1. Signed-cookie session + verify-session + admin-session (highest value).
Source: ods/extensions/services/dashboard-api/session_signer.py (171 lines),
routers/auth.py (180 lines). Target: src/hal0/api/auth/session_signer.py +
src/hal0/api/routes/auth.py. hal0 already has HMAC cookie machinery in
api/agents/_auth.py:60-120 — the port is generalising it off the agent
namespace and adding the two endpoints so any reverse proxy can gate on it.
Shape worth copying verbatim:
def verify(cookie_value: str) -> Tuple[bool, str]:
if not _SECRET: return False, "no-secret"
parts = cookie_value.split(".")
if len(parts) != 3: return False, "malformed"
random_id, expiry_str, claimed_sig = parts
if not hmac.compare_digest(_sign(f"{random_id}.{expiry_str}").encode(),
claimed_sig.encode()):
return False, "bad-signature"
if int(expiry_str) <= int(time.time()): return False, "expired"
return True, "ok"
Size ~250 lines + tests. Dependencies: none (stdlib). Risk: low. Immediate
win — put /api/agent/approvals/* behind it.
D2. Caddy forward_auth sidecar. Source: hermes-proxy/Caddyfile:54-129,
compose.yaml, auth-required/index.html. hal0 has no container layer, so port
the Caddyfile and run Caddy as another systemd unit in front of hal0-api on
the LAN port. The three non-obvious lines to copy exactly: the explicit
route {} wrapper, the six header_up -Sec-Websocket-*/-Upgrade/-Connection
strips, and redir * /auth/required 303. Size ~150 lines of config. Depends on
D1. Risk: medium (changes the LAN trust model — see F).
D3. OAuth passthrough + provider registry. Source:
routers/oauth_passthrough.py (570 lines), hermes/oauth-providers.json,
SOUL.md.template:26-60. Target: src/hal0/api/routes/oauth.py +
installer/agents/hermes/oauth-providers.json + an addendum to
hermes_templates/SOUL.md.j2. Size ~600 lines + a persona section.
Dependencies: a shared operator-owned directory both hal0-api and the agent can
read — hal0 already has /var/lib/hal0 and both run as hal0, which is
simpler than ODS's cross-container arrangement. Risk: low-medium; keep
the nonce-consumed-before-write ordering and the init-bound return_url.
D4. The pooled WS bridge pattern. Source: hermes_bridge.py:162-528. Even
though hal0's chat_proxy.py proxies the browser's own WS rather than pooling
server-side, the findings transfer directly: same-WS session create+submit,
warm-prompt-cache pooling, per-key opening locks, and — most importantly — the
HermesConnectionStale discipline of retrying only before prompt.submit
lands. If hal0 ever adds a REST/SSE chat surface (a phone portal), this is the
blueprint. Size ~400 lines. Risk: medium.
D5. Tool-name → spinner caption table. Source: routers/talk.py:404-442.
Target: ui/src/dash/agents/ or the chat proxy. ~40 lines, zero dependencies,
risk none. Copy the honest fallback:
return f"Using `{tool_name}`…"
D6. Generated-config-writers matrix. Source:
docs/INSTALLER-ARCHITECTURE.md:118-135. Target: a section in hal0's
ARCHITECTURE.md or CONTRIBUTING.md listing every writer of config.yaml,
hermes.env, the gateway drop-in, personas and runtime.json across
install / reprovision / repair / upgrade. Documentation only; risk none; high
value given hal0 has at least four writers of Hermes state.
D7. ods doctor / ods repair hermes-workers pattern. Source:
scripts/ods-doctor.sh:479-489, scripts/prune-hermes-slash-workers.sh,
policy env knobs HERMES_SLASH_WORKER_MAX_COUNT/MAX_AGE_SECONDS. hal0's
watchdog restarts a hung agent but nothing prunes leaked children. Size ~120
lines. Risk: low. Explicitly manual, never automatic — copy that stance too.
D8. Persona voice guidance. Source: SOUL.md.template:18-24,77-83,103-120.
Target: a voice/concise persona TOML in hal0's persona store rather than
overwriting the admin SOUL.md.j2. ~80 lines of prose. Risk none.
E. Do not copy
ods/extensions/services/ape/. The engine is decent, but it is wired to nothing —pre_tool_callappears only in a doc, anddocs/HERMES.md:140concedes there is no enforcement. Its defaultWriteFile.allowed_pathsstill names/home/node/.openclaw/workspace(ape/main.py:127-140). hal0'smcp_client.pythree-tier classifier plusApprovalQueuealready occupies this slot and is actually on the call path. Cherry-pick only the ideas — sliding-window tiers, circuit breaker,require_approvalas advisory non-raising escalation (main.py:36-52).ods/agents/templates/*.yaml. OpenClaw-era, unreferenced by any code, validated once in Feb 2026 against a model neither project ships.memory-shepherd/as shipped. The baseline/---/scratch pattern is worth stealing conceptually, but the shipped config targets OpenClaw workspace paths (memory-shepherd.conf:19-35) and the whole thing is orthogonal to Hermes's own memory. hal0's Hindsight namespaces already solve durable memory; bolting on a periodic reset would fight it.patch-hermes-config.py's regex YAML surgery. hal0'shermes config set+ targeted deep-merge is strictly better. Do not regress.dashboard-api/routers/agents.py+agent_monitor.py. Metrics with no data source (agent_monitor.py:27-30) rendering an HTML fragment from an f-string. hal0's/api/agents+memory_stats+activityare better.- Hard-coding
HERMES_MIN_CONTEXT = 65536(context_policy.py:3) and auto-raisingMAX_CONTEXTat install (03-features.sh:93-102). hal0's read-it-from-the-installed-Hermes + refuse-to-mutate posture is the correct one; adopting ODS's would be a regression. HERMES_DASHBOARD_INSECURE=1(compose.yaml:59). Only defensible because 9119 is unbound. hal0's Hermes is already loopback-bound and does not need it.- Scraping
window.__HERMES_SESSION_TOKEN__out of HTML (hermes_bridge.py:43). hal0 has a realruntime.jsonembed token; regex-ing the SPA is a fragile fallback, not a pattern.
F. Open questions for the owner
- Container Hermes vs pip Hermes. ODS gets a hermetic multi-GB image with
playwright/ffmpeg/chromium and one-line rollback; hal0 gets systemd
sandboxing, no Docker dependency, and direct loopback access, at the cost of
a
git clone+ venv build and an OS toolchain preflight (installer/agents/hermes-prereqs.sh:1-40). Recommendation: keep pip. The container's isolation is weaker than the unit'sProtectSystem=strict, and ODS itself had to punchHERMES_DASHBOARD_INSECURE=1through it. - Trusted-LAN vs authenticated. hal0's standing decision is "LAN trust plus
an upstream reverse proxy own authentication" (
ARCHITECTURE.md:405-408), yet/api/agent/approvals/{id}/approveexecutes gated tools unauthenticated (approvals.py:24-27). Either the standing decision needs a carve-out for mutation endpoints (D1 alone), or hal0 adopts the full ODS posture (D1+D2). This is the decision that gates D2, D3 and D4. (#2246 answered the first half on main with a posture-coupled gate for LAN-bound boxes rather than a blanket carve-out; the D2–D4 question is untouched.) - Do you want a phone surface? ODS Talk (magic link → mobile portal → pooled bridge → TTS) is roughly three of the port candidates and the single biggest feature difference. If the answer is no, D2/D3/D4 drop to nice-to-have and D1 alone is sufficient.
- Whose persona wins? ODS's SOUL is a consumer voice assistant; hal0's is a homelab admin. Porting D8 as a persona option rather than a replacement keeps both, but someone must decide the default.
- Policy engine: build up
mcp_clientor adopt an APE-shaped sidecar? Recommendation: extendmcp_client.pywith APE's window tiers and circuit breaker rather than running a second service. - Fix hal0's own drift first. The five items in B.1 — especially the duplicated 629-line memory-provider tree and the stale 15-phase description — will mislead anyone doing this porting work. They are cheap to fix and should land before any port.
02-extensions.md Extension system
ODS extension system → hal0: a source-level comparative study
Scope: the ODS extension mechanism (/home/user/ods/ods/), verified against source,
and a concrete proposal for expressing the same shape in hal0's systemd/podman world
(/home/user/hal0/). All line numbers are from the trees as read on 2026-09-05.
A. How ODS does it (mechanism-level)
A.1 The unit of extension is a directory with a manifest
Every ODS service — bundled or user-installed — is one directory whose only
mandatory contents are manifest.yaml and (for non-core services) compose.yaml.
/home/user/ods/ods/extensions/README.md:51 states the contract plainly:
The core contract is simple: a service manifest describes what the service is, and a compose fragment describes how it runs. The registry, CLI, dashboard, health checks, and compose resolver discover the service from those files.
That is literally true in code. There are four independent walkers over
extensions/services/*/ (plus data/user-extensions/*/), each doing its own
manifest read, none of them requiring registration anywhere else:
| Walker | File:line | What it produces |
|---|---|---|
| Bash service registry | /home/user/ods/ods/lib/service-registry.sh:5,50 |
14 associative arrays: aliases, container names, compose paths, categories, deps, health paths/timeouts, ports/port-envs, host-network flag, display names, setup hooks, GPU backends |
| Compose resolver | /home/user/ods/ods/scripts/resolve-compose-stack.sh:517 |
the -f a.yml -f b.yml … flag string for docker compose |
| Dashboard API | /home/user/ods/ods/extensions/services/dashboard-api/config.py:297 |
SERVICES dict (health checks, quicklinks) + FEATURES list |
| Catalog generator | /home/user/ods/ods/scripts/generate-extensions-catalog.py:123 |
config/extensions-catalog.json for the dashboard Extensions page |
All four gate on the same sentinel, schema_version: ods.services.v1
(service-registry.sh:136, resolve-compose-stack.sh:540, config.py:336), and all
four accept manifest.yaml | manifest.yml | manifest.json
(service-registry.sh:123-127, resolve-compose-stack.sh:524-528, config.py:319).
A.2 The manifest contract (extensions/schema/service-manifest.v1.json)
Root: schema_version (const), optional compatibility{ods_min,ods_max}, service
(required), features[], tags[]. The schema is additionalProperties: true
throughout, so forks add x_-prefixed fields without breaking validation
(docs/SERVICE_MANIFEST_V2_PLAN.md:79-82). Its library copy
(extensions/library/schema/service-manifest.v1.json) is byte-identical — verified by
diff — and kept so by scripts/sync-manifest-schema.py.
Per-field consumers (the load-bearing ones):
id— registry key, compose-service name, CLI arg, container-name defaultods-<id>(service-registry.sh:158), directory name (enforced byscripts/audit-extensions.py:463service-id-directory-mismatch).aliases[]— CLI shorthand;sr_resolve()also strips a leadingods-so a name copied out ofdocker psresolves (service-registry.sh:263-269).port/external_port_env/external_port_default—SERVICE_PORTSseeded from the manifest, then overridden from.envbysr_resolve_ports()(service-registry.sh:239-247); the same resolution in Python atconfig.py:371-377.health/health_timeout— the generic probe URL is built ashttp://127.0.0.1:${port}${health}inscripts/health-check.sh:159-188.category(core|recommended|optional) —coremeans "defined indocker-compose.base.yml, no compose fragment, cannot be disabled" (ods-cli:2762,ods-cli:2879).compose_file— relative path; the resolver rejects one that escapes the extension dir (resolve-compose-stack.sh:418-426) and treats a.disabledsuffix as "off".depends_on[]— drives the enable-cascade prompt (ods-cli:2805-2831), the reverse-dependent warning on disable (ods-cli:2881-2902), and installer-timevalidate_service_dependencies(installers/phases/11-services.sh:1133).gpu_backends[]— the resolver skips a manifest whose backends don't include the detected one, withall/noneas wildcards (resolve-compose-stack.sh:544-547);ods enablewarns-and-prompts instead of refusing (ods-cli:2768-2803).hooks.{pre,post}_{install,start,uninstall}/ legacysetup_hook—SERVICE_SETUP_HOOKSprefershooks.post_install(service-registry.sh:196-203); installer phase 11 runs every non-empty hook with$1=INSTALL_DIR $2=GPU_BACKEND(installers/phases/11-services.sh:1374-1386). Live example:extensions/services/langfuse/hooks/post_install.shchowns bind mounts for the uid-70 postgres / uid-101 clickhouse images.startup_check/startup_timeout— one-shot CLI extensions whose container exits cleanly (service-registry.sh:186, dashboard_is_one_shot_extension,routers/extensions.py:317-325).host_network— services withnetwork_mode: hosthave no mapped port and no HTTP health; the flag switches off both checks (service-registry.sh:38-42,audit-extensions.py:489-503).ui_path/external_link/public_url_env— the dashboard sidebar quicklink, built entirely from manifests atextensions/services/dashboard-api/main.py:1567-1583.llm{consumes,route,pinning,min_context,probe}— the swap-safety contract (docs/SWAP-SAFE-EXTENSIONS.md:38-72).route: gatewaymeanshttp://litellm:4000/v1+ modelods/current; the schema conditionally requiresroute,pinningandprobeonceconsumes: true.features[]— dashboard feature cards. Requirements split into runtime health (requirements.services/services_any/vram_gb) and enablement (enabled_services_all/enabled_services_any), pluslaunch{type,service,path}andpriority.
Supporting docs: docs/EXTENSIONS.md (authoring + runtime lifecycle, 514 lines),
docs/COMPOSE_RESOLVER_CONTRACTS.md:38-58 ("Adding A Service" = exactly two files),
docs/SWAP-SAFE-EXTENSIONS.md (LLM tiers), docs/EXTENSION-PR-BRANCHING.md:9-14
(core vs. library branch targets), docs/BUILD-ON-ODS-SERVER.md:37-53 (fork guidance),
docs/SERVICE_MANIFEST_V2_PLAN.md (planning note only; v1 is the shipped schema).
A.3 The compose layer model
scripts/resolve-compose-stack.sh is 772 lines, of which ~700 are one embedded Python
program. Order of assembly:
- Base + hardware overlay (lines 54-122): a cascade selecting
docker-compose.base.ymlplus one ofnvidia|amd|apple|arc|intel|cpu|cloud| lemonade-external, keyed on--tier,--gpu-backend,--ods-mode, and env (LEMONADE_EXTERNAL,EXTERNAL_LLM_URL). - Multi-GPU overlay (143-146):
docker-compose.multigpu-<backend>.ymlwhen--gpu-count > 1. - Bundled extensions (517-586): for each
extensions/services/<id>/with a v1 manifest and a compatiblegpu_backends, appendcompose.yaml, then — by filesystem probe, not by manifest —compose.<backend>.yaml(563),compose.local.yaml(578, only in local/hybrid mode, non-Apple, non-external-LLM),compose.multigpu-<backend>.yaml(584). - User extensions (607-725): same walk over
data/user-extensions/*/, but every fragment is content-scanned first by_scan_user_compose_content(resolve-compose-stack.sh:270-406) — rejectingprivileged,build:,user: root, host net/pid/ipc/userns, dangerous caps,seccomp:unconfined,devices:, GPU reservations, docker-socket mounts, absolute bind mounts,extra_hosts,sysctls, reservedcom.docker.compose.*labels, non-loopback port bindings, and any service name colliding withconfig/core-service-ids.json. docker-compose.external-llm.yml(730-735) anddocker-compose.override.yml(742-748, also content-scanned).
Extension composes carry no networks: block for the bundled set (see
extensions/services/searxng/compose.yaml) — they land on the compose project's
default network and are reachable by their compose service name. The library
template does declare networks: [ods-network] with external: true
(extensions/library/templates/compose-template.yaml:64-92).
ods-cli caches the resolved string in $INSTALL_DIR/.compose-flags and revalidates
every referenced path before using it (ods-cli:852+); _regenerate_compose_flags
(ods-cli:755) re-runs the resolver whenever an extension is enabled or disabled.
A.4 Enable/disable is a file rename
ods enable <id> renames compose.yaml.disabled → compose.yaml; ods disable
stops the container then renames back (ods-cli:2732-2920, dispatched at
ods-cli:6567-6568). There is no state database. ods list derives status by asking
whether the compose path exists and appears in the currently active flag string
(ods-cli:3004-3065). The dashboard API does the same rename under flock
(docs/EXTENSIONS.md:437-444), and config.py:329-333 skips any extension directory
containing compose.yaml.disabled so a disabled service vanishes from health checks
and feature recommendations without touching its manifest.
.env feature flags (ENABLE_VOICE, ENABLE_WORKFLOWS, ENABLE_COMFYUI, …) are a
second, installer-only gate that works by performing the same rename.
installers/phases/03-features.sh:104-131 is explicit about why:
### Without this sync, an extension's compose.yaml is ALWAYS picked up by
### resolve-compose-stack.sh regardless of the ENABLE_* flag — the flag then
### only gates cosmetic things (image pre-pull, health checks, summary URLs)
### and the service still starts. Every optional service must be listed here
### or the user can't opt out of it.
The 21 _sync_extension_compose calls at lines 179-204 are hand-maintained. This is
the single biggest seam in an otherwise auto-discovering system.
A.5 Ports
config/ports.json is not a runtime input. Runtime ports come from the manifest
plus .env. ports.json exists only for tests/contracts/test-port-contracts.py,
which cross-checks each declared entry against .env.schema.json defaults,
.env.example defaults, the manifest's external_port_env/external_port_default,
and the actual compose port mapping (lines 106-195). Because the test iterates
ports.json entries — not manifests — a new extension with a new port env var passes
CI without being added. Adding it is opt-in rigour for bundled services.
A.6 Health checks
scripts/health-check.sh is fully registry-driven: test_service() (line 159) reads
SERVICE_PORT_ENVS, SERVICE_PORTS, SERVICE_HEALTH, SERVICE_HEALTH_TIMEOUTS,
checks the container state via docker inspect first, then curls
http://127.0.0.1:<port><health>. It loops all core services (line 304) and all
enabled extension services (369, gated on -f "$ext_compose") in parallel; a core
failure is critical (exit 2), an extension failure is degraded (exit 1).
Installer phase 12 (installers/phases/12-health.sh) is the exception: it does
sr_load + sr_resolve_ports (lines 23-30) and pulls ports/paths/container names from
the registry, but each probe is a hand-written line gated on an ENABLE_* flag
(e.g. line 616 for n8n). A new extension gets no installer-phase probe for free; it
does get scripts/health-check.sh and the dashboard's /api/services for free.
A.7 Dashboard surface
GET /api/features(routers/features.py:113) computes per-feature status from manifestfeatures[]+ live service health + GPU VRAM. Vocabulary (features.py:70-77):enabled(allenabled_services_allhealthy) →insufficient_vram(total VRAM <requirements.vram_gb) →services_needed(required services not healthy) →available. It also emitssuggestions[]("Your hardware can run X. Enable it?" / "X needs Y to be running.", lines 130-144) and coarserecommendations[]bucketed by VRAM (162-177).GET /api/extensions/catalog(routers/extensions.py:1090) serves the staticconfig/extensions-catalog.jsonenriched by_compute_extension_status(extensions.py:328-402):installing/setting_up/error/cli_installed/enabled/unhealthy/stopped/disabled/incompatible/not_installed. The React page renders exactly that vocabulary (extensions/services/dashboard/src/pages/Extensions.jsx:60-80,336-337).GET /api/external-links(main.py:1567) turns every manifest with an external port into a sidebar quicklink. A new extension appears in the sidebar with no UI change.- The dashboard plugin registry (
dashboard/src/plugins/registry.js) is the only place needing a code edit — and only for an extension that wants its own internal dashboard route, not for a quicklink.
Install flow for catalog extensions: POST /api/extensions/{id}/install copies from
$ODS_DATA_DIR/extensions-library/<id>/ to $ODS_DATA_DIR/user-extensions/<id>/ after
a 50 MB size check and the compose security scan, then asks the host agent
(bin/ods-host-agent.py, POST /v1/extension/start) to bring it up
(docs/EXTENSIONS.md:425-455).
A.8 Pre-wiring: what an extension author must actually know
Four conventions, and only four:
- Docker DNS = compose service name.
open-webuireaches search ashttp://searxng:8080/search?q=<query>&format=json(docker-compose.base.yml:117), embeddings ashttp://embeddings:80/v1(127), STT ashttp://whisper:8000/v1(159), TTS ashttp://tts:8880/v1(164). - LLM = the gateway alias.
OPENAI_BASE_URL=http://litellm:4000/v1, modelods/current, key${LITELLM_KEY}(docs/SWAP-SAFE-EXTENSIONS.md:17-33). Direct routes are permitted but must declarepinning: dynamic+ a probe. In practice several bundled services take the direct route with a documented fallback chain —extensions/services/perplexica/compose.yaml:13:OPENAI_BASE_URL=${HERMES_LLM_BASE_URL:-${LLM_API_URL:-http://llama-server:8080}/v1}. - Ports bind loopback with an env override:
"${BIND_ADDRESS:-127.0.0.1}:${MY_PORT:-1234}:1234"— enforced for user extensions by the resolver scan (resolve-compose-stack.sh:315-333). - Data lives at
./data/<id>/, config at./config/<id>/— seeextensions/services/n8n/compose.yaml:24-25.
Templates encode all of this:
extensions/library/templates/service-template.yaml (151 commented lines),
compose-template.yaml, compose-gpu-swap.yaml, compose-gpu-only.yaml,
dashboard-plugin-template.js.
A.9 Three worked examples
searxng — 3 files, 55 lines of contract. manifest.yaml (22 lines: id, alias
search, 8080→SEARXNG_PORT:8888, health: /healthz, gpu_backends: [all],
category: recommended, no features), compose.yaml (33 lines: pinned image,
SEARXNG_SECRET with a :? guard, one ./config/searxng bind mount, loopback port,
healthcheck, cpu/mem limits), README.md. Touched by: registry, resolver,
ods enable search, generic health check, /api/external-links, ENABLE_SEARXNG
(03-features.sh:180), and — as a consumer — docker-compose.base.yml:117 and
perplexica/compose.yaml:11. Note the coupling inversion: SearXNG knows nothing about
its consumers; they name it.
n8n — 5 files, one feature card. manifest.yaml (54 lines) adds
health_timeout: 15, four documented env_vars (two required, one secret), and a
features[] block (workflows, icon Workflow, launch.type: service, priority 4).
compose.yaml (46 lines) runs as ${ODS_UID}:${ODS_GID} with a custom entrypoint,
bind-mounting two helper scripts from the extension dir itself
(./extensions/services/n8n/n8n-entrypoint.sh:/opt/ods/…:ro,z). Extra surfaces: the
/api/features card, a hard-coded enable-instruction block
(routers/features.py:236), an installer probe (12-health.sh:616), an
ENABLE_WORKFLOWS sync (03-features.sh:184), a workflow catalog under
extensions/library/workflows/n8n/, and a config/n8n mount.
comfyui — the GPU-only pattern. compose.yaml is a 14-line stub services: {}
whose only job is to exist so the registry counts ComfyUI as installed; the real
definitions live in compose.nvidia.yaml (local build: + GPU reservations),
compose.amd.yaml (ROCm image, /dev/dri + /dev/kfd,
group_add: [$VIDEO_GID, $RENDER_GID], MIOpen tuning) and two compose.multigpu-*
overlays. The manifest's gpu_backends: [amd, nvidia] makes the resolver drop it
entirely on CPU/Apple. Non-manifest coupling: 03-features.sh:82-91 force-disables it
on Tier 0/1 (shm_size: 8g), 11-services.sh:1167 adds it to the local-build list.
It is the most-coupled extension in the bundled set — and still needs no registry edit.
B. Minimal-extension proof
The claim holds. For a service that runs from a pre-built image on the current GPU backend, the true minimal set is:
extensions/services/<id>/manifest.yaml ~20-32 lines
extensions/services/<id>/compose.yaml ~30-50 lines
Empirically, the smallest complete shipped examples:
| Extension | manifest | compose | other |
|---|---|---|---|
extensions/library/services/chromadb/ |
33 | 31 | README only |
extensions/library/services/gitea/ |
32 | 49 | README only |
extensions/services/searxng/ |
22 | 33 | README only |
Zero edits to any other file. No registry list, no CLI case statement, no compose
include, no UI import, no ports.json row, no .env.schema.json property.
Everything that auto-picks it up from those two files:
lib/service-registry.sh:50— all 14 lookup tables, on nextsr_load.scripts/resolve-compose-stack.sh:517— compose-fflag, plus anycompose.{nvidia,amd,apple,cpu}.yaml,compose.local.yaml,compose.multigpu-*.yamlsiblings, purely by filename.ods list(ods-cli:3004) — appears with category + enabled/disabled status.ods enable|disable|purge(ods-cli:2732,2859,2922) — including alias resolution, dependency cascade, GPU-compat prompt, and Dockerfile-build hint.ods start|stop|restart|logs|shell <id-or-alias>— viasr_resolve/sr_container.scripts/health-check.sh:369— parallel extension probe, JSON output, exit code.extensions/services/dashboard-api/config.py:297—SERVICESentry ⇒/api/services,/api/status, per-service health polling.GET /api/external-links(main.py:1567) ⇒ dashboard sidebar quicklink with icon, port,ui_path, health needle.GET /api/features(routers/features.py:113) ⇒ a feature card, iffeatures[]present, with status/suggestion/launch.installers/phases/11-services.sh:1374⇒ setup hook execution, if declared.scripts/audit-extensions.py+scripts/validate-manifests.sh⇒ CI validation.scripts/generate-extensions-catalog.py⇒ catalog row (library dir only).ods preset save/apply⇒ captured inextensions.list.bin/ods-host-agent.py/v1/extension/{start,stop,logs}⇒ dashboard lifecycle.
What still needs a hand-edit, and why:
| Want | Extra edit | File:line |
|---|---|---|
An ENABLE_X installer opt-out |
one _sync_extension_compose line |
installers/phases/03-features.sh:179-204 |
| Blocking installer health gate | one _check_health line |
installers/phases/12-health.sh:~616 |
| Port-contract CI parity (bundled only) | config/ports.json + .env.schema.json + .env.example |
tests/contracts/test-port-contracts.py:106 |
A local build: for a user extension |
not possible — the scan rejects it | resolve-compose-stack.sh:251-252 |
| An internal dashboard route (not a quicklink) | plugin registration | dashboard/src/plugins/registry.js:14-20 |
| Feature "how to enable" prose | dict entry | routers/features.py:224-248 |
| Catalog entry for a bundled service | generator only reads extensions/library/services |
scripts/generate-extensions-catalog.py:33 |
So: one to two files for a working extension; three to five if you want it to be a first-class bundled service with an installer toggle and CI port parity. The owner's description is accurate.
C. hal0 today: what exists, what's missing
hal0's runtime model is one podman container per inference slot under
hal0-slot@<name>.service, rendered as a Podman Quadlet .container file
(src/hal0/providers/container.py:490,803), plus a small set of companion units.
Mapping ODS concepts onto what exists:
| ODS concept | hal0 nearest analogue | Verdict |
|---|---|---|
extensions/services/<id>/manifest.yaml |
src/hal0/services/registry.py:65 SERVICES: tuple[ServiceDef, ...] |
Hardcoded Python, 4 entries (openwebui, comfyui, hermes, hindsight). Same fields as an ODS manifest (id, name, unit, port, probe, public_url_env, actions, mdns, loopback_port, hints) — but no filesystem discovery. |
compose.yaml fragment |
packaging/systemd/hal0-openwebui.service (static unit) or RuntimeLaunchPlan → Quadlet (providers/base.py:163, container.py:803) |
Two unrelated mechanisms. Slots get a typed, rendered plan; companions get a hand-written .service with a literal podman run. |
resolve-compose-stack.sh |
(none) | No layering engine. Hardware selection lives inside ContainerProvider + profiles.toml, not in a merged file set. |
ods enable/disable/list |
hal0 app install/list/uninstall (src/hal0/cli/app_commands.py:33-36) |
_SUPPORTED = frozenset({"openwebui"}), _APP_UNITS a 1-entry dict. The verb shape is right; the registry behind it is a literal. |
| Feature/extension registry | src/hal0/install/extensions.py:26 EXTENSIONS: list[Extension] |
Literally named "the first-run Extensions registry (spec §6.4)" — three hardcoded entries (openwebui, comfyui, hermes), with install_extension() an if/elif chain (lines 145-165). |
/api/features status vocabulary |
GET /api/capabilities + GET /api/services |
Capability cards exist (src/hal0/capabilities/catalog.py) but they group models per capability, not "which optional services can this box run". No insufficient_vram / services_needed / available vocabulary for services. |
config/ports.json |
src/hal0/ports/authority.py |
hal0 is better here. No stored table; claims are recomputed from slot TOMLs, live runtime snapshots, reserved ports and actual LISTEN sockets, so deletion releases a port atomically. |
docker-compose.override.yml |
Quadlet drop-ins hal0-slot@<t>.container.d/ |
Exists; explicitly the recommended replacement for the deprecated extra_args (container.py:955). |
extensions/library/ optional catalog |
docs/reference/profile-addons.mdx (.hal0profile.json envelope + index.json) and stacks/portable.py (.hal0stack.json) |
The distribution pattern already exists — signed-ish JSON envelopes with content checksums, dashboard Import, an addons index served from a site. It carries profiles and stacks; it does not carry services. |
| Setup hooks | hal0.agents.hermes_provision (15 phases), install/extensions.py |
Bespoke per component. |
| Dashboard Extensions page | ui/src/dash/services.jsx (431 lines), services-card.jsx |
Registry-driven from GET /api/services and already fail-soft. |
Two findings worth calling out:
- hal0's UI already wants this.
ui/src/dash/dashboard-redesign.jsx:858andui/src/dash/services.jsx:4,46both listn8ninSERVICE_ORDERand ship an n8n icon — but there is noServiceDef, no unit, and no install path anywhere insrc/. The extension-shaped hole is already visible in the product. - The privileged seam is the real constraint. hal0-api runs as
User=hal0; every unit write goes throughsudo -n hal0-systemctl write-quadlet <token>and the root side allow-lists the body: sections[Unit]/[Container]/[Service]/[Install]in strictly increasing order, exactly 3[Unit]keys, ~14[Container]keys, 9[Service]keys, and[Install] WantedBy=hal0.targetonly (installer/wrappers/hal0-systemctl:389-516). Unit-file paths are pinned to thehal0-slot@prefix (src/hal0/system/seam.py:283-312). NoExec*=on the host side,PodmanArgs=restricted to--group-add/--security-opt/--ipc/--ulimit. "Drop in an arbitrary third-party unit fragment" is architecturally refused today, deliberately (#1740, #1759).
Net: hal0 has all the ingredients — a typed launch plan, a validated unit writer, a
services registry with the right fields, a portable-envelope distribution pattern, a
port authority, a registry-driven Services page, and an app install verb — but they
are wired together by hand for four fixed services. There is no manifest, no
discovery walk, and no way for an operator or a third party to add a fifth.
D. Proposed hal0 extension model
Design goal, restated in hal0 terms: an extension author writes one manifest; hal0 renders the unit, claims the port, wires the env, registers the service, probes its health, and shows it in the dashboard — with no code change anywhere.
D.1 File layout
/usr/lib/hal0/extensions/<id>/ # shipped with the release (read-only)
/var/lib/hal0/extensions/<id>/ # operator- or catalog-installed
manifest.toml # REQUIRED (~25-40 lines)
unit.toml # OPTIONAL quadlet overrides (rare)
hooks/post_install.sh # OPTIONAL
README.md # OPTIONAL
/etc/hal0/extensions.toml # enable/disable state + operator overrides
/etc/containers/systemd/hal0-ext@<id>.container # GENERATED, never authored (rootful system Quadlet — see Decision 3)
Discovery order mirrors hal0.config.paths conventions (/var/lib wins over
/usr/lib, HAL0_EXTENSIONS_DIR overrides both for tests) — exactly the resolution
shape already used by hal0.bundles.tiers._candidate_roots
(src/hal0/bundles/tiers.py:44-56).
D.2 Manifest schema (manifest.toml, schema_version = "hal0.ext.v1")
schema_version = "hal0.ext.v1"
[extension]
id = "n8n" # [a-z0-9][a-z0-9-]*, == directory name
name = "n8n (Workflows)"
category = "optional" # core | recommended | optional
hal0_min = "0.4.0" # ≙ compatibility.ods_min
[runtime]
image = "docker.io/n8nio/n8n@sha256:…" # digest-pinned, like RUNNER_IMAGES
port = 5678 # in-container; publish_port defaults to it
network = "bridge" # bridge | host (slots use host)
volumes = [ { source = "$DATA/n8n", target = "/home/node/.n8n" } ]
user = "$HAL0_UID:$HAL0_GID"
### devices/group_add gated — see D.6; [runtime.rocm]/[runtime.vulkan]/[runtime.cpu]
### sub-tables replace ODS's compose.<backend>.yaml overlays.
[health]
kind = "http" # http | tcp | container | none
path = "/healthz"
timeout_s = 15
[wiring] # <- "prewired to existing slots"
openai_base_url_env = "OPENAI_BASE_URL" # ⇒ http://host.docker.internal:8080/v1
openai_api_key_env = "OPENAI_API_KEY"
model_env = "OPENAI_MODEL" # ⇒ "hal0/brain" — never a concrete id
slot_env = { WHISPER_URL = "slot:stt", TTS_URL = "slot:tts", COMFY_URL = "slot:img" }
[[env]]
key = "N8N_PASS"; required = true; secret = true
[[capability]] # ≙ ODS features[]
id = "workflows"; name = "Workflow Automation"; icon = "Workflow"
requires_services = ["n8n"]; requires_vram_gb = 0; priority = 40
launch = { kind = "service", service = "n8n" }
[hooks]
post_install = "hooks/post_install.sh"
Near-verbatim ports from ODS (semantics unchanged, TOML instead of YAML, Pydantic
instead of JSON Schema): id/name/category/aliases, port/external_port,
health+health_timeout, depends_on, env_vars[], hooks{},
compatibility.ods_min → hal0_min, and the whole features[] → [[capability]]
block including the enabled_* vs requires_* split and launch{}.
Deliberately dropped: compose_file (no compose), gpu_backends (hal0 derives device
class from hardware.json; use [runtime] requires_device = "gpu-rocm" instead),
container_uid (hal0 owns data-dir ownership through hal0.install.perms),
host_network (a policy decision, not an author's).
D.3 Discovery points (the "everything auto-attaches" contract)
| Consumer | Implementation | ODS counterpart |
|---|---|---|
hal0.extensions.registry.load() |
walk the two roots, parse+validate manifests, memoise on mtime — a direct Python port of sr_load (lib/service-registry.sh:50) minus the bash-emitting subprocess |
service-registry.sh |
hal0.services.registry.SERVICES |
becomes builtin_services() + extension_services(); ServiceDef gains source: "builtin"\|"extension" and manifest_path |
config.py:load_extension_manifests |
GET /api/services + Services page |
zero changes — it already renders whatever the registry returns (src/hal0/api/routes/services.py:140) |
/api/services |
| Unit rendering | ExtensionProvider.launch_plan(manifest) -> RuntimeLaunchPlan, then the existing _render_quadlet_from_plan (container.py:803) — parameterised for the artifact family (correction 2026-10-04): as written it names every container through slot_container_name() (slots/naming.py:70; ContainerName=hal0-slot-<token>, container.py:826,880), so an extension id equal to a slot token would collide with that slot's podman container; extensions need their own namer (hal0-ext-<id>) threaded through the renderer |
compose fragment |
| Unit write | SystemCtlSeam.write_quadlet extended to accept the hal0-ext@ prefix (system/seam.py:283), wrapper validate_quadlet_body reused unchanged |
resolver -f flag |
| Port claim | hal0.ports.authority.collect_claims gains an "extension-config" source |
config/ports.json (better) |
| Health | the probe field already dispatches http/systemd/comfyui in services_health.py; add tcp/container |
health-check.sh:159 |
| Capability cards | GET /api/capabilities merges [[capability]] rows and computes enabled\|available\|insufficient_vram\|services_needed |
routers/features.py:70-77 |
| mDNS | services/mdns.py already writes hal0-addon-<id>.service per advertised id — extensions inherit it free |
dashboard quicklinks |
| Journald logs | GET /api/logs?unit=hal0-ext@<id>.service — already generic |
ods logs <id> |
| Catalog | hal0.extensions.portable — a .hal0ext.json envelope + index.json, modelled byte-for-byte on stacks/portable.py:41 and docs/reference/profile-addons.mdx |
extensions/library/ + extensions-catalog.json |
D.4 CLI verbs
Promote hal0 app from a 1-entry frozenset to the extension registry, keeping the
existing verb names (src/hal0/cli/app_commands.py):
hal0 app list # id, category, state (enabled|disabled|not-installed|incompatible)
hal0 app install <id> # render unit → seam write → daemon-reload → enable --now → post_install hook (bundled only; see below)
hal0 app enable|disable <id> # flips /etc/hal0/extensions.toml, converges the unit
hal0 app uninstall <id> # disable + remove unit; --purge also drops /var/lib/hal0/ext/<id>
hal0 app import <file|url> # .hal0ext.json envelope, checksum-verified
hal0 app doctor [<id>] # manifest validation ≙ scripts/audit-extensions.py
Correction (2026-10-04) — hooks and the trust boundary. post_install as drawn runs on the
host, after the unit is enabled, as whoever invoked the verb (hal0 from the API, root from a
shell). For an imported manifest that is arbitrary third-party code outside every rootless,
capability-drop, read-only-root and mount restriction §E describes. Rule: host hooks are honoured
only for bundled extensions that ship in the hal0 repository (already inside the trust boundary);
an imported manifest's hooks/ are either refused at hal0 app import or run as a one-shot
inside the extension's own container under the same confinement (ExecStartPost= →
podman exec), never on the host.
hal0 doctor gains an extensions arm; the ODS audit-extensions.py check list
(schema version, id↔directory match, category/type enum, positive port, health starts
with /, alias collisions, dangling depends_on, duplicate capability ids) ports
almost line-for-line into hal0/extensions/audit.py.
D.5 Pre-wiring, translated
ODS pre-wiring is docker-network DNS + compose interpolation. hal0 has neither:
slot containers run Network=host with a loopback fence
(container.py:_loopback_fence_command), so there is no service-name DNS. The
translation is rendered env, computed at unit-render time from live truth:
OPENAI_BASE_URL→http://host.docker.internal:8080/v1for bridge-network extensions (podman honours--add-host=host.docker.internal:host-gateway, already used atpackaging/systemd/hal0-openwebui.serviceExecStart), orhttp://127.0.0.1:8080/v1for host-network ones. This is exactly whathal0.openwebui.env_writeralready does for OpenWebUI (src/hal0/openwebui/env_writer.py:11-30) — generalise that writer.OPENAI_MODEL→hal0/brain(the virtual model that already resolves through the dispatcher), never a concrete model id. This is the direct analogue of ODS'sods/currentswap-safety rule and should be documented the same way (docs/SWAP-SAFE-EXTENSIONS.md:17-33ports nearly verbatim).slot:<name>refs in[wiring].slot_envresolve at render time throughhal0.ports.authority. Correction (2026-10-04): the first draft resolved them tohttp://127.0.0.1:<slot port>unconditionally and claimed the URL "follows the slot if the operator moves its port". Both halves were wrong. (i) Inside a bridge-network extension127.0.0.1is the extension's own loopback, andhost.docker.internaldoes not help either: slots are deliberately loopback-only on the host —_loopback_fence_commandflips every0.0.0.0bind to127.0.0.1underNetwork=host(src/hal0/providers/container.py:619-647;providers/base.py:206"host port the container listens on (127.0.0.1 only)") — so the host-gateway address cannot reach a slot port at all. A bridge extension must therefore go through hal0-api, which does listen on the LAN interface:slot:stt→http://host.docker.internal:8080plus the gateway's own route for that modality (POST /v1/audio/transcriptions→ the stt slot,routes/v1.py:1398;/v1/images/generations→ the img slot). This is exactly howopenwebui/env_writer.py:112-128already wiresAUDIO_STT_OPENAI_API_BASE_URLandAUDIO_TTS_OPENAI_API_BASE_URL— at hal0-api:8080, never at a slot. Only anetwork = "host"extension may be handedhttp://127.0.0.1:<slot port>directly. (ii) A render-time value does not follow anything: the generated env file keeps the old port, and nothing in the convergence list re-renders dependants when a slot TOML changes. Gateway URLs sidestep this (hal0-api's port is stable and the dispatcher follows the slot); for the host-network direct form, recorddepends_on_slotsin the extension registry and have the slot-config write path (SlotConfigStore) trigger extension convergence — a hook that does not exist today and must ship with the feature. ODS's static DNS names and.envports have the same staleness problem; the honest claim is "gateway-routed by default, so a slot move is invisible", not "the URL follows the port". (iii) Correction (2026-10-04):host.docker.internalresolves to the host's address on the podman bridge (--add-host=host.docker.internal:host-gateway,env_writer.py:102-105), not to 127.0.0.1. Report 05's gateway work wants the direct ports off the LAN; if that is done by binding hal0-api to loopback alone, every bridge extension — and hal0's own Open WebUI — loses inference, STT, TTS and image routes. The two must be designed together: hal0-api keeps a listener the bridge can reach (a second bind on the podman gateway address, or an authenticated container-facing listener on a shared internal network) while the LAN-facing :8080/:3001 go away, or the LAN cut is an nftables rule that exempts the bridge subnet.HAL0_BIND_HOSTis a single value today (config/network.py:37-45), so the two-listener form is new work, not configuration.- Env lands in
/etc/hal0/extensions/<id>.env, written atomically byhal0.config.env.write_env_atomic(the same primitive slot env files use) and referenced from the generated unit — keeping secrets out of the unit file and out of thepodman runargv, which is the lesson of #1466 already recorded inenv_writer.py:44-52.
D.6 Near-verbatim ports vs. things needing translation
Near-verbatim (port the logic, change the language):
- lib/service-registry.sh:50-235 → hal0/extensions/registry.py. The loader is
already a Python program embedded in bash; lifting it out and dropping the _esc()
bash-quoting layer is a simplification, not a rewrite.
- extensions/schema/service-manifest.v1.json → Pydantic models in
hal0/extensions/schema.py. Keep field names that carry meaning (category,
health, depends_on, env_vars, hooks, compatibility) so ODS-authored
manifests convert mechanically.
- routers/features.py:19-110 calculate_feature_status → capability-status
computation, including the four-value vocabulary and the requires/enabled split.
- scripts/audit-extensions.py:417-747 → hal0/extensions/audit.py.
- docs/EXTENSIONS.md §"30-Minute Path" + service-template.yaml → a hal0
write-an-extension.mdx and a heavily-commented manifest.toml template. The
commented template is a large part of why ODS's model is usable; do not skip it.
- docs/SWAP-SAFE-EXTENSIONS.md tiers 1-3 — hal0's slots load/unload constantly, so
this doc is more necessary here, not less.
Needs real translation:
- Compose merge → one rendered unit. ODS gets layering free from docker compose
-f a -f b. hal0 must merge itself: manifest.toml → RuntimeLaunchPlan → optional
operator override via a hal0-ext@<id>.container.d/ drop-in. That mechanism already
exists as the documented replacement for extra_args (container.py:947-957), so
the story is base manifest + typed drop-in, not N merged files.
- GPU overlays. ODS picks compose.<backend>.yaml by filename; hal0 already
resolves device class from hardware.json and image from RUNNER_IMAGES. Use
[runtime.rocm] / [runtime.vulkan] / [runtime.cpu] sub-tables — one file, not four.
- The seam allow-list. installer/wrappers/hal0-systemctl:389-465 must gain the
hal0-ext@ prefix and two new [Container] keys (EnvironmentFile=, AddHost=;
Environment=/PublishPort=/Volume= are already allowed). Every widening is a
root-boundary change needing the scrutiny #1740 got. Highest-risk piece here.
- ENABLE_* .env gating → /etc/hal0/extensions.toml. ODS's rename-the-file trick
has no analogue and no merit here; use a state file with the atomic write/rollback
shape SlotConfigStore already implements (src/hal0/slot_config/__init__.py).
- Devices / GPU passthrough. ODS refuses devices: for user extensions outright
(resolve-compose-stack.sh:275-276). hal0 should default to the same refusal, with an
operator-only escape for locally-authored manifests.
E. Things NOT to copy
- Enable/disable by renaming
compose.yaml↔compose.yaml.disabled. It makes the content tree the state store, so agit pull, a backup restore, or an rsync silently changes which services run. ODS pays for this with the "stale disabled marker" reconciliation atods-cli:2833-2842. hal0 hasSlotConfigStorewith atomic commit/revert; use a state file. - The hand-maintained
_sync_extension_composetable. 21 lines atinstallers/phases/03-features.sh:179-204that must be edited for every new optional service — the exact seam the rest of the design avoids, and the reason the file carries a 10-line comment explaining why omission is a bug. Derive the toggle fromcategory+ a state file instead. - Two parallel status vocabularies.
/api/featuresreturnsenabled|available|insufficient_vram|services_needed;/api/extensions/catalogreturns ten different values. The dashboard has to teach users both (Extensions.jsx:73-80is a legend). Pick one lifecycle vocabulary and one eligibility vocabulary and derive card copy from them. - A generated static catalog JSON checked into the repo.
config/extensions-catalog.jsonis produced byscripts/generate-extensions-catalog.pyfrom manifests and then committed, so it can drift (hence the reminder indocs/EXTENSION-PR-BRANCHING.md:12). Serve it from the live manifest walk. config/ports.jsonas a fourth place ports are written. hal0'sports/authority.pydocstring already argues the case against stored allocation tables; do not regress.- Bash + embedded-Python + associative arrays.
service-registry.shneeds a Bash 4 guard and re-execs under Homebrew bash on macOS (health-check.sh:11-19); the compose resolver hard-fails without PyYAML. hal0 is a Python service — the whole registry is ~150 lines of ordinary Python. - Fine-grained
llm.probein v1. ODS's post-swap probe descriptor (schema:…llm.probe) is genuinely useful but only pays off once you have a swap harness driving it. Ship[wiring]first; add probes when there is a consumer. - Allowing arbitrary unit/quadlet content. ODS's content scan is a deny-list over
compose keys (
resolve-compose-stack.sh:249-334) — a shape that has to be kept in sync with Docker's evolving schema, and which is duplicated in three places (resolver, dashboard_scan_compose_content, host agent). hal0's allow-list over a rendered unit is strictly stronger. Keep rendering from a typed manifest; never accept a unit body an author wrote.
F. Risks and decisions for the owner
Decision 1 — Are extensions services, or also slots? ODS conflates them: a
manifest can describe a core LLM server or a git host. hal0's slot layer is a
specialised, heavily-invariant subsystem (GpuArbiter, NPU trio exclusivity,
model registry binding). Recommendation: extensions are companion services only
(hal0-ext@<id>.service), a strictly separate unit family from hal0-slot@. An
extension that wants inference asks hal0 over HTTP like every other client.
Decision 2 — How wide does the seam allow-list open? Every [Container] key added
to installer/wrappers/hal0-systemctl:407-438 is a root-boundary change. The minimum
new set is EnvironmentFile= (pinned to /etc/hal0/extensions/<id>.env) and
AddHost=host.docker.internal:host-gateway (pinned to that literal). Recommendation:
pin both to literals rather than charset regexes, and require that Volume= sources for
extensions live under /var/lib/hal0/ext/<id>/ — the path constraint ODS cannot make
because compose mounts are relative to a project dir the operator owns.
Decision 3 — Rootful vs. rootless for extension containers. Slots run rootful
podman today, and the wrapper's own HONEST BOUNDARY note (hal0-systemctl:52-70)
concedes that anyone who can author a container spec can mount host paths. Third-party
extensions raise the stakes materially. Correction (2026-10-04): the first draft
recommended "rootless under the hal0 user from day one" while the implementation above
(D.1, D.6) writes /etc/containers/systemd/hal0-ext@<id>.container through the
system-scope SystemCtlSeam — that is the rootful Quadlet path (hal0-systemctl:111
pins QUADLET_DIR="/etc/containers/systemd"; the wrapper has no --user mode), and a
User= inside the container changes the process uid, not the Podman boundary. The two
options, stated honestly:
- Explicitly rootful v1 (recommended for first-party extensions). Keep the
system-scope seam; compensate with what the manifest can enforce: digest-pinned images,
Volume=sources confined to/var/lib/hal0/ext/<id>/, nodevices:,NoNewPrivileges=yes,DropCapability=all,ReadOnly=yeswhere the image allows, andUser=as defence in depth. Say in the docs that an extension runs in the same boundary as a slot. Do not enablehal0 app importof third-party manifests under this mode. - Real rootless path (prerequisite for third-party import). Quadlets under
~hal0/.config/containers/systemd/(or/etc/containers/systemd/users/<uid>/),loginctl enable-linger hal0, a user-scope variant of the seam (systemctl --user, a second allow-list), the subuid/subgid ranges the installer already allocates (installer/install.sh:760), GPU access via therendergroup membershipgpu_permsalready manages, and a lab check thathost-gatewayresolves under the rootless network backend (pasta vs. slirp4netns) before relying onAddHost=host.docker.internal:host-gateway.
Recommendation: ship v1 explicitly rootful with the compensating controls and the import verb disabled; make the rootless seam the gate for Decision 4's third-party import rather than claiming rootless while rendering rootful.
Decision 4 — Who may publish? ODS ships 33 library extensions it does not test on
every hardware class and lets the dashboard install them at a click. hal0's culture
(digest pins, reproducible runner recipes, packaging/runner/*/manifest.toml
provenance notes) is much stricter. Recommendation: ship a small first-party
extension set (n8n is already half-designed in the UI), require digest-pinned images in
the manifest schema, and gate third-party import behind an explicit
hal0 app import --i-understand with the same envelope-checksum machinery
stacks/portable.py already has.
Decision 5 — Migration of the existing four. OpenWebUI, ComfyUI, Hermes and
Hindsight should not all become extensions. ComfyUI is a slot
(hal0-slot@img.service) and its start verb is GPU-arbiter-special-cased
(services/registry.py:88-96); Hermes has a 15-phase provisioner. Recommendation:
convert OpenWebUI only as the proof — it is a plain podman container with a
prewired env file, i.e. exactly the extension shape — and leave ServiceDef as a union
of builtin and extension-derived rows. If OpenWebUI cannot be expressed in
manifest.toml, the schema is wrong.
Risk — the "everything auto-attaches" promise is only as good as the weakest
walker. ODS delivers it because four independent consumers each re-walk the
directory. hal0 must fund all of them in one go: registry, unit render, port claim,
health probe, capability card, mDNS, logs, CLI. Shipping the manifest without the
capability card or without hal0 app doctor produces exactly the half-wired feeling
the owner wants to avoid.
Risk — id collisions across sources. ODS mirrors a core-service-id deny list in
three places (resolve-compose-stack.sh:178-188, routers/extensions.py,
helpers.py:CORE_SERVICE_IDS_FALLBACK) and silently drops duplicates
(service-registry.sh:146-148). hal0 should make that one function with one test, and
reserve the hal0- id prefix outright.
Open question to settle before code: may an extension consume a specific slot
("n8n needs stt"), or only the API? [wiring].slot_env assumes the former, which is
what brings ODS's depends_on + services_needed status into play and is the main
reason to port calculate_feature_status. If the answer is "API only", the whole
capability/eligibility half of the design collapses and the first release gets
meaningfully cheaper.
03-installer-reliability.md Installer and reliability
03 — Installation, Upgrade, and Reliability Engineering: ODS vs hal0
Source-level comparative study. All paths are absolute. ODS product runtime is
/home/user/ods/ods/; hal0 is /home/user/hal0/.
The owner's premise — "ODS has me beat on installation and reliability" — is half right. ODS beats hal0 decisively on installer structure, install-time observability, and failure forensics. hal0 beats ODS decisively on update integrity, diagnosis modelling, and preflight depth. The gaps are asymmetric and mostly complementary, which is good news: almost everything worth porting is additive.
A. How ODS does it
A.1 The install pipeline is a sourced-phase state machine
/home/user/ods/ods/install-core.sh is a 363-line orchestrator and nothing
else. It does five things in order:
- Sets
INSTALL_PHASEand installs anERRtrap (install-core.sh:23-38) that reports which phase died, where the log is, and what the partial state is:bash export INSTALL_PHASE="init" cleanup_on_error() { local exit_code=$? echo -e "\033[0;31m[ERROR] Installation failed during phase: ${INSTALL_PHASE}\033[0m" echo -e "\033[0;33m Log file: ${LOG_FILE:-/tmp/ods-install.log}\033[0m" ... echo "To retry, run the installer again. It will resume safely." } trap cleanup_on_error ERR - Installs a double-tap SIGINT handler (
:45-64) — one Ctrl+C warns, a second within 3s aborts and callscancel_active_download;SIGTSTPis trapped to nothing so Ctrl+Z can't background a live install. - Sources 15 pure libraries in dependency order (
:74-93). - Parses ~40 flags (
:220-273):--dry-run,--non-interactive,--summary-json PATH,--no-bootstrap,--offline,--lan,--tier N, … - Sources the 14 phase files, assigning
INSTALL_PHASEbefore each (:338-363). Phase 13 runs underset +ebecause it is cosmetic and "must never fail the install".
The lib/phase split is a hard architectural rule
(docs/INSTALLER-ARCHITECTURE.md): libraries define functions only and are safe
to source; phases execute on source. Everything shares one bash namespace.
docs/INSTALLER_PHASE_CONTRACTS.md then gives each phase a contract row — Owns
/ Inputs / Outputs / Idempotency expectation / Common failure modes — plus a
"Required Validation By Phase" table and a written Reinstall Contract ("keep
user secrets; keep downloaded models when valid; keep runtime data under data/;
exit non-zero only when the installed product is not recoverable").
A.2 The standardized module header
Every file under installers/lib/ and installers/phases/ carries the same
header. This is the highest-leverage, lowest-cost thing in the whole repo:
#!/bin/bash
### ============================================================================
### ODS Installer — <Module Name>
### ============================================================================
### Part of: installers/lib/ (or installers/phases/)
### Purpose: <one-line description>
### ### Expects: <globals/functions this file reads>
### Provides: <globals/functions this file defines>
### ### Modder notes:
### <when and why you'd edit this file>
### ============================================================================
The Expects/Provides chain is how a reader traces dataflow through a global
namespace without reading every line
(docs/INSTALLER-ARCHITECTURE.md, "File Header Convention"). Real examples:
installers/lib/detection.sh:1-22, installers/phases/02-detection.sh:1-22,
installers/lib/ui.sh:1-24.
The same doc carries a Mod Recipes table ("Add a hardware tier → edit
lib/tier-map.sh + lib/detection.sh") and a Generated Config Writers table
naming, per config surface, the Linux/macOS/Windows/upgrade writers that must
stay in sync — because "only one platform writer was updated" is their most
common regression class.
A.3 Bootstrap: get-ods.sh
/home/user/ods/ods/get-ods.sh (507 lines) is the curl | bash entry point:
- CWD anchoring (
:14-21): re-runs from a deleted directory (post-uninstall) otherwise fail with a misleading "check your internet connection". - Error classification (
:52-65):format_git_clone_errormaps git stderr to a specific human cause (getcwd, DNS, permissions). ODS_REFpinning (:90-108, 414-438): a 40-hex SHA gets--depth 1+git fetch origin <sha>+git checkout --detach; a branch/tag gets--branch. Sparse checkout of onlyods/, with a full-clone fallback.- Coexistence refusal (
:110-215): before cloning it scans sibling directories for an install signature (.env+ a compose file namingopen-webui,dashboard-api, andllama-server|litellm) and queries Docker for compose projects with the same service tuple, refusing with an explicitODS_ALLOW_LEGACY_PARALLEL=1escape hatch. - Runtime/dev split (
:443-468): rsync excludestests/,docs/,*.md,.github/— the installed tree is smaller than the repo tree. docs/INSTALLER_TRUST.mddocuments the provenance story honestly, including what is not yet true ("ODS does not yet publish a complete signed-release or checksum/SBOM chain"), with a numbered roadmap.
A.4 The pure libraries
Highlights, all under /home/user/ods/ods/installers/lib/:
detection.sh(842 lines).ods_in_container()(:150-163) checks/.dockerenv,/run/.containerenv,systemd-detect-virt --container,/proc/1/cgroupregex, and/proc/1/environ;ods_container_label()(:165-185) names it (lxc,docker, …). This feedsshow_amd_gpu_device_guidance()(:219-233), which prints LXD-specific remedies (lxc config device add <container> kfd unix-char path=/dev/kfd— LXD syntax; the Proxmox equivalent is adev0:/dev1:line in/etc/pve/lxc/<CTID>.conf, cf. hal0'sdocs/getting-started/proxmox.mdx:91-101) when inside a container and modprobe remedies when not.apply_cpu_gpu_fallback()(:235-257) then degrades the whole install to CPU coherently — it rewritesGPU_BACKEND,CAP_*, and clears compose overlays in one place, rather than letting a half-set backend leak downstream.select_cpu_fallback_tier()(:259-274) picks a tier from RAM alone.- CPU budget (
detection.sh:92-147):get_docker_available_cpus()prefersdocker info --format '{{.NCPU}}'overnproc, thencalculate_llama_cpu_budget()clamps per-backend limit/reservation to what's actually available.llama-memory-budget.shdoes the same for RAM, takingmin(host, docker)so Docker Desktop VMs don't get over-committed. progress.sh(25 lines, quoted in full in D.2) — a GUI progress protocol that is a complete no-op unlessODS_INSTALLER_GUI=1.docker-images.sh/compose-images.sh— validate an image tag exists before compose runs (validate_docker_image_or_fallback), and discover the external image set from the resolved compose stack rather than a hardcoded list.readiness-summary.sh(128 lines) — takesname|health_url|container|open_urllines on stdin, probes each, printsReady now: N/Mwith per-service state (ready/starting/not detected/needs attention) plus a "Next:" block.model-lifecycle-lock.sh(83 lines) —flockmutual exclusion between the installer and the detached background upgrader, keyed by acksumof the resolved install dir so an SSH session and a systemd unit with differentXDG_RUNTIME_DIRstill contend on the same lock.sudo.sh—ods_prepare_sudo()setsODS_SUDO_AVAILABLE;ods_sudo()then skips root-only extras rather than hanging on a password prompt under--non-interactive.podman-registries.sh(234) repairs Podman short-name search config idempotently (Bash-3.2-safe empty-array guard);packaging.sh(312) is the apt/dnf/yum/pacman/zypper abstraction;lib/rootless-ownership.sh(repo root) providesods_fix_rootless_ownership()for rootless-Docker bind-mount UID mapping, called fromphases/06-directories.sh:1161and fromods-cli.
A.5 Installer UX: the narrator
installers/lib/ui.sh:97-103 is the entire narrator API:
ai() { echo -e " ${GRN}▸${NC} $1" | tee -a "$LOG_FILE"; }
ai_ok() { echo -e " ${BGRN}✓${NC} $1" | tee -a "$LOG_FILE"; }
ai_warn() { echo -e " ${AMB}⚠${NC} $1" | tee -a "$LOG_FILE"; }
ai_bad() { echo -e " ${RED}✗${NC} $1" | tee -a "$LOG_FILE"; }
signal() { echo -e " ${GRN}░▒▓█▓▒░${NC} $1" | tee -a "$LOG_FILE"; }
Every one of them tees to $LOG_FILE. That single detail is why ODS can hand
a user a log after a failure and why ods-doctor can mine install-report-*.txt
later.
show_phase (ui.sh:115-125) prints PHASE n/6 — NAME with a wall-clock
timestamp and an EST. TIME: line supplied by the caller; call sites give
honest ranges (phases/08-images.sh:21: "~5-10 min + ~30 min ComfyUI build").
spin_task (ui.sh:236-266) drives a braille spinner with an [mm:ss] counter,
optionally rendering live progress from a .part file; format_download_progress
(ui.sh:187-208) is deliberately pure so it is BATS-testable.
pull_with_progress (ui.sh:305-370) is the retry-protected image pull:
timeout 3600 docker pull, up to ODS_DOCKER_PULL_MAX_ATTEMPTS (default 4)
attempts with backoff 5/15/30s then doubling (_docker_pull_retry_delay,
ui.sh:269-302), a post-pull docker inspect validation step, and a
non-retryable classifier that bails immediately on
unauthorized|denied|not found|404|no space left on device|cannot connect to the
docker daemon. check_service (ui.sh:375-460) is the health-check twin:
exponential backoff capped at 8s, distinguishing curl exit 124 (timeout), 7
(refused), 56/52 ("starting up"), while simultaneously inspecting the container —
if it is exited|dead|missing it stops retrying instead of burning 30 attempts.
Other surfaces: show_hardware_summary/show_tier_recommendation
(ui.sh:462-503); show_install_menu (ui.sh:507-597, Full/Core/Custom, which
auto-disables ComfyUI on tier 0/1 with an explanation); show_success_card
(:600-632); a .desktop file + GNOME favourites pin
(phases/13-summary.sh:229-254); the readiness summary (:365-396); and the
--summary-json writer (:435-502), a versioned payload (version,
installer_version, tier, runtime.{gpu_backend,backend_service,llm_model,compose_flags,dry_run},
hardware_class, preflight_report) written with mkstemp + fsync +
os.replace. docs/SETUP-CARD.md covers a printable 4×6 QR card generator;
docs/POST-INSTALL-CHECKLIST.md is a six-step "did it work" ritual.
A.6 Bootstrap mode: tiny model first, hot-swap later
ODS's signature reliability trick; hal0 has no equivalent.
installers/lib/bootstrap-model.sh (57 lines) pins a ~1.22 GiB Qwen3.5-2B (file,
URL, SHA256, model id, 64K context floor) and defines bootstrap_needed():
true only when tier > 0, the full GGUF is not already on disk, --no-bootstrap
was not passed, and the install is not offline / cloud / external-Lemonade.
phases/11-services.sh:673-690 stashes the full-model config into FULL_* vars
and swaps GGUF_FILE/GGUF_URL/LLM_MODEL/MAX_CONTEXT to the bootstrap values, so
the entire rest of the phase (download, models.ini, .env patch, compose up)
runs unmodified against the tiny model — the user is chatting in ~2 minutes.
:1329-1372 then persists retry metadata to data/bootstrap-upgrade.args (mode
600) and launches the upgrader detached, inside a subshell that closes
inherited non-stdio FDs first so the parent's flock FDs aren't held for the
duration of a multi-GB download:
(
_phase11_close_inherited_fds_for_daemon
exec nohup bash "$SCRIPT_DIR/scripts/bootstrap-upgrade.sh" \
"$INSTALL_DIR" "$FULL_GGUF_FILE" "$FULL_GGUF_URL" \
"$FULL_GGUF_SHA256" "$FULL_LLM_MODEL" "$FULL_MAX_CONTEXT" \
"$BOOTSTRAP_GGUF_FILE" \
> "$INSTALL_DIR/logs/model-upgrade.log" 2>&1
) &
scripts/bootstrap-upgrade.sh (3412 lines) runs set -uo pipefail without
-e on purpose ("we handle errors explicitly to avoid killing the background
process on transient failures"). Its shape:
- Status file:
write_status()writesdata/bootstrap-status.jsonatomically (status,percent,bytesDownloaded/Total,speedBytesPerSec,eta); amonitor_downloadloop refreshes it every 2s from the.partsize — this is what drives the dashboard progress bar. - Download (
:2189-2325): script-owned resume, not curl's internal retry (a curl retry after a long reset restarts at byte zero and truncates a good multi-GB.part). Bounded wall-clock budget with rounds; over/undersized.partdetection; SHA256 verify; failure preserves the.partfor resume. - Transaction snapshot (
:305-380):snapshot_active_model_config()copies.env,config/llama-server/models.ini,config/litellm/lemonade.yamlinto amktemp -dunderdata/, recording absence as a.missingsentinel so restore can delete a file that didn't exist.restore_active_model_config()is the exact inverse. - Swap:
.envpromotion,models.inirewrite, thendocker compose … up -d --force-recreate --no-deps llama-serverwithenv -u GGUF_FILE -u LLM_MODEL -u MAX_CONTEXT -u CTX_SIZEso the freshly written.envwins compose interpolation over inherited shell vars. - Post-swap assertion (
:2712-2745): inspects the recreated container's.Config.Cmdand fails loudly if--modelstill points at the old GGUF — catching the "compose started rather than recreated" bug that would otherwise surface hours later as a 502. - Rollback (
restore_docker_llama_server_after_swap_failure,:386-452): restore snapshot → recreate with retry → poll health 60×5s → on AMD also restart LiteLLM and prove a completion through the restored route before declaring rollback successful. Called from four failure sites (:2826, :2845, :2865, :3046), each writing a human-readablewrite_status "failed" … "<what happened>. <rollback state>. Re-run to retry." - Cleanup (
:3200-3202): the bootstrap GGUF is deleted only whenHOT_SWAP_VERIFIED=true.
Net effect: a failed upgrade leaves the user on a working small model with a resumable partial download and an explanatory status string. That is the single best reliability idea in the repo.
A.7 Reliability tooling
| Tool | File | What it is |
|---|---|---|
| Preflight engine | scripts/preflight-engine.sh (362) |
Takes --tier/--ram-gb/--disk-gb/--gpu-backend/--gpu-vram-mb/--platform-id/--compose-overlays, emits blockers/warnings + JSON report (/tmp/ods-preflight-report.json) and an --env mode for shell integration. Pure inputs → pure verdict, so it is fixture-testable. |
| Preflight fixtures | tests/contracts/test-preflight-fixtures.sh (132) |
Six named scenarios (linux-nvidia-good, windows-mvp-good, macos-mvp-good, disk-blocker, …) asserting .summary.blockers. Runs in make test. |
| Runtime preflight | ods-preflight.sh (332) |
Post-install service probe with bounded curl (--connect-timeout 3 --max-time 10); --install-env delegates to scripts/linux-install-preflight.sh (456) for a JSON environment report. |
| Doctor | scripts/ods-doctor.sh (1730), docs/ODS-DOCTOR.md |
--json report with capability_profile, preflight, install_artifacts, diagnoses[] (stable IDs like ODS-COMPOSE-CWD-MISMATCH, ODS-DOCKER-IMAGE-UNRESOLVED, plus a whole ODS-RUNTIME-* inference-contract family) each carrying severity/confidence/evidence/impact/next_steps, runtime, summary, autofix_hints. Exit 0/1. |
| Support bundle | scripts/ods-support-bundle.sh (792), docs/SUPPORT-BUNDLE.md |
Redacted tarball: doctor output, extension audit, compose resolution + validation, docker info, container log tails, config/env.redacted. Best-effort — a failing probe is recorded in the bundle, not fatal. |
| Compose failure report | installers/lib/compose-failure-report.sh (187) |
On compose failure writes install-report-<ts>.txt: privacy note, phase, compose command, cached flags, model/runtime env, likely failed image(s) grepped out of the installer log, per-port occupancy with the owning process, docker version/info, redacted compose config tail, compose ps -a, and 160 lines of installer log. The redactor (_ods_report_redact_stream) reads .env and masks both key-name matches and the literal secret values. |
| Update | ods-update.sh |
check/status/backup/update/rollback/changelog/health. snapshot_pre_update() (:235-320) copies .env*, all docker-compose*.yml, .compose-flags, config/{litellm,n8n,openclaw,searxng}/, .version into data/backups/pre-update-<ts>/ with a snapshot.json integrity marker and MAX_BACKUPS pruning. cmd_update = snapshot → git pull → run migrations/migrate-v*.sh in order → restart → wait_for_healthy (:374-407). Any non-zero step calls _update_rollback() (:409-450), which restores + restarts and, on restore failure, prints literal manual recovery commands. |
| Migrations | migrations/ + scripts/migrate-config.sh |
Versioned migrate-vX.Y.Z.sh with a .migration-state ledger; check/diff/migrate/backup. |
| Backup / uninstall / stack test | ods-backup.sh, ods-restore.sh, ods-uninstall.sh, test-stack.sh |
Retention count + free-space precheck; --keep-models/--keep-data/--force; test-stack.sh runs --quick/--stress/--voice with no set -e and exits 0/1/2. |
| Policy docs | docs/KNOWN-GOOD-VERSIONS.md, docs/ADR-IMAGE-TAG-PINNING.md, docs/RELEASE_VALIDATION.md, docs/VALIDATION-MATRIX.md, docs/HIGH_RISK_CHANGE_MAP.md |
Tested-version baselines per platform; a written, reasoned decision to not pin three :latest tags; the "User Green" gate (Zero-prereq bootstrap / Install / Product / Capability / Model Switchboard / Lifecycle Green); the four-layer validation matrix; and a risk table mapping changed area → required validation. |
| Gate + tests | Makefile, tests/ |
make gate = lint + test + bats + smoke + simulate; test alone runs ~60 named contract scripts. 33 BATS files, 32 contracts/, 6 smoke/, plus scripts/simulate-installers.sh (292) which runs install-core.sh --dry-run --non-interactive --skip-docker --force --summary-json, the macOS installer in MVP mode, the preflight engine as a Windows stand-in, and doctor — merging all of it into artifacts/installer-sim/SUMMARY.{json,md}. |
A.8 Tauri desktop installer
/home/user/ods/installer/ is a Vite + React + Tauri v2 app (7 pages: Welcome →
Prerequisites → SystemCheck → GpuDetected → Features → Installing → Complete;
src-tauri/src/ is 1246 lines of Rust). installer.rs::run_install (:33-172)
is thin by design: ensure_checkout() does git clone --depth 1 --branch, then
it maps GUI feature toggles to installer flags and spawns install.sh (or
install.ps1) with ODS_INSTALLER_GUI=1, piping stdout through
parse_progress_line(). stderr is collected on a separate thread and the last
10 lines are surfaced in the error message. PROGRESS_INTEGRATION.md documents
the one-line-per-phase contract. It is a wrapper, not a second installer — the
shell path stays the single source of truth.
A.9 Platform breadth
The phase model is replicated, not shared. Windows:
installers/windows/install-windows.ps1 dot-sources phases/01-preflight.ps1 …
07-devtools.ps1 plus 15 lib/*.ps1 modules — including a PowerShell
tier-map.ps1, readiness-summary.ps1, install-report.ps1,
compose-diagnostics.ps1. macOS: installers/macos/install-macos.sh (3327
lines) is one file but speaks the same show_phase 1 6 … "30 seconds" vocabulary
(:1132, :1337, :1451, :1567, :1929) with its own
lib/{ui,detection,tier-map,env-generator,preflight-fs,bridge-manager}.sh;
llama-server runs natively with Metal on the host and Docker services reach it
via host.docker.internal through a loopback Colima bridge. WSL:
installers/dispatch.sh routes linux|wsl → install-core.sh, with divergences
isolated in detection and covered by tests/smoke/wsl-logic.sh. The cost of
replication is the "Generated Config Writers" table in
docs/INSTALLER-ARCHITECTURE.md — an explicit admission that four writers must be
edited together.
B. hal0 today (verified)
Everything below was grepped, not assumed.
Install flow. curl -fsSL https://hal0.dev/install.sh | sudo bash →
installer/bootstrap.sh (638) → installer/install.sh (4002). Bootstrap's
provenance chain is materially stronger than ODS's: a digest-pinned cosign
(per-arch sha256 constants, kept fresh by scripts/update-cosign-pin.sh and a
weekly .github/workflows/cosign-pin.yml), keyless-OIDC verification of the
channel manifest bytes before any JSON is parsed, then sha256 + Sigstore
verification of the release tarball. Channel (stable|preview|nightly) is
validated before admission (bootstrap.sh:241-261).
Installer structure. One 4002-line file with 16 ui_step calls
(UI_STEP_TOTAL=16, install.sh:246, with tests/install/test_ui_step_total.py
asserting the constant matches the call count). No step files, no lib/step
separation beyond lib/{ui,distro,preflight,run-as-hal0}.sh, no
Expects/Provides headers. The ERR trap (install.sh:248-269) is genuinely good
— it names CURRENT_STEP and prints step-specific recovery advice via a
case, including the exact ss -ltnp sport = :PORT invocation — but that advice
is hardcoded in the trap rather than owned by the step.
UI. installer/lib/ui.sh (326) is a competent narrator: ui_banner (:122),
ui_step (:149, ── (n/16) Title ──), ui_spinner_run (:178 — background
command + braille spinner + live tail of the command's last line, replaying
the last 50 captured lines on failure), ui_box (:270, ANSI-width-aware), and
info/warn/err/die (:116-119) with UI_WARN_COUNT/UI_ERR_COUNT so
preflight_all can say "passed with N warning(s)". HAL0_PLAIN=1/NO_COLOR/
non-TTY all degrade cleanly with ASCII glyph fallbacks — better than ODS.
Preflight. installer/lib/preflight.sh (2127) is deeper than ODS's engine
on substance: hal0_lxc_kind() (:155) classifies
none|lxc-privileged|lxc-unprivileged and decides which remedies are even
possible from inside; preflight_gpu() (:994) prints the exact Proxmox LXC
dev0/gid fix and has a soft/hard mode (HAL0_GPU_GATE=1) with distinct return
codes (HAL0_GPU_RC_BROKEN_GID, HAL0_GPU_RC_NO_DEVICE) so the installer can
smart-block a broken passthrough rather than "succeed" into CPU-only;
preflight_container_runtime() (:682) auto-installs podman under
HAL0_CONTAINER_REQUIRED=1; preflight_ports() (:1536) exempts hal0's own
units from "port in use". Dual-mode — sourced by install.sh, executed directly
by hal0 doctor (:1900-1907). It emits no JSON and has no fixture corpus.
Doctor. Stronger modelling than ODS. src/hal0/diagnostics.py defines frozen
dataclasses Evidence(kind, summary, data), NextStep(kind, label, target),
Diagnosis(id, severity, confidence, summary, detail, evidence, next_steps,
fixable), a frozen DIAGNOSIS_IDS taxonomy (~30 stable IDs, snapshot-tested)
and overall_verdict(). src/hal0/health_report.py (298) holds the pure
classifiers shared by hal0 doctor verify and GET /api/doctor, with a layering
test (tests/diagnostics/test_layering.py) forbidding hal0.api → hal0.cli.
cli/doctor_all.py (1285) is one read-only evidence pass; doctor_commands.py
(2272) holds perms/models/migrations/profiles/ports with --fix autofixes.
Support bundle. src/hal0/cli/doctor_bundle.py:452 (build_bundle) —
hal0 doctor bundle writes manifest.json, commands.tsv, system/, config/
(redacted via the canonical hal0.api._redact.redact_config), diagnostics/
(doctor JSONs + API payloads), logs/ (journalctl), doctor-summary.txt.
hal0 already has this.
Updater. src/hal0/updater/updater.py (5006 lines): manifest → 0700
root-only staging → sha256 re-derived from the same open file object that gets
extracted → cosign verify-blob (:870) against a GitHub-Actions OIDC identity
derived from the release kind → extract to /usr/lib/hal0-<version>/ → config
migrations → _atomic_symlink_swap() (:735, os.symlink(tmp) + os.replace)
→ re-pip into the venv, rolling the symlink back if the re-pip fails → record
/var/lib/hal0/hal0.previous. rollback() (:4851) swaps back and warns on
schema downgrade. Typed error codes (system.update_cosign_failed,
system.update_swap_failed, system.update_rollback_unavailable).
Components converge. src/hal0/components/registry.py — one ComponentDef
per updatable component (id, kind, service_id, pinned(), installed(),
converge()), with status.py merging catalog × recorded state × live probes. A
declarative reconcile model ODS has no analogue for.
Setup. hal0 setup is hidden=True and internal-only
(cli/setup_command.py:1-20); install.sh drives it as
hal0 setup --auto --no-pull --no-extensions. The Stage-2 handoff prompt was
deliberately deleted (install.sh:3985-3997) with a comment forbidding its
return. README:42 states the contract: "The installer is the whole setup."
Post-install output. A qrencode QR of the dashboard URL
(install.sh:3907-3917, soft-skips when absent), then ui_box "hal0 is ready"
with CLI/config/data paths, dashboard/chat URLs, an IPv4+IPv6 reachability
list, a repeated Verify FAILED: block for failed post-install smoke probes,
and next steps.
Tests. CONTRIBUTING.md:123-215 defines three tiers: α unit (make test,
pure pytest, ~3s, 425+ tests), β integration (documented, but the Makefile
notes it was retired in v0.2), γ release-gate (make release-test, SSH into
an hal0-test LXC, seven-row matrix, tests/release-gate-report.json, statuses
pass|fail|skip|deferred). Plus make harness (scripts/harness.sh: installer →
cli → runtime → agents → cleanup, merged JSON), make harness-install
(scripts/fresh-test-ct.sh, clones a Proxmox CT template for a full
install→smoke→uninstall→destroy cycle, appending JSONL), and
scripts/release-check.sh (pre-tag ritual). tests/installer/ has 30 files;
tests/install/ 18; tests/golden_paths/ 5. tests/release-validation/ adds an
agent-driven RC kit — kit.toml, boxes.toml, known-issues.yaml
(do-not-re-report), regressions.yaml (every prior finding re-probed each
release), lanes/{readonly,stateful,update}/*.md, committed reports/*.md —
which is ahead of ODS's VALIDATION-MATRIX.md on carry-forward learning.
Verified absent in hal0: installer summary JSON; any persisted install log
file (no tee, no LOG_FILE — output is terminal-only); a failure-report
artifact written to disk; per-step time estimates; a GUI/machine-readable
progress protocol; image-pull retry/backoff (the only pull is a fire-and-forget
(podman pull … || true) & at install.sh:2177); a preflight JSON report or
fixture corpus; a background model download with hot-swap and rollback; an
installer --dry-run; a documented per-step contract table; a high-risk change
map; a known-good-versions baseline doc.
C. Better / worse / equivalent
ODS is better:
- Installer decomposition. 14 phase files with written contracts + 19 pure
libs vs one 4002-line script. This is the biggest gap and it compounds: it is
why ODS can BATS-test
tier-map,compose-select,detection,progress, anduias units, and hal0 cannot. - Install-time forensics.
tee "$LOG_FILE"on every narrator call +write_compose_failure_report()+ aninstall-report-*.txtthat doctor later mines. hal0 loses the install transcript the moment the terminal scrolls. - Bootstrap-then-upgrade model flow. ~2 min to first token vs a 15–31 GB blocking decision at install time, plus snapshot/rollback around the swap.
- Retry discipline on network operations.
pull_with_progress(bounded attempts, backoff table, post-pull validation, non-retryable classifier) andcheck_service(backoff + curl-exit-code classification + container-state short-circuit). hal0 haswait_active— a flat 0.5s poll — and no pull retry. - Machine-readable install output.
--summary-json+ preflight JSON + doctor JSON, all merged bysimulate-installers.shinto one artifact set. - Installer dry-run as a CI lane.
- Written per-phase contracts, a reinstall contract, and a high-risk change map.
- Time estimates and a progress percentage in the UX.
hal0 is better:
- Update integrity. cosign keyless-OIDC + sha256 re-derived from the same fd
+ atomic symlink swap + venv re-pip rollback, vs
git pull origin main. Not close. - Diagnosis modelling. Typed
Diagnosis/Evidence/NextStep, a frozen ID taxonomy with a snapshot test, one shared classifier for CLI and API. ODS's doctor has the same idea but implemented as 1730 lines of bash with the shape enforced only by convention. - Preflight substance. LXC privilege classification, GPU device-node +
gid-mapping remediation, soft/hard gating with distinct return codes,
auto-install of missing runtimes. ODS's
preflight-engine.shis a threshold-checker by comparison — more testable, not more capable. - Degradation quality.
HAL0_PLAIN/NO_COLOR/non-TTY with ASCII glyph fallbacks throughoutui.sh; ODS'stype_linemerely skips the animation. - Declarative component convergence (
components/registry.py). - Release-validation carry-forward (
regressions.yaml+known-issues.yaml). - Single-entry-point discipline — no first-run wizard, enforced by
tests/installer/test_install_single_entry_point.py. ODS still ships a dashboard setup wizard alongside its installer.
Equivalent: support bundle (both redact, both best-effort); QR code; final summary card; distro/package-manager abstraction; interactive-vs-piped detection; idempotent re-run as a stated contract; ADR practice; install-time secret generation.
D. Port candidates, ranked
D.1 — Module header convention (impact: high, effort: trivial, risk: none)
ODS: docs/INSTALLER-ARCHITECTURE.md "File Header Convention".
hal0 target: installer/lib/*.sh, and every extracted step file from D.4.
Size: ~12 lines per file. Deps: none.
Adopt verbatim, renamed:
#!/usr/bin/env bash
### ============================================================================
### hal0 installer — <Module Name>
### ============================================================================
### Part of: installer/lib/ (or installer/steps/)
### Purpose: <one line>
### ### Expects: <globals/functions read>
### Provides: <globals/functions defined>
### ### Modder notes:
### <when and why you'd edit this file>
### ============================================================================
Do this first — it is the prerequisite that makes D.4 reviewable.
D.2 — Progress protocol (impact: medium-high, effort: trivial, risk: none)
ODS: installers/lib/progress.sh (25 lines) — copy essentially verbatim.
hal0 target: installer/lib/ui.sh (add to the existing lib).
### Emit structured progress events for a GUI/CI consumer.
### Format: HAL0_PROGRESS:<percent>:<phase_id>:<human message>
### Complete no-op unless HAL0_INSTALLER_GUI=1.
hal0_progress() {
local percent="$1" phase="$2" message="$3"
if [[ "${HAL0_INSTALLER_GUI:-0}" == "1" ]]; then
echo "HAL0_PROGRESS:${percent}:${phase}:${message}"
fi
}
Then one call at the top of each ui_step. Immediately buys: a parseable
install transcript for scripts/fresh-test-ct.sh, a progress bar for any future
GUI, and a way for the harness to assert which step a failed install reached.
D.3 — Persisted install log + failure report (impact: high, effort: low, risk: low)
ODS: installers/lib/logging.sh (the | tee -a "$LOG_FILE" idiom) and
installers/lib/compose-failure-report.sh:91-187.
hal0 target: installer/lib/ui.sh (info/warn/err) plus a new
installer/lib/failure-report.sh. Size: ~30 + ~150 lines. Deps: a
writable log dir. Risk: low — must not break --dev (no root) or non-TTY.
Two parts. First, tee the narrator — every info/warn/err also appends its
ANSI-stripped form to $HAL0_INSTALL_LOG. Second, a
hal0_write_failure_report invoked from the existing ERR trap, copying ODS's
report shape: privacy note → step name → environment (systemctl --failed,
podman info, podman images — each under timeout 10, a timed-out probe
recorded as evidence rather than hanging the trap on the very subsystem that
failed; correction 2026-10-04) → port occupancy with owning process →
redacted api.env/hal0.toml → last 160 lines of the install log →
journalctl -u hal0-api -n 100. Redact in two passes (correction 2026-10-04:
the key-name redactor alone is not enough for a shareable file):
hal0.api._redact.redact_config for the structured api.env/hal0.toml
section (it masks by key name, _redact.py:52-68, and its text pass knows only
a few token shapes), then — as ODS's _ods_report_redact_stream does
(compose-failure-report.sh:51-83) — collect the literal values of every
sensitive key and replace them across the whole assembled report, because a
provider token masked in the config section can still sit verbatim in the
install log, a command line, a URL or the journal under an innocuous key. Print
the path in the trap so the user has one file to attach. Payoff: hal0 doctor gains an install_artifacts surface exactly
as ODS's does, and a HAL0-INSTALL-* diagnosis family can key off it.
D.4 — Decompose install.sh into steps (impact: very high, effort: high, risk: medium)
ODS: install-core.sh + installers/phases/01..13.
hal0 target: installer/install.sh as a ~300-line orchestrator +
installer/steps/01-preflight.sh … 16-summary.sh. Size: 4002 lines
redistributed, ~0 net new logic. Deps: D.1. Risk: medium — one global
bash namespace, and any missed set -u interaction bites at install time.
Mitigate by moving one step per PR, keeping UI_STEP_TOTAL and its test green
throughout, and running make harness-install per move.
Keep ODS's exact discipline: installer/lib/* defines functions only and is safe
to source; installer/steps/* executes on source; the orchestrator sets
CURRENT_STEP (already exists) before each source. Then write
docs/internal/install-step-contracts.md modelled on
docs/INSTALLER_PHASE_CONTRACTS.md — Owns / Inputs / Outputs / Idempotency /
Failure modes per step — plus hal0's own Reinstall Contract (hal0 claims
idempotency in the README; nothing states what that guarantees). This is the
change that converts hal0's installer from "a script we're careful with" into "a
system with per-unit tests".
D.5 — Retry-protected image pulls (impact: high, effort: low, risk: low)
ODS: installers/lib/ui.sh:269-370. hal0 target: a ui_pull_with_retry
alongside ui_spinner_run, plus the Python pulls in
src/hal0/components/runner_images_arm.py / cli/runner_image_commands.py.
Size: ~70 lines bash, ~40 Python.
Copy three ideas, not the code: (1) a backoff table with env override
(5 15 30, then doubling); (2) a post-pull validation step
(podman image inspect) — a "successful" pull that produced nothing is a real
failure mode; (3) a non-retryable classifier that bails immediately on
unauthorized|denied|not found|404|no space left on device|cannot connect.
hal0's current (podman pull … || true) & at install.sh:2177 silently accepts
every one of those.
D.6 — Preflight JSON report + fixture corpus (impact: high, effort: medium, risk: low)
ODS: scripts/preflight-engine.sh + tests/contracts/test-preflight-fixtures.sh.
hal0 target: installer/lib/preflight.sh gains a --report PATH /
HAL0_PREFLIGHT_REPORT writer; tests/installer/test_preflight_fixtures.py gains
the corpus. Size: ~80 lines in preflight.sh, ~120 test lines. Deps: none.
Do not port ODS's engine — hal0's checks are better. Port the contract:
every preflight_* appends {id, severity, summary, detail, next_steps} to an
array; preflight_all serialises it, reusing the Diagnosis field names from
src/hal0/diagnostics.py so the JSON is already doctor-shaped, with
HAL0-PREFLIGHT-* IDs reserved in DIAGNOSIS_IDS. Then add the fixture table —
named scenarios with injected inputs asserting blocker counts:
linux-cpu-good → 0 blockers
lxc-unprivileged-no-gpu → 0 blockers, ≥1 warning
lxc-broken-gid → 1 blocker (HAL0-GPU-BROKEN-GID)
disk-below-floor → 1 blocker
port-8080-foreign → 1 blocker
port-8080-own-unit → 0 blockers (the exemption path)
hal0's preflight already has the hardest part (the detection); it just has no way to prove the verdicts stay stable across refactors.
D.7 — Bootstrap-then-upgrade model flow (impact: very high, effort: high, risk: medium-high)
ODS: installers/lib/bootstrap-model.sh (57), phases/11-services.sh:673-690
and :1329-1372, scripts/bootstrap-upgrade.sh (3412),
installers/lib/model-lifecycle-lock.sh (83), installers/lib/background-tasks.sh (194).
hal0 target: src/hal0/install/agent_model.py (which already computes a
curated plan and a size string) + a new src/hal0/install/background_pull.py +
a hal0-model-pull@.service transient unit. Size: ~400 lines Python if
written natively. Deps: the slot-config writer, the registry, systemd-run.
Risk: medium-high — this is the one item where copying the implementation
would be a mistake.
hal0's shape is already better suited: it has slots, a registry, and a brain model that is the "tiny model first". The port is conceptual:
- Keep the unconditional
lfm2.5-2.6bbrain pull (that is the bootstrap model). - Replace the blocking 15–31 GB agent-model prompt with: ask, then hand the pull
to a transient systemd unit (
systemd-run --uid=hal0 --gid=hal0 --unit=hal0-model-pull@…; correction 2026-10-04:--unit=only names the unit — without--uid/--gidthe pull inherits the installer's root identity and leaves model directories, status files and registry state root-owned for theUser=hal0API,packaging/sudoers/hal0-systemctl:3) rather thannohup— hal0 is a systemd product and gets journald logging andsystemctl statusfor free. Restart is not free (correction 2026-10-04): a transient unit defaults toRestart=no, so a registry or network hiccup leaves it failed; pass--property=Restart=on-failure --property=RestartSec=30s(and aStartLimitBurstthe pull can survive), or use a statichal0-model-pull@.servicetemplate that declares them. - Copy ODS's status-file contract (
data/bootstrap-status.json:status/percent/bytesDownloaded/bytesTotal/speedBytesPerSec/eta, written atomically) so the dashboard can render progress. - Copy the transaction snapshot shape verbatim
(
bootstrap-upgrade.sh:305-380) — snapshot the slot TOML + registry entry before binding, restore on failure, and use a.missingsentinel so restore can delete a file that did not previously exist:bash snapshot_file_state() { # writes <snap> or <snap>.missing if [[ -f "$src" ]]; then cp -p "$src" "$snap"; else : > "$snap.missing"; fi } restore_file_state() { if [[ -f "$snap" ]]; then cp -p "$snap" "$dst" elif [[ -f "$snap.missing" ]]; then rm -f "$dst"; fi } - Copy the verified-before-cleanup rule — only delete after the new slot has
answered a real completion (
HOT_SWAP_VERIFIED=true,bootstrap-upgrade.sh:3200-3202) — but make the what reference-aware (correction 2026-10-04). ODS replaces one bootstrap model in one server; herelfm2.5-2.6bstays bound to the brain slot while the agent model lands in a separate slot, so there is no "old model" to remove, and treating the brain model as one would strip a live slot of its weights. Cleanup must consultregistry.discover.referenced_model_ids()(discover.py:551) and go through the refcount GC inregistry/gc.py, reaping only models no slot or stack references — never the brain model. - Copy the lock:
model-lifecycle-lock.shis 83 lines and directly reusable — the installer and a background pull must not both rewrite a slot config. hal0 already hashal0_config_file_lockinconfig/loader.py; extend that rather than adding a second mechanism.
D.8 — --summary-json (impact: medium, effort: low, risk: none)
ODS: installers/phases/13-summary.sh:435-502, including the
mkstemp+fsync+os.replace atomic write. hal0 target: the install.sh
summary block, writing ${VAR_DIR}/install-summary.json by default. Size: ~40
lines. Payload: version, channel, install_root, models_dir, slots[],
services[] with start verdicts, smoke_failed[], preflight_report path.
scripts/fresh-test-ct.sh currently parses terminal output; this replaces that
with a contract.
D.9 — Time estimates on steps (impact: medium, effort: trivial, risk: none)
ODS: show_phase 4 6 "Downloading Modules" "~5-10 min + ~30 min ComfyUI build".
hal0 target: ui_step "Python environment" "~2 min" — one optional second
arg, rendered dim at the right of the rule. Honest ranges matter more than
precision; ODS's ComfyUI line is the model.
D.10 — Installer --dry-run + simulate harness (impact: medium, effort: medium, risk: low)
ODS: every phase guards mutations with if $DRY_RUN;
scripts/simulate-installers.sh runs it in CI and merges artifacts. Feasible for
hal0 only after D.4 — a 4002-line script cannot be retrofitted with dry-run
guards safely. Add HAL0_DRY_RUN=1 per step, then a scripts/simulate-install.sh
emitting artifacts/install-sim/SUMMARY.json from D.8 + D.6.
D.11 — Governance docs (impact: medium, effort: low, risk: none)
Port three, adapted: docs/HIGH_RISK_CHANGE_MAP.md → a CONTRIBUTING.md section
mapping area → risk level → required validation (hal0 has the tiers α/γ/harness/
rc-validate but no table saying which to run for a given diff — cheapest
reliability win available); docs/KNOWN-GOOD-VERSIONS.md →
docs/reference/known-good-versions.mdx with tested kernel/podman/ROCm/driver
baselines per hardware class, as the support-triage companion to the existing
hardware-matrix.mdx; and INSTALLER_PHASE_CONTRACTS.md's Reinstall Contract
→ installer/README.md, because hal0 asserts idempotency without defining it.
E. Do NOT copy
ods-update.sh's update mechanism.git fetch && git pull origin mainwith no signature verification, into the live install directory. hal0's cosign-verified, staged, atomic-symlink updater is a different class of thing; copying here would be a downgrade. (The pre-update snapshot discipline at:235-320is worth reading, buthal0.previous+ config migrations already cover that ground.)- The CRT/"lore" theme —
LORE_MESSAGES[],show_stranger_boot(),type_line_dramatic(), "THE GATEWAY IS OPEN", the terminal bell. The mechanism underneath (print something every N seconds so a long pull doesn't look hung) is worth having, andui_spinner_runalready does it better by tailing the command's real output. The ideology is not hal0's voice and adds 100+ lines of untestable string data. type_line()character-by-character animation (ui.sh:28-49) — a 0.035s sleep per character is dead wall-clock on every install and mangles log capture.scripts/bootstrap-upgrade.shas a file — 3412 lines with Windows Lemonade, macOS native-PID, and Linux Docker paths interleaved underset -uo pipefailwithout-e. Port the five ideas in D.7; not the artifact.- The Tauri desktop installer. hal0 installs
curl | sudo bashonto a headless box; a GUI wrapper solves a problem hal0 does not have. The one transferable piece is the 25-line progress protocol (D.2), useful without a GUI. - Duplicating the phase model per platform (
installers/windows/phases/*.ps1,installers/macos/lib/tier-map.sh). ODS pays for this with the "Generated Config Writers" sync table. hal0 is Linux+systemd only — do not acquire that cost speculatively. ADR-IMAGE-TAG-PINNING.md's conclusion (retain:latestfor three third-party images). hal0 pins by digest with lockstep tests (tests/installer/test_owui_pin_lockstep.py,test_hindsight_pin_lockstep.py) and a weekly cosign-pin workflow. The ADR's format is worth emulating; its decision is not.ods-preflight.sh(the runtime one) —hal0 doctor verify+health_report.pyalready cover that surface with typed results.
F. Decisions for the owner
-
Decompose
install.sh— yes or no? The load-bearing decision. 4002 lines in one file is why hal0 has no per-step tests, no dry-run, no per-step contracts, and why the ERR trap hardcodes recovery advice for four named steps. Cost: several weeks of one-step-at-a-time PRs withmake harness-installbetween each. Recommendation: yes, sequenced D.1 → D.3 → D.4, so the headers and the install log land first and make the decomposition reviewable. -
Background model pull: systemd unit or nohup? ODS uses
nohup+ a PID registry because it has no service-manager guarantee; hal0 does. Recommendsystemd-run --unit=hal0-model-pull@<slot>— journald logging,systemctl status, and restart policy for free, with the status JSON reduced to a dashboard-progress concern rather than the only observability. -
Should preflight emit
Diagnosisrows? Reusingdiagnostics.py's field names from bash means preflight output drops straight intohal0 doctor --jsonand the support bundle with no translation layer, and preflight IDs join the frozen taxonomy. The cost is a contractinstaller/lib/preflight.shmust not drift from. Recommend yes, reservingHAL0-PREFLIGHT-*IDs up front (ODS reserves IDs before the emitter lands; same pattern). -
Where does the install log live? ODS uses
/tmp/ods-install.log, lost on reboot. Options:${VAR_DIR}/logs/install-<ts>.log(survives; needs the dir before the filesystem step) or journald viasystemd-cat(survives, no ordering problem, harder to attach to an issue). Recommend the file, created in step 1, path echoed by the ERR trap — and havehal0 doctor bundleglob it and any failure report intodiagnostics/, so a support case gets evidence the user never thought to send. -
Is "the installer is the whole setup" negotiable? Currently enforced by a test and a do-not-reintroduce comment. Every port above respects it — D.7 strengthens it by removing the one install-time question big enough to make people defer. Worth re-affirming before D.4 starts, since decomposition is exactly when a "quick post-install wizard step" gets proposed.
-
What hal0 does not need from ODS: multi-platform installers, compose-stack resolution, tier→model mapping (slots + the curated ladder already generalise it), and the desktop GUI. Do not let the breadth of ODS's installer tree imply hal0 is missing surface area — hal0 is missing structure and instrumentation on the surface it already has.
04-models.md Models
Model handling: ODS vs. hal0
Scope: selection, storage, download, switching, rollback, and discovery. ODS paths are relative to /home/user/ods/ods/ unless given in full; hal0 paths are absolute under /home/user/hal0/.
A. ODS mechanism
A.1 Catalog + selection
config/model-library.json (schema "version": 2, 56 entries) is the single versioned catalog. Each entry carries id, family, gguf_file/gguf_url/gguf_sha256, size_bytes/size_mb, vram_required_gb, context_length/max_context_length, quantization, specialty, tokens_per_sec_estimate, llama_server_image, and — the distinctive part — an app_compatibility block per consumer app (openai_chat, hermes_talk, perplexica, agent_viability, …) with status, label, reason, evidence (a path into a fleet-test run directory), recordedAt, productSha/harnessSha, sometimes hostScope. This is real fleet-test provenance, not hand-written metadata: qwen3.5-2b-q4's hermes_talk.status is verified across six named hosts while openai_chat is unsupported_until_revalidated because two consumers truncated output on one host (config/model-library.json:9-63). Some entries also carry publisher_evidence, quality_evidence, source_evidence, rejected_artifacts, and runtime_profiles (VRAM/arch-gated alternate launch recipes).
scripts/select-model.py (562 lines) is a deterministic, offline selector invoked by the installer and by ods model swap. It loads only source ∈ {"", "curated"} records (load_catalog, :121-131) — HF imports in data/model-imports.json are excluded from auto-selection. usable_memory_gb() (:134-144) computes a memory envelope: unified-memory hosts get max(ram_gb*0.55, 2.0) GB, CPU/no-GPU hosts get min(max(ram_gb*0.35,3.0),8.0) GB, else raw VRAM/1024. selector_required_memory_gb() (:177-182) adds an estimated KV-cache footprint on top of on-disk size: estimated_param_billions() (:151-167) infers B-params from declared fields or a regex on id/name/gguf filename, then estimated_context_kv_gb() (:170-174) scales params_b*0.12 (clamped 0.35–3.5 GB) by context/32768. score_model() (:237-259) ranks by specialty weight (Code=4.4 > Quality=4.1 > General=3.8 > … > Bootstrap=1.0), a family-affinity bonus, a context bonus, a size term, and a headroom penalty above 92%/98% memory utilization. MODEL_PROFILE is qwen | gemma4 | auto (:38-44), auto resolving to qwen on CLOUD/0 tier and gemma4 elsewhere (:68-73); family_allowed() (:230-234) filters the catalog to one family.
Unified-memory routing (arch_policy_model, :326-364): on the qwen profile, when the tier is NV_ULTRA on arm64 (Spark/GB10) or memory_type == "unified" and the otherwise-selected model is qwen3-coder-next (is_spark_aarch64_excluded_model, :367-372), the selector force-substitutes qwen3.6-35b-a3b-ud-q4 under policy tag unified-memory-coder-next-a3b-v1, because "qwen3-coder-next produces all-? tokens on every chat completion" on those backends (:24-31,100-108). Output is JSON or --env shell assignments (LLM_MODEL, GGUF_FILE, GGUF_URL, GGUF_SHA256, MAX_CONTEXT, LLM_MODEL_SIZE_MB, MODEL_RECOMMENDATION_*, and, when a runtime_profile matched, MODEL_RUNTIME_PROFILE*/LLAMA_SERVER_IMAGE).
installers/lib/tier-map.sh (393 lines) is the static, non-catalog tier table actually used for install-time defaults and ods model swap: set_qwen_tier_config()/set_gemma4_tier_config() hard-code LLM_MODEL/GGUF_FILE/GGUF_URL/GGUF_SHA256/MAX_CONTEXT/LLM_MODEL_SIZE_MB per tier (CLOUD, ARC, ARC_LITE, NV_ULTRA, SH_LARGE, SH_COMPACT, 0-4). Two in-source incident notes matter: the NV_ULTRA block (:92-118) documents the same coder-next bug on Blackwell-aarch64, and the SH_LARGE block (:119-138) documents a live 2026-05-19 Strix Halo repro ("bootstrap-upgrade downloaded the 48.5GB coder-next-Q4_K_M.gguf, stalled at 91%, and the model would not have worked on this hardware even if the download completed") and the fix (substitute Qwen3.6-35B-A3B-UD-Q4_K_M, ~21.1 GB). tier_to_model() (:333-393) is what ods model swap <tier> uses to resolve a tier without re-running detection.
config/backends/{amd,nvidia,apple,cpu}.json are small per-backend service contracts (engine, API port/path, provider URL, runtime image) — amd.json:1-21 pins llm_engine: lemonade, lemonade-server:v10.2.0, linux_backend: rocm / windows_backend: vulkan. config/hardware-classes.json maps (platform_id, gpu_vendor, memory_type, vram_mb range) → {backend, tier, compose_overlays}; strix_unified_large matches AMD unified memory ≥ 92160 MB → SH_LARGE+docker-compose.amd.yml (:3-18). docs/CAPABILITY-PROFILE.md documents the derived CAP_* env contract install-core consumes for tier/backend/overlay normalization.
installers/lib/llama-memory-budget.sh (56 lines) is narrower than its name: it reads Docker-visible memory, takes min(host, Docker) RAM, and derives the NVIDIA llama-server container memory limit — reserve 3 GiB under 16 GiB RAM else 4 GiB, clamp usable to [1,64] GiB (:33-56). Context/batch sizing itself lives in tier-map.sh's static MAX_CONTEXT, not a separate budgeting module.
docs/MODEL-SWITCHBOARD.md (938 lines) is a PR plan, not all-implemented status: its header says the state store/data-plane/reconciler/consumer-migration work "are not implemented" as of its audit date (:5-7), but extensions/services/model-router/app/main.py (PR 3's target) exists and is wired into every non-cloud compose file (A.4) — the doc is architecture rationale/roadmap; cite code for current truth.
docs/ENGINE-PROVIDER-MODES.md defines four provider modes — local, cloud (LiteLLM → remote APIs), hybrid (local-first + cloud fallback), lemonade (ODS-managed or external) — with a shared env contract (ODS_MODE, LLM_BACKEND, LEMONADE_EXTERNAL, LEMONADE_BASE_URL/LEMONADE_CONTAINER_BASE_URL, LEMONADE_API_BASE_PATH=/api/v1) and a capability-proof contract (liveness/chat/embeddings/STT/TTS/rerank/stats must each pass through the app-visible route or the mode fails readiness) (:14-60). LiteLLM (config/litellm/{local,lemonade,hybrid,cloud,switchboard}.yaml) is the OpenAI-compatible front door in every mode.
A.2 Storage + download
Models live at data/models/*.gguf (one file per model, flat). scripts/pre-download.sh (425 lines) is legacy/independent: its own nano/edge/pro/cluster tiers map to non-GGUF HF repo ids (e.g. Qwen/Qwen2.5-32B-Instruct-AWQ) via huggingface_hub.snapshot_download (:162-176) — it shares no tiers/ids/filenames with config/model-library.json and has no SHA256 check. Not the production path.
The real download path is installers/phases/11-services.sh: checks an existing file's SHA256 against GGUF_SHA256 and deletes-and-redownloads on mismatch (:697-722); downloads with curl -fSL -C - --connect-timeout 30 --max-time 3600 --retry 3 --retry-delay 5 --retry-all-errors into a .part file so an interrupted install resumes (:751-762); on curl failure falls back to scripts/download-hf-artifact.py (89 lines), which parses a /resolve/<rev>/<file> URL and calls huggingface_hub.hf_hub_download — the documented hf_xet fallback for Xet-backed HF files plain curl can fail against (:1-7,36-56); re-verifies SHA256 after either path and hard-fails the install if it never passes (:791-829,894-897). scripts/download-hf-snapshot.py (74 lines) is the same helper for whole-repo snapshots.
installers/lib/bootstrap-model.sh pins the fast-start model as constants (BOOTSTRAP_GGUF_FILE=Qwen3.5-2B-Q4_K_M.gguf, sha, LLM_MODEL=qwen3.5-2b, MAX_CONTEXT=65536, :16-24); bootstrap_needed() (:34-52) gates on tier>0, full model absent, no --no-bootstrap, not offline/cloud/external-Lemonade. .env gets GGUF_FILE/LLM_MODEL/MAX_CONTEXT/MODEL_RECOMMENDATION_* from select-model.py --env. config/llama-server/models.ini is a plain per-model INI stanza (installers/phases/11-services.sh:929-939):
[${LLM_MODEL}]
filename = ${GGUF_FILE}
load-on-startup = true
n-ctx = ${MAX_CONTEXT}
Background hot-swap: phase 11 backgrounds scripts/bootstrap-upgrade.sh (3412 lines) with the full model's file/url/sha/name/context (installers/phases/11-services.sh:1328-1362); it downloads, re-verifies sha (:242-275), and hot-swaps llama-server with minimal downtime, safely re-triggerable by ods start/ods restart/re-install on failure.
No mmproj/vision-projector handling exists anywhere in ODS's model catalog, download scripts, or models.ini generation — ODS's local models are text-only.
A.3 Switching + rollback
ods model {current,list,swap} (ods-cli:3584-3710): current reads .env's LLM_MODEL/TIER; list prints the static tier table; swap <tier> resolves the tier via tier-map.sh, looks up the catalog id for that tier's GGUF_FILE via jq, requires the file already downloaded (:3623-3625), then POSTs {model_id, tier, context_length} to the host agent's POST /v1/model/activate (bearer-authed, key fed over stdin so it never appears in argv), treating anything but HTTP 200 + status:"activated" as failure (:3635-3697).
The activation transaction, _do_model_activate() in bin/ods-host-agent.py (12,834 lines total; handler at :7635), snapshots every file it's about to touch before mutating — .env, models.ini, config/litellm/{lemonade,local,switchboard}.yaml, config/model-router/endpoints.json, the activation receipt, Hermes/OpenCode/Perplexica config, plus container state (:7836-7876). restore_backups()/restore_previous_runtime()/rollback_and_prove() (:7878-8020) implement rollback: on any failure past the mutation point, every snapshot is restored, the previous runtime relaunched, and — critically — the previous model must itself re-pass identity+completion proof before rollback is reported successful (:7967-8020); a rollback that can't re-prove the old model is reported as a failed restoration, not a silent success. installers/lib/model-lifecycle-lock.sh is a flock-based, per-install-dir advisory lock so a background bootstrap-upgrade and a foreground ods model swap/install can never race on the same files (:15-83).
docs/MODEL-MANAGEMENT.md (466 lines) documents the six-verb dashboard flow (discover → download → load → use-through-every-app → restore → delete): multi-GPU reassignment re-runs the topology planner when a target model's VRAM envelope exceeds the current GPU subset (:137-160); context above a model's declared max is allowed with a warning but must pass runtime identity+context proof or roll back (:101-131); HF imports pin an immutable revision and reject projectors/adapters/split files (:38-70); the .env/models.ini fields ODS's activation transaction owns are read-only in the Dashboard settings editor specifically so a manual edit can't bypass verification/rollback (:194-199).
A.4 The model-router service
extensions/services/model-router/app/main.py (1126 lines) is fully implemented, not just planned. It is the Switchboard's "PR 3" data plane: a public stable alias (ods/current, default) every ODS app can hardcode, decoupling app config from the concrete model. Key mechanics: reads data/model-state.json with strict shape validation and only trusts an active route with completed identity+completion proof and a monotonic seq/routeSeq (_has_verified_active_route, :493-538) — regressed/mutated-at-same-sequence state is rejected; endpointId resolves only through a read-only, install-generated allowlist so state can never name an arbitrary upstream (:317-347,541-562); forwards only /v1/chat/completions|completions|responses (:84-88), stripping client-auth headers and injecting its own upstream credential; rewrites model back to the requested alias on every response and every SSE chunk (_SSERewriter, :680-786) while emitting X-ODS-{Request-Id,Requested-Model,Routed-Model,Backend,Route-Seq} headers; bounded queue-during-reload returns HTTP 503 model_swap_in_progress with Retry-After on timeout (:894-935); signed fleet-probe route evidence (HMAC-SHA256 over a UUID, :605-620) is recorded in a bounded TTL'd map behind an internal bearer-authed endpoint, never prompts/generations/credentials.
It exists alongside LiteLLM, not instead of it: at design time LiteLLM's alias-to-model mapping was a static YAML mount with no DB-backed management plane, so changing ods/current would otherwise mean rewriting LiteLLM's YAML and restarting it on every swap (docs/MODEL-SWITCHBOARD.md:94-98). model-router is the small, ODS-owned, hot-reloadable indirection layer; LiteLLM keeps owning auth/OpenAI-compat/multi-provider policy and always points ods/current/default/"*" at http://model-router:9099/v1 (confirmed live in config/litellm/switchboard.yaml, docker-compose.base.yml:225,299-303,328).
extensions/services/llama-server/ ships only a manifest (no compose.yaml — its compose lives in docker-compose.base.yml as a core service); the manifest declares aliases: [llm] (so ods restart llm works), port: 8080, health: /health, gpu_backends: [amd, nvidia].
A.5 AMD Strix Halo specifics
config/backends/amd.json pins the Lemonade runtime: ghcr.io/lemonade-sdk/lemonade-server:v10.2.0, linux_backend: rocm, windows_backend: vulkan, health at /api/v1/health. config/gpu-database.json has exact device matches: strix_halo_395 (device_ids: ["0x1586"], "Radeon 8060S"/"RYZEN AI MAX+ 395"/"Strix Halo", memory_mb: 98304, rdna-3.5, bandwidth_gbps: 256, compute_units: 40) → tier SH_LARGE; strix_halo_390 (RYZEN AI MAX 390, memory_mb: 65536) → SH_COMPACT (:29-68). config/hardware-classes.json gates SH_LARGE at min_vram_mb: 92160, SH_COMPACT at 65536 (both AMD, unified, linux/wsl).
SH_COMPACT ships qwen3-30b-a3b (18.6 GB, 128K ctx); SH_LARGE ships qwen3.6-35b-a3b (21.1 GB UD-Q4_K_M, 128K ctx) — not qwen3-coder-next, per the routing note in A.1/A.3 (installers/lib/tier-map.sh:119-146). docs/LEMONADE-SDK-COMPAT.md documents wrapping an already-running Lemonade SDK install (--use-existing-lemonade) without ODS managing that process or downloading its own GGUF, auto-detecting the first non-image model from Lemonade's /api/v1/models unless LEMONADE_MODEL is set, with Phase 12 proving the model via a real chat completion (:90-118). docs/AMD-ODS2-GAMMA-BRIEF.md is a go-to-market/appliance-onboarding deck (QR setup, magic-link auth, mDNS) with Strix Halo as flagship demo hardware and no model-selection content of its own; it self-flags "code present, packaged appliance images need end-to-end validation."
The one fact hal0 most needs from this domain: ODS's own fleet testing found qwen3-coder-next (an 80B-A3B MoE, ~48.5 GB Q4_K_M) produces garbage output ("all-? tokens on every chat completion") specifically on unified-memory backends — verified independently on Strix Halo (AMD, gfx1151) and NVIDIA Grace-Blackwell aarch64 (Spark/GB10) — and now hard-routes both away from it to Qwen3.6-35B-A3B-UD-Q4_K_M (installers/lib/tier-map.sh:100-138; scripts/select-model.py:24-31,326-372). The model stays in the catalog (install_recommendation: true) for manual/discrete-GPU use; it's excluded only on unified memory.
B. hal0 today
Registry (src/hal0/registry/): store.py (624 lines) is mid-cutover — TomlModelRegistry is the original atomic-tmpfile+fsync+os.replace implementation (_atomic_write, :305-341) plus a cross-process flock on a stable-inode sidecar (registry_write_lock, :101-134) and an mtime-cache, but the public name ModelRegistry is rebound at the bottom of the file to SqliteModelRegistry (:594-612) — SQLite is the actual runtime source of truth since the "ML-1" migration, TOML kept alive only for export/import-sqlite. gguf_header.py (379 lines) mmaps up to 8 MiB and walks the real GGUF v1-v3 KV block rather than trusting the filename, extracting general.architecture, general.embedding_length, general.tags, <arch>.context_length, <arch>.pooling_type, <arch>.attention.causal (:1-36,85-90). detect.py (579 lines) layers capability inference cheapest-first: header pooling_type (high confidence) → tag heuristics → filename tokens (medium/low), extracts quantization from general.file_type or a filename regex, and _hf_repo_name_from_path() (:214-229) walks up a path looking for a models--ORG--REPO HF-cache dir to recover org/repo when scanning blob-hash-named cache files directly. discover.py (717 lines) is the filesystem scanner (POST /api/models/scan + startup auto-scan), skipping HF-cache internals and non-standalone accessory dirs (mmproj, vae, clip, controlnet, …).
model_fit / capacity: model_fit.py (118 lines) — evaluate_model_fit() is a pure, reason-bearing verdict (allowed|blocked|degraded) combining model classification, slot-type match, registry resolvability, and profile/device-class compatibility (NPU/image mismatches hard-block, GPU/CPU degrade). slots/capacity.py's gtt_feasibility_verdict()/estimate_file_size_kv_mb() (:177-282) is a live memory-fit check: it reads AMD amdgpu sysfs GTT counters at request time, explicitly because on a UMA/LXC host /proc/meminfo can lie ("the actual weight allocation fails with cudaMalloc failed: out of memory", :222-228), returning fits|tight|exceeds|exceeds_total|unknown — advisory only, never blocking. POST /api/models/feasibility (src/hal0/api/routes/models.py:604-692) batches this per (model, ctx) pick for the picker UI, degrading any bad row to "unknown" rather than 500ing the batch.
profiles/: profiles are device-agnostic "logical tune" templates (chat template, sampler, KV type, batch sizes) — explicitly not hardware flags (NGL/threads/tensor-split live on the slot). 10 shipped seeds cover one profile per runtime family plus generic chat/cpu-chat/embedding/reranking; 8 workload variants (long-context, dense, MoE, named family recipes like chadrock-dense with --no-mmproj/mtp=true toggles tuned to a real model card) are demoted to legacy_seed_profiles.toml, auto-migrated once into an existing install as editable custom entries, never re-injected fresh. -fa auto (never -fa on) is a deliberate policy so a generic profile never forces an unschedulable fused-attention kernel.
capabilities/: a UX overlay grouping flat slots into "capability cards" (embed=embed+rerank, voice=stt+tts, img) via a _CHILD_TO_SLOT bridge; selections persist in capabilities.toml, and CapabilityOrchestrator.apply() reconciles against slots/*.toml through slot_config.SlotConfigStore (compute-only apply() → ChangeSet, atomic commit()/revert() across both files). NPU picks (device=npu) don't spawn a standalone slot — they drive one shared flm serve anchor serving chat+embed+ASR coresident ("NPU trio"), a multiplexing model ODS's single-llama-server design has no equivalent of. slot_config/ is the single reconciled-write owner for both files.
API routes: models.py exposes GET /api/models, GET /api/models/catalogue, POST /api/models/scan{,/preview}, POST /api/models/add-from-path, POST /api/models (register), POST /api/models/validate, GET /api/models/pulls, GET /api/models/updates/check + POST /api/models/{id}/update (HF LFS-sha diffing against the stored sha, src/hal0/registry/update_check.py:1-17 — "is a newer upstream build available," which ODS's pinned-revision catalog doesn't offer), POST /api/models/feasibility, GET/PUT/DELETE /api/models/{id}, POST .../{seed-profile,default}, POST .../pull + GET .../pull/{status,stream} (SSE) + POST .../pull/cancel, POST /api/models/inspect. hf.py adds GET /api/models/huggingface/search (thinner than ODS's — see D).
Download (src/hal0/registry/pull.py, 2169 lines): per-file streaming SHA256 (computed while downloading, :739), true HTTP Range resume via a .part+.part.json sidecar keyed by model id and filename (:546-610), and — where curl-based ODS only verifies when a hash is pre-pinned in its own catalog — hal0 extracts HF's own advertised LFS sha256 off X-Linked-ETag (:659-684) so an unpinned community pull still gets verified against what HF claims to serve. PullFile.kind: "model"|"mmproj" and a documented two-file pull shape (:128-152,973-1111) give hal0 first-class mmproj/vision support ODS has none of; CuratedModel.mmproj_file (src/hal0/registry/curated.py:107-116) lets an entry declare a companion projector downloaded alongside the main file and stamped onto Model.mmproj directly.
Curated catalog (curated.py, 1299 lines, 41 entries): code-shipped, not remote-fetched, "so users can't end up on an outdated pick list," sized for "Strix Halo's 100 GB unified pool." Its own header is candid about gaps: STT/TTS curated picks are blocked because Moonshine/Kokoro/VibeVoice weights don't fit the single-file pull shape yet (:17-58) — the wizard's STT/TTS dropdowns are empty on a fresh install. Entries carry tool_calling: bool|None ("verified, not aspirational") plus free-text notes, but no cross-app, cross-host pass/fail ledger like ODS's app_compatibility.
UI (ui/src/dash/): tests for model-filters, model-usage, model-drawer — a drawer/picker pattern rather than ODS's dedicated Models.jsx page, consistent with the flatter "everything is a slot" surface.
Brain steward install-time download (src/hal0/install/brain_model.py, 573 lines): provisions the dashboard sidebar-chat "steward" model (hal0/brain) at install time, defaulting to a hardware-blind LFM2.5-2.6B Q8_0 GGUF loadable by both the FPX runner and stock llama.cpp. Never fatal (a failed pull leaves the slot model-less, not the install broken), reuses the same run_pull_and_activate gate as every model activation, idempotent via already_pulled(). A long in-source postmortem (:53-70) documents a real 0.9.8→1.0.0 upgrade regression where scaffold/seed-loop ordering silently reverted a successful bind — the same "write the incident, then the invariant" style as ODS's tier-map.sh notes.
C. Better / worse / equivalent
ODS ahead:
1. Fleet-verified catalog metadata — app_compatibility (real run evidence, per-consumer pass/fail, host scope, product/harness SHAs) has no hal0 analogue; hal0's tool_calling flag + free-text notes cover a fraction of the same signal.
2. HF discovery + verified import UX end to end — routers/models.py's pipeline (_hf_gguf_artifacts, _hf_llm_runtime_compatibility, Hub-parsed context length, revision pinning, projector/adapter/split-file rejection, gated-repo HF_TOKEN handling) is more complete than hal0's hf.py search endpoint plus the general pull path.
3. Automatic rollback with re-proof of the old model — rollback_and_prove requires the restored previous model to itself pass identity+completion proof before rollback counts as successful. hal0 has no equivalent transactional multi-file-snapshot + app-config-fanout + verified-rollback path; a failed slot load is a state-machine transition, not a snapshot/restore across config plus every downstream consumer.
4. Stable-alias routing layer (model-router) decoupling every app from the concrete model — hal0 has no equivalent; each slot binds to a concrete model id, and a swap is a reconfigure+restart, not an atomic alias flip behind a shared gateway.
5. Static, pre-flight memory-envelope selection with documented hardware exclusions (the unified-memory coder-next routing) — hard-won hardware knowledge hal0's selection logic doesn't encode anywhere (see F).
6. Bootstrap-then-hot-swap fast start — tiny model first, full model swapped in transparently in the background, resumable independent of the foreground install; hal0's brain-model install is "never fatal" but a single pull, not a fast-start/upgrade pair.
hal0 ahead:
1. Multi-slot architecture — ODS runs one llama-server process; hal0 runs N independent hal0-slot@<name> containers (chat/embed/rerank/stt/tts/img, plus an NPU trio) with per-slot lifecycle.
2. Registry as real infrastructure, not a catalog — atomic writes, cross-process flock, mtime-cache, one-level-deep merges with explicit null-clears, and (ML-1) a SQLite-backed store behind the same interface. ODS has nothing playing this role for locally-discovered/imported models beyond flat data/model-imports.json.
3. GGUF magic-byte + KV-header parsing for capability/context/quant, vs. ODS's filename/catalog-declared metadata only.
4. Live, hardware-truth memory-fit check (gtt_feasibility_verdict against real amdgpu sysfs counters) per-request, vs. ODS's static, install-time-only envelope math.
5. mmproj/vision as a first-class, documented two-file pull shape — ODS has none.
6. HF update-checking for already-pulled models (LFS sha diff) — ODS's pinned-revision catalog doesn't offer this (and arguably shouldn't — see E).
7. Capability cards + NPU multiplexing — a richer UX/runtime model for grouping embed/voice/image slots and sharing one NPU process across chat+embed+ASR.
Roughly equivalent: both do atomic-write+fsync+rename config persistence; both have real streaming-SHA256 verification with resume; both classify GGUF-vs-non-GGUF; both fall back to a sane on-disk model when the configured one is missing (ODS's rank_models pool vs. hal0's fallback_local_model heuristic) — hal0's is arguably more careful (explicit looks_diffusion_or_nontext guard after a real incident where a 25 GB video model was picked for a chat slot).
D. Port candidates
Ordered by value/effort. "Risk" reflects integration risk in hal0, not code quality.
- Unified-memory model exclusion/substitution policy — port the knowledge, not the code. Source:
installers/lib/tier-map.sh:100-138,scripts/select-model.py:24-31,326-372. hal0 target: a check insrc/hal0/registry/curated.py(flagqwen3-coder-next,:260-275) and/ormodel_fit.py/profile_fit.pygating it off unified-memory hosts, at minimum anotes=warning and demotion from thehal0-Maxbundle'schat.coderslot (ARCHITECTURE.md:844). Size: ~20-40 lines + one bundle-manifest edit. Risk: low (pure catalog data) — but verify against hal0's actual llama-server/ROCm build first; ODS's repro was againstllama.cpp b9014specifically (see F.1). - Fleet-verified per-app compatibility metadata shape — port the schema idea:
app_compatibility.{consumer}.{status,label,reason,evidence,recordedAt}(config/model-library.json:9-63) as an optional field onCuratedModel/Model.metadata. Size: ~30 lines schema addition; populating it is a separate, larger effort. Risk: medium — no fleet harness exists in hal0 today to populate it. - Re-verify-on-existing-file SHA256 check — hal0 already streams SHA256 during pull (
pull.py:739); the portable idea is ODS's install-time re-verify-on-existing-file check (installers/phases/11-services.sh:697-722) so a corrupted on-disk file is caught and auto-redownloaded outside the pull path. hal0 target: averify_installed(model_id)helper inregistry/pull.py, callable from a CLI/API path. Size: ~40-60 lines. Risk: low. - Bootstrap-then-hot-swap fast start — for
hal0-Lite/hal0-Defaultfirst-run tiers: a small model synchronously sohal0-apiis usable within seconds, then background-pull + hot-swap the tier's real pick. Source pattern:installers/lib/bootstrap-model.sh+installers/phases/11-services.sh:670-690,1328-1362+scripts/bootstrap-upgrade.sh. hal0 target: generalizesrc/hal0/install/brain_model.pyor a newinstall/bootstrap_chat.pypaired with the bundle picker (ARCHITECTURE.md:825-859). Size: medium (200-400 lines — needs a background-task mechanism plus a hot-swap reusingrun_pull_and_activate). Risk: medium — hal0's installer currently appears synchronous. - model-router-style stable alias — not a straight port (different runtime shape), but the idea — one stable identifier a consumer binds to once — maps to hal0's existing
hal0/brainvirtual-model pattern and could generalize tohal0/chatforchat.primaryso external consumers survive amodel_swapunreconfigured. Size: large (src/hal0/dispatcher/). Risk: medium-high — touches the dispatcher's routing table and the omni-router's tool-calling gate; needs its own design pass. - "Read-only after activation" config fields — ODS makes model/tier/context fields read-only in its Settings editor specifically so a hand-edit can't bypass verified activation (
docs/MODEL-MANAGEMENT.md:194-199). hal0 target: auditslot_config'swrite_slot_toml()single-write-path claim for any UI/CLI path that still writes[model].defaultoutsiderun_pull_and_activate. Size: audit only, <1 day. Risk: low.
E. Do-not-copy
scripts/pre-download.sh— stale, a different tier vocabulary and non-GGUF HF repos than the rest of ODS, no SHA256 check. Would only confuse hal0's already-cleaner pull model.docs/MODEL-SWITCHBOARD.mdas a literal process to follow — a planning artifact whose "not implemented" disclaimer is stale (model-router is built); cite the current code (extensions/services/model-router/app/main.py) instead of the plan.- A single-process, single-model inference server as the target shape — ODS's one-
llama-server/tier-swap-the-whole-model design is a worse fit for hal0's multi-slot/capability-card architecture than what hal0 already has. Port the verification discipline (D.3/D.6), not the single-process shape. - ODS's static memory-envelope heuristics wholesale (
estimated_param_billions's filename regex,estimated_context_kv_gb's flat per-B-param constant) — hal0's live GTT-sysfs check is strictly better evidence where it applies; only worth adapting as a fallback for non-AMD/no-live-telemetry hosts. - Bundling fleet-evidence JSON paths (
fleet-test/runs/...) into a public catalog file — meaningless dead links outside ODS's private harness. If hal0 adopts the schema idea (D.2), populateevidencefrom hal0's own repo/CI.
F. Owner decisions
- Verify or retire
qwen3-coder-nextas hal0's flagship Strix Halo coder pick — the single highest-value finding here. hal0'shal0-Maxfirst-run bundle tier ("100 GB Strix Halo," hal0's own reference hardware class) setschat.coder = Qwen3-Coder-Next-80B-A3B(ARCHITECTURE.md:844), andcurated.py:260-275's entry is explicitly described as "Sized for a Strix Halo unified memory pool (~100 GB)" / "needs the full Strix Halo pool," pulling the sameunsloth/Qwen3-Coder-Next-GGUFrepo family ODS pulls. ODS's own fleet testing found this exact model produces "all-?tokens on every chat completion" on unified-memory backends including Strix Halo, verified on two different unified-memory architectures (installers/lib/tier-map.sh:100-138), and now hard-routes every unified-memory host away from it. Load this model on a real Ryzen AI Max+ 395 box and check for garbled/?-only output before shippinghal0-Max— if it reproduces on hal0's llama.cpp/ROCm build, this is a correctness bug in the flagship pick for hal0's own target hardware, not a nice-to-have. - Decide whether hal0 wants a fleet-verification ledger at all. ODS's
app_compatibilitymetadata is expensive (private, six-host harness) but genuinely differentiates its catalog. hal0 could invest in an equivalent (even single-host, CI-driven) pass, or explicitly document that curated entries are unverified beyond "it loads." - Decide the rollback bar for slot/model swaps. ODS requires the previous model to re-prove identity+completion before a rollback counts as successful — meaningfully stronger than "the state machine transitioned back." Worth adopting the same discipline for
chat.primary/brain slots that gate the dashboard's own usability. - Decide whether mmproj pull failures need a louder story — with mmproj already a first-class two-file pull (
pull.py:973-1111), is a partial pull (GGUF landed, mmproj failed) ever left silently vision-less rather than failing loud? Worth a targeted test if one doesn't exist. - Scope any model-router analogue deliberately. Decide whether the actual pain point is "apps hardcode a concrete model id" (already solved for one slot by the
hal0/brainvirtual-model pattern) versus a genuine need for LiteLLM-style multi-consumer alias indirection — different problems, different right-sized solutions.
05-network-dns.md Networking, mDNS, auth posture
Networking, DNS/mDNS, LAN access, onboarding, and auth posture: ODS vs. hal0
Scope: /home/user/ods/ods (ODS, Apache-2.0) vs. /home/user/hal0 (hal0). All
paths below are repo-relative to those roots unless given in full. Both repos
were read-only; nothing was modified.
A. ODS mechanism
A1. mDNS and the single-:80 entry
ODS's .local story has two independent layers, and the docs are explicit
about how they compose (docs/MDNS.md:5-12, docs/ODS-PROXY.md:1-20):
- Name → IP:
bin/ods-mdns.py(381 lines) is a standalone Python daemon, installed asscripts/systemd/ods-mdns.service, usingpython3-zeroconf— not a container, not host networking, a plain systemd unit as the install user. It polls.envevery 30s (ODS_MDNS_POLL_INTERVAL, default 30,bin/ods-mdns.py:93) and republishes on change; it's a no-op on macOS (Bonjour already does this) and unimplemented on Windows (bin/ods-mdns.py:338-350)._build_services()(bin/ods-mdns.py:173-259) emits two record shapes: direct-port SRV records (<device>-chat._http._tcp.local→ :3000), gated onBIND_ADDRESSactually being LAN-facing (_direct_ports_lan_reachable, lines 161-171); and per-subdomain A records (chat.,dashboard.,auth.,api.,hermes.,talk., bare<device>.local) that always resolve to the proxy's IP:port regardless ofBIND_ADDRESS, since the proxy is the one thing meant to be reachable. The installer apt/dnf/pacman/zypper-installspython3-zeroconfor falls back topip install --user(installers/phases/07-devtools.sh:479-511) and renders/enables the systemd unit (lines 518-563) — non-fatal on failure ("device still reachable by IP"). - Port → service:
extensions/services/ods-proxyis Caddy (caddy:2.11.3-alpine,extensions/services/ods-proxy/compose.yaml:5) doing host-based routing, not path-based — an earlier path-based draft broke Open WebUI's root-relative assets, websockets, and OAuth callbacks (docs/ODS-PROXY.md:22-30). The Caddyfile (extensions/services/ods-proxy/Caddyfile, 156 lines) has onehttp://block per subdomain, e.g.:
caddyfile
http://chat.{$ODS_DEVICE_NAME:ods}.local {
request_body { max_size 200MB }
reverse_proxy open-webui:8080 {
header_up X-Forwarded-Proto {scheme}
header_up X-Forwarded-Host {host}
}
}
http://{$ODS_DEVICE_NAME:ods}.local {
redir http://chat.{$ODS_DEVICE_NAME:ods}.local{uri} 302
}
(extensions/services/ods-proxy/Caddyfile:66-78,154-156)
The proxy is the one exception to the loopback-by-default rule:
ODS_PROXY_BIND defaults to 0.0.0.0 while every other service keeps
BIND_ADDRESS=127.0.0.1 (extensions/services/ods-proxy/compose.yaml:26-31,
extensions/services/ods-proxy/manifest.yaml:55-60). The
ods-session cookie is Domain=<device>.local-scoped
(ODS_COOKIE_DOMAIN, installer-set), so one magic-link redemption on
auth.<device>.local authenticates chat/dashboard/hermes subdomains too
(docs/ODS-PROXY.md:32-37). TLS is out of scope for v1 — HTTP only, with
Tailscale certs or a self-signed CA documented as follow-ups
(docs/ODS-PROXY.md:80-89).
docs/NETWORK.md covers a third, unrelated thing: Wi-Fi joining
(dashboard → dashboard-api → ods-host-agent(root) → nmcli,
docs/NETWORK.md:19-28), Linux+NetworkManager only, 501 elsewhere.
A2. Docker-internal DNS and ports
docker-compose.base.yml:7 names the project ods; the network is pinned
explicitly: networks: default: name: ods-network
(docker-compose.base.yml:473-475). Services address each other by Compose
service name on the internal port (http://llama-server:8080,
docker-compose.base.yml:94,217,251; searxng:8080, line 117). Only
dashboard-api gets extra_hosts: host.docker.internal:host-gateway
(docker-compose.base.yml:197-198), used for the Apple-silicon path where
llama-server runs natively on the host and containers reach it via
host.docker.internal:8080 (ARCHITECTURE.md:120).
config/ports.json (135 lines) is the canonical external-port contract —
one JSON array of {env_var, external_default, service_id, internal_port,
manifest_service} per service (llama-server→11434/8080,
open-webui→3000/8080, dashboard→3001, dashboard-api→3002, litellm→4000,
qdrant→6333/6334, etc.). Every port binding in every compose file is
"${BIND_ADDRESS:-127.0.0.1}:${WEBUI_PORT:-3000}:8080"-shaped
(docker-compose.base.yml:32,172,200,454), so one env var demotes/promotes
every service at once. --lan sets exactly that var
(install-core.sh:267: --lan) BIND_ADDRESS="0.0.0.0"...), and ods-proxy's
own bind is a separate, proxy-only override
(ODS_PROXY_BIND) precisely so --lan isn't required to get the friendly
URL — docs/ODS-PROXY.md:57 explicitly warns against setting global
BIND_ADDRESS=0.0.0.0 "just for this."
Phase 04 (installers/phases/04-requirements.sh:150-297) checks port
conflicts with a three-tool fallback (lsof → ss → netstat,
lines 154-213), special-cases a running host ollama process
(lines 232-256), and only checks the ports of enabled features
(lines 275-283) — voice, workflows, qdrant, comfyui ports are conditional.
A3. Headless / first-boot
Three complementary documents, all pointing at real code
(docs/HEADLESS-SETUP.md:49-59 is the index):
- AP mode (
docs/AP-MODE.md,scripts/ap-mode.sh,scripts/systemd/ods-ap-mode.service) — the device hosts its ownODS-Setup-XXXXWi-Fi AP viahostapd+dnsmasq, DNATs every :80/:443 request to the gateway IP (captive-portal trick,docs/AP-MODE.md:22-34), and lands the phone on the dashboard's/setupwizard throughods-proxy. Disabled by default — "Bringing up an AP is destructive... auto-enabling that would disconnect [users] from their own Wi-Fi" (docs/AP-MODE.md:7-11). Linux-only, requires NetworkManager release/reclaim of the wireless interface. - Setup cards (
docs/SETUP-CARD.md,scripts/generate-setup-card.py) — a 4×6", 300 DPI PNG/PDF with a Wi-Fi QR (WIFI:T:WPA;S:...;P:...;;) and a setup-URL or owner-magic-link QR side by side, meant to ship in the box (docs/SETUP-CARD.md:6-16). - Terminal QR fallback (
lib/qrcode.sh, 153 lines) —print_dashboard_qr()shells out toqrencode -t ANSIUTF8if present, else falls back to an ASCII box (lib/qrcode.sh:32-53,56-68);print_success_card()prints a boxed local+LAN URL summary plus the QR (lib/qrcode.sh:75-127). LAN IP detection reads thesrcfield ofip route get 1.1.1.1(lib/qrcode.sh:16-28). installers/mobile/install-mobile.sh(23 lines) is a stub — it detectsandroid-termux/ios-ashelland then exits 1 with "Mobile preview installer is split into a follow-up PR" (lines 12-18). Not implemented today.EDGE-QUICKSTART.mdis marked "Status: Planned — Not Yet Available" (line 3);docker-compose.edge.ymldoes not exist.
A4. Remote access + auth
- Tailscale (
docs/TAILSCALE.md,extensions/services/tailscale/) —network_mode: hostso the tailnet IP lands on the host itself (extensions/services/tailscale/compose.yaml:21); the doc is emphatic that this alone does nothing —ods-proxy+BIND_ADDRESS=0.0.0.0are still required for anything to answer on the tailnet interface (docs/TAILSCALE.md:47-54). Opt-in, Linux-only, no Funnel by default. - remote-provider-egress / remote-provider-ssh-tunnel are not
user-facing remote access — they are internal-only (
port: 0,external_link: false), an "egress boundary" that injects a private API key at the last hop for optional cloud LLM providers (extensions/services/remote-provider-egress/manifest.yaml:22-26) and an SSH transport supervisor for the same (extensions/services/remote-provider-ssh-tunnel/manifest.yaml:22-25). Do not confuse with Tailscale — this is outbound-to-a-vendor, not inbound-for-a-user. - hermes-proxy gates the Hermes agent with real cryptographic
verification, not header-presence: Caddy
forward_authtodashboard-api:3002/api/auth/verify-session(extensions/services/hermes-proxy/Caddyfile:101-129), which HMAC-verifies theods-sessioncookie (extensions/services/dashboard-api/routers/auth.py:65-107). The Caddyfile comment is candid about the prior, weaker design: "An earlier draft... only checked that the cookie HEADER was present... any LAN user can setCookie: ods-session=foo" (hermes-proxy/Caddyfile:8-12).docs/HERMES-SSO.md:9-13is equally candid that this is gateway auth, not per-user auth: Hermes itself is one process with one session token baked into the SPA HTML, so "anyone with a valid invite" shares one identity. - Magic links (
extensions/services/dashboard-api/routers/magic_link.py, 896 lines) — 32-bytesecrets.token_urlsafetokens, only the SHA-256 hash persisted, single-use guest links (60 min TTL) vs. reusable revoke-only owner cards, 5-failed-attempts/min rate limit, QR rendered server-side via theqrcodePyPI package into adata:image/png;base64URL (magic_link.py:376-395).dashboard-api/security.py(43 lines) separately gates the admin/api/*surface with a BearerDASHBOARD_API_KEY, auto-generated and written to/data/dashboard-api-key.txt(mode 0600) if unset (security.py:13-23). docs/MULTI-USER-SETUP.mdis explicit that the out-of-the-box posture is single-user: "Binds every service to127.0.0.1... Launchesllama-serverwith--parallel 1" (docs/MULTI-USER-SETUP.md:15-19); going multi-user is a six-step opt-in the operator does deliberately.docs/OAUTH_PROVIDER_SETUP.mdis a separate, narrower thing — an OAuth callback capture helper for Hermes skill setup, metadata-only, no secrets committed.
A5. Security posture summary
Default posture: everything on 127.0.0.1 except ods-proxy, which is
LAN-open by design once enabled (SECURITY.md:55-63). --lan (or
BIND_ADDRESS=0.0.0.0 + ods restart) widens every service at once
(SECURITY.md:57-68); the host-agent has its own separate, narrower
ODS_AGENT_BIND that auto-detects the Docker gateway IP on Linux rather
than defaulting to 0.0.0.0 (SECURITY.md:84-102). Behind the proxy, each
backend keeps its own check — dashboard-api's Bearer key, Open WebUI's own
WEBUI_AUTH, hermes-proxy's forward_auth — the proxy itself "adds NO auth
layer... duplicating without strengthening" (docs/ODS-PROXY.md:62-72).
privacy-shield (extensions/services/privacy-shield, port 8085) is
content-layer, not access-layer — a PII-scrubbing proxy in front of the LLM,
unrelated to who's allowed to connect.
B. hal0 today (verified by grep)
Correction to the task's framing: grep -rn -i 'avahi\|mdns\|\.local\|zeroconf\|bonjour' across hal0 returns over 300 matches in src/, installer/, ui/, tests/, and docs/ — hal0 already has a real, tested mDNS subsystem. It is architecturally different from ODS's, not absent.
src/hal0/services/mdns.py(150 lines) is the mirror ofbin/ods-mdns.py, but it does not run its own responder — it writes standard avahi service-group XML files that the system'savahi-daemoninotify-watches (services/mdns.py:10-11), the same mechanism ODS's macOS path leans on natively.mdns_hostname()returns<HAL0_HOSTNAME or socket.gethostname()>.local(services/mdns.py:40-49);advertise()/withdraw()write/prune onehal0-addon-<id>.servicefile per LAN-published, mDNS-capable service (services/mdns.py:100-141):
python
def _service_group_xml(name: str, port: int) -> str:
safe = escape(name)
return f"""<?xml version="1.0" standalone='no'?>
<service-group>
<name replace-wildcards="yes">{safe} on %h</name>
<service>
<type>_http._tcp</type>
<port>{port}</port>
</service>
</service-group>
"""
(services/mdns.py:81-97, abridged)
Only two services are mdns=True in the catalog:
services/registry.py:65-114 — OpenWebUI (:3001) and ComfyUI (:8188).
Hermes (:9119) and Hindsight (:9177) are loopback_port-only with no
mDNS/host:port fallback at all (registry.py:96-113) — closer to ODS's
"loopback stays loopback" instinct than the headline framing suggests.
- This is wired to a real UI: ui/src/dash/services.jsx has a
DiscoveryCard ("DISCOVERY (mDNS)") with an avahi active/inactive dot and
an advertise/withdraw toggle (services.jsx:279-315), backed by
GET/POST /api/services/mdns (api/routes/services.py:269-311,
useMdnsAdvertise/useMdnsStatus in useServices.ts:114-119), and
covered by tests/api/test_services_page.py:262-309 and
tests/installer/test_avahi_hostname.py (avahi-daemon.conf host-name=
pinning, installer/install.sh:1372-1451).
- Gap found in this subsystem: mdns.status() reports
base_advertised: (services_dir()/"hal0.service").is_file()
(services/mdns.py:76), i.e. whether hal0-api itself (not an addon) is
announced. Nothing in the current tree writes that file — the docstring
says "if something else drops" it (services/mdns.py:3), and
CHANGELOG.md:3676 records a stale avahi/hal0.service unit being
deleted, not a service-group file being added. In practice
base_advertised is very likely always False on a stock install; the
dashboard's own copy says as much ("No base hal0.service avahi file —
the installer writes it when avahi is present," services.jsx:304) —
a promise the installer does not currently keep.
Bind posture — confirmed exactly as briefed, with one nuance the brief undersells:
src/hal0/install/network.py:43:DEFAULT_BIND_HOST = "0.0.0.0"— this is whatinstall.shseeds into/etc/hal0/api.env.hal0-apibinds0.0.0.0:8080on a stock install, no--lanequivalent needed.src/hal0/openwebui/env_writer.py:164: OpenWebUI's companion follows the sameHAL0_BIND_HOST(also defaulting to0.0.0.0) — but this is a fix, not the original design:CHANGELOG.md:3091documents #1515/#1514, where OpenWebUI used to hardcode-p 0.0.0.0:3001:8080regardless of the operator'sHAL0_BIND_HOST, so someone who bound the API to loopback got "a wide-open chat UI on :3001 regardless."WEBUI_AUTH=FalseunlessHAL0_OWUI_TRUSTED_EMAIL_HEADERis set (env_writer.py:166-172).- Nuance: inference slot containers (llama-server, the NPU trio,
ComfyUI-as-slot) are not on
HAL0_BIND_HOST— they publish loopback-only by default ([slots].publish_host, fail-soft to127.0.0.1,src/hal0/providers/container.py:17-23,550-561), and underNetwork=hostthe code rewrites any--host 0.0.0.0/--listen 0.0.0.0in the child's own argv back to loopback, since host networking has noPublishPortfence (container.py:610-651). So: control-plane API + chat UI default open, raw inference ports default closed.src/hal0/config/network.py:34's own library fallback (no env at all) is also loopback — it's specifically the installer's choice to write0.0.0.0intoapi.envthat makes the shipped default LAN-wide. src/hal0/api/agents/_auth.py:1-8states the underlying incident plainly: "exposing a long-running JSON-RPC bridge to the hermes runtime over an unauthenticated0.0.0.0:8080is LAN-RCE."
src/hal0/service_identity.py (307 lines) resolves the box's own
service-to-service key (HAL0_ADMIN_KEY/HAL0_CLIENT_KEY, env then
/etc/hal0/api.env, service_identity.py:71-88) for hal0's own internal
callers, and owns key rotation (rotate_api_env_key, atomic tmp+rename,
0640, service_identity.py:148-234) plus keeping hindsight-llm.env in
sync (refresh_hindsight_llm_env, lines 245-296).
src/hal0/security/ — exposure.py (436 lines) is a single ordered,
first-match-wins classification table (RULES, lines 153-326) sorting
every route into OPEN / BOOTSTRAP / CLIENT / ADMIN
(AuthClass, lines 71-77), deny-by-default for anything unlisted
(classify(), lines 346-354). ratelimit.py (138 lines) backs the login
and key-rotation rate limiter. This is consumed by
src/hal0/api/auth.py (542 lines): AuthEnforcementMiddleware is a
pure-ASGI gate (lines 431-530) resolving a request to anon/client/
admin via cookie → Bearer → ?api_key= (resolve_principal, lines
289-321), plus an Origin-based CSRF check for browser-driven state changes
(_origin_allowed, lines 344-382). Crucially, require_auth_enabled()
defaults to False unconditionally (api/auth.py:128-157) — a
documented reversal: an earlier KB-1 design auto-enabled enforcement
whenever the bind was non-loopback or a key existed, and that "locked
operators out of a dashboard that shipped no login UI... so they disabled
auth wholesale" (api/auth.py:137-145). So today: LAN-open bind and
auth-off are both the shipped default, independently.
src/hal0/ports/ — authority.py (389 lines) is PortAuthority, the
single writer of port ownership (one port_claim DB row per live slot,
partial-unique-indexed, authority.py:1-19); __init__.py (229 lines,
not fully read) is the read-side harvester. Unlike ODS's static
config/ports.json contract, hal0's port map is dynamic and allocated at
runtime per slot — there is no single canonical file enumerating "which
port serves what," except the small, fixed services/registry.py catalog
for the four companion services.
Docs already say this out loud: README.md:563-565 ("Auth ships off by
default. A fresh v1.0 install is trusted-LAN-open — the right posture for a
homelab appliance"), docs/operate/auth.mdx (336 lines, full three-tier
reference, <Aside type="caution"> at lines 15-27), and
docs/concepts/security.mdx:3 ("hal0 ships open on your LAN by default,
but v1.0 adds a real, opt-in three-tier auth layer"). Install itself prints
a QR to the dashboard URL via qrencode when present
(installer/install.sh:3908-3916, HAL0_NO_QR=1 to skip) and an explicit,
if quietly-styled, line in the same summary box:
"Auth open on the trusted LAN — front with a reverse proxy if
exposed" (installer/install.sh:3934).
Issue #1822, pinned to the exact gap: src/hal0/cli/doctor_all.py:62-83,
the one function whose entire job is to flag auth misconfiguration:
def check_auth_posture(auth: dict[str, Any] | None) -> Check:
"""... The only misconfiguration we flag is "auth required but no admin
key configured" (nobody can log in); an intentionally open dev install
passes with a note.
"""
...
if not required:
return Check("auth", "Auth posture", _PASS,
"open (auth not required — dev/loopback)")
grep -n 'bind_host\|BIND_HOST\|0\.0\.0\.0' src/hal0/cli/doctor_all.py
returns nothing: this check never looks at HAL0_BIND_HOST at all. A stock
install — 0.0.0.0 bind, auth off — gets an unconditional green PASS
whose own message ("dev/loopback") describes a configuration the box is
not actually in. hal0 doctor all / hal0 doctor verify and GET
/api/doctor (security/exposure.py:252 classifies it ADMIN) are the tools
built precisely to catch this class of drift, and none of them cross-check
bind against auth. The install-time line (install.sh:3934) and the
Settings ▸ Security page (ui/src/dash/settings/pages/server/
SecurityPage.jsx:154: "Auth is off — hal0 runs trusted-LAN open") are the
only two surfaces that say it, and both require the operator to already be
looking in the right place, once, rather than being told by the tool whose
job is to tell them what's wrong.
One historical note for anyone tracing "why no Caddy/auth already":
CHANGELOG.md:5050-5053 records v0.3.0-alpha.1 as "Caddy and the auth
surface are removed" per ADR-0012, which superseded ADR-0001's softer
plan. docs/adr/ on disk (at 108b366) only goes up to 0006 and has no 0012 (reconstructed
into the tree by #2244 on main, together with 0004, 0013, 0015, 0020 and 0023) — this
is not drift, CHANGELOG.md:14 explains ADRs are filed under a
docs/internal/ tree that is gitignored (issue #638), same convention
as this repo's current docs/.devdocs//docs/superpowers/. So ADR-0012's
own reasoning is genuinely unavailable in this checkout; only its effects
(the Caddy+auth removal, and the later KB-1 opt-in re-addition) are
reconstructable from CHANGELOG.md and the code. ARCHITECTURE.md:
111-113 ("dedicated auth packages... were removed") is accurate about
that historical cut but silent about api/auth.py/security/exposure.py
existing today as an opt-in, off-by-default replacement — a live instance
of this repo's own "verify against source, not memory" rule.
C. Better / worse / equivalent
| Dimension | ODS | hal0 | Verdict |
|---|---|---|---|
| Name→IP mechanism | Own zeroconf daemon + systemd unit + pip/apt dep | System avahi-daemon + plain XML files, no extra process | hal0 lighter, equivalent outcome |
| Port→service | Single Caddy on :80, host-based vhosts, one URL | Each own port; dashboard derives links from request Host (_behind_proxy/_resolve_host, services.py:38-73) |
ODS better UX; hal0 needs no proxy at all for LAN-IP/raw-port use |
| Canonical port map | config/ports.json, static, one file |
services/registry.py (4 svcs) + dynamic PortAuthority DB for slots |
Different shapes, each fits its own domain |
| Default bind (control plane) | Loopback; proxy is the sole opt-in LAN surface | LAN-open (0.0.0.0) by default |
ODS safer default |
| Default bind (inference) | Loopback (BIND_ADDRESS) |
Loopback ([slots].publish_host) |
Equivalent |
| Auth model | Cookie SSO across Caddy subdomains + per-service checks + magic links | Deny-by-default classifier, three tiers, off by default | hal0 more rigorous design; ODS safer default |
| Auth-off visibility | N/A — LAN needs --lan, an explicit act |
Told once at install + in Settings, but doctor_all.py:75 actively says "PASS" |
ODS better — hal0's own health-check contradicts its docs |
| Headless/QR onboarding | Full stack: AP mode, setup cards, lib/qrcode.sh, mobile stub |
One QR to the dashboard URL at install; no AP mode, no card | ODS far ahead |
| Remote access | Tailscale extension, documented prerequisites, opt-in | None | ODS ahead (feature gap) |
| Egress-boundary services | remote-provider-egress/ssh-tunnel: real, scoped, internal-only | No equivalent (upstreams/ handles external providers differently) |
Different architectures, not a gap |
D. Port candidates
| # | Candidate | hal0 target file(s) | Size | Risk | Notes / excerpt |
|---|---|---|---|---|---|
| 1 | Close the #1822 gap: cross-check bind + auth in the doctor | src/hal0/cli/doctor_all.py:62-83 (check_auth_posture); needs the daemon's bind as the signal — GET /api/config/urls (api/routes/config.py; add the bind host to it if it does not carry one) or a parse of /etc/hal0/api.env, the unit's EnvironmentFile (install.sh:1605), the way service_identity.py:47-57 already reads it — not hal0.config.network.bind_host() from the CLI process: doctor runs in the operator's shell, so that call sees no HAL0_BIND_HOST and returns its loopback fallback (network.py:37-45), reproducing the false PASS (correction 2026-10-04) |
~20-30 line diff + 1 test | Very low — read-only, additive, no behavior change | Demote to _WARN when <daemon bind> not in _LOOPBACK_BIND_HOSTS and not required; message: "open on a LAN-reachable bind — run 'hal0 auth require on' or set HAL0_BIND_HOST=127.0.0.1". This is the single highest-leverage, lowest-risk fix in this whole report. |
| 2 | Write the missing base avahi file | src/hal0/services/mdns.py (extend advertise()/add a advertise_base()); hook from installer/install.sh near the existing avahi block (install.sh:835-851) |
~30-50 lines | Low — same fail-soft, tmp+rename pattern already proven for addons | Makes base_advertised (services/mdns.py:76) true on a stock install, and the dashboard's own caption (services.jsx:304) stop describing a promise nothing keeps. |
| 3 | Single-port LAN gateway (ODS-proxy-equivalent) | New packaging/hal0-proxy/Caddyfile + a systemd/quadlet unit; installer/install.sh hookup; must respect X-Forwarded-Host in _behind_proxy/_resolve_host (services.py:38-73) so links don't regress to raw ports |
~150-250 lines | Medium — WS/SSE passthrough needs the same header-stripping care hermes-proxy/Caddyfile:104-119 documents; must not fight hal0's existing request-Host link trick (useConfigUrls.ts:1-12) |
Ports almost verbatim from ods-proxy/Caddyfile's chat.<device>.local block (lines 66-78), service names swapped for 127.0.0.1:8080/:3001/:8188. Correction (2026-10-04) — two prerequisites this row omitted. (a) A stock install seeds API_BIND_HOST="0.0.0.0" (installer/install.sh:157) into api.env and publishes Open WebUI on ${API_BIND_HOST}:3001 (install.sh:1567), so a forward_auth listener on :80 leaves :8080 and :3001 reachable directly and the auth is bypassable; the gateway work must take every upstream off the LAN interface and keep that converged across upgrades before it is the single audited edge — and not by HAL0_BIND_HOST=127.0.0.1 alone: Open WebUI and every bridge-network extension reach hal0-api through host.docker.internal, the podman bridge gateway address, not loopback (openwebui/env_writer.py:102-128), so hal0-api needs a bridge-reachable listener (second bind or shared internal network) or the LAN cut is an nftables rule that exempts the bridge subnet (report 02 §D.5 (iii)). (b) src/hal0/services/mdns.py:82-96 renders _http._tcp service records on the machine hostname and publishes no A/AAAA records, so chat.<host>.local needs an address publisher (avahi-publish -a per name, or a zeroconf daemon like ODS's bin/ods-mdns.py:173-187) whose name set is pinned to the Caddyfile the way ODS's tests/test_mdns_subdomains.py pins it. |
| 4 | Headless setup card + two-QR onboarding | New installer/lib/qrcode.sh-equivalent (model: lib/qrcode.sh, 153 lines) + a hal0 setup-card command reusing detect_lan_ips (install/network.py:58-96) |
~100-150 lines | Low — additive, operator-invoked, no daemon | hal0 already has the terminal-QR primitive (install.sh:3908-3916); the gap is a printable card and, separately, a phone-driven first-boot path — install.sh is a foreground console script, not a served wizard. |
| 5 | Elevate the install-summary auth line to a real warning | installer/install.sh:3934 (currently ${DIM}-styled) |
Trivial | Very low | Cheapest partial mitigation for #1822 if the doctor fix (D1) is deferred. |
| 6 | AP mode / true out-of-box Wi-Fi setup | Greenfield: hostapd/dnsmasq/iptables orchestration, captive portal, wizard integration |
300-600+ new lines + unit + hardware testing | Medium-high — ODS itself ships this disabled, calling it "destructive" (docs/AP-MODE.md:7-11) |
Worth it only if hal0 moves toward a sealed-appliance framing, not "operator runs the installer on their own box." |
E. Do-not-copy
- ODS's dedicated zeroconf daemon wholesale.
bin/ods-mdns.py+python3-zeroconf+ its own systemd unit exists because ODS wants fine-grained control over SRV and subdomain records independent of whatever avahi happens to be doing. hal0's existing avahi-XML approach (services/mdns.py) gets the same.localresolution with zero extra process, zero extra dependency, and no polling loop — for hal0's simpler shape (one host, a couple of LAN ports) it is strictly less to run and maintain. Adopting ODS's daemon would be a regression in simplicity for no discovery-fidelity gain hal0 currently needs. - The
ODS_COOKIE_DOMAIN=<device>.localmulti-subdomain SSO trick in isolation. It solves a problem (one login acrosschat./dashboard./hermes.origins) that only exists because ODS chose host-based routing. hal0'shal0_sessioncookie (api/agents/_auth.py) is already host-only and origin-checked; grafting aDomain=scope on without first adopting subdomain routing does nothing useful and widens the cookie's reach for no reason. - AP mode, unless the product direction genuinely wants a sealed
appliance. ODS itself treats it as dangerous enough to ship disabled
(
docs/AP-MODE.md:7-11); it is Linux-only, chipset-dependent (docs/AP-MODE.md:137), and every failure mode stops the box from joining any network at all. - remote-provider-egress/ssh-tunnel as a "remote access" model. They are outbound egress-boundary services for optional cloud LLM providers, unrelated to letting a user reach the box from outside. Don't reach for them when the actual ask is "Tailscale for hal0."
- The full magic-link/owner-card persistence subsystem
(
magic_link.py, 896 lines) as a literal port. It's built for a multi-person-household product frame (revoke-only owner cards, guest invites, redemption audit trail). hal0's KB-1 model (one admin-equivalent browser session, two static keys) is deliberately simpler for a single-operator appliance; porting the whole subsystem imports a product assumption hal0 hasn't made yet.
F. Owner decisions
- Fix #1822 now, cheaply. Candidate D1 (doctor cross-check) is a
~20-line, no-risk change that stops
hal0 doctor allfrom actively asserting "PASS... dev/loopback" on a box that is neither. This should ship regardless of anything else below. - Pick a LAN-exposure model deliberately, don't drift into one.
Either (a) keep today's "each service its own port, links follow
request Host" shape and simply default
HAL0_BIND_HOSTto loopback the wayhal0.config.network's own library fallback already does (config/network.py:34), pushing LAN exposure behind an explicit opt-in the way ODS's--landoes — not as a one-line default change (correction 2026-10-04): a loopback-only hal0-api is unreachable from Open WebUI and bridge extensions, which usehost.docker.internal, so (a) also needs the bridge-reachable second listener or LAN-ingress firewall cut described in D3 and report 02 §D.5 (iii); or (b) build the single-gateway proxy (D3) and accept the added moving part in exchange for one URL and one audited edge. KB-1's own history (api/auth.py:137-145) shows an auto-enable-on-LAN-bind policy was tried and reverted because there was no login UI to catch the user — that objection no longer holds now thatdocs/operate/auth.mdxand the Settings ▸ Security page exist, so revisiting a safer default bind is worth a fresh look, not a repeat of the same argument. - Decide how much headless/appliance investment is warranted. ODS's full stack (AP mode, setup cards, mobile installer) is aimed at ship-in-a-box, hand-to-a-relative use. If hal0 stays "operator runs the installer on their own hardware," candidates D4/D5 (QR card, sharper warning) are enough; D6 (AP mode) is only worth it if hal0 explicitly adopts the sealed-appliance framing.
- Close the dead
base_advertisedpath (D2) — small, and removes a UI caption that currently describes something the installer doesn't do. - Scope remote access separately from egress. If Tailscale-for-hal0
is wanted, design it as its own feature against hal0's actual shape
(single systemd unit, not docker-compose extensions) rather than by
analogy to ODS's
remote-provider-*services, which solve a different problem.
06-mcp-memory.md MCP and memory
06 — MCP & Memory: ODS vs hal0
Source-level comparative study. All paths absolute. ODS = /home/user/ods, hal0 = /home/user/hal0.
Headline correction to the brief. The task premise — "how Hermes's
cli-config.yaml.templatedeclaresmcp_servers(stdio vs http, env passing, per-server toolsets)" — is false against the tree on disk. ODS ships no MCP anywhere in its product runtime. There is nomcp_serversblock, no MCP client, no MCP server, no MCP UI, no MCP CLI verb. Conversely, hal0 already ships ~70% of the "user-added MCP servers" feature the owner asked for (registry, REST API, CLI, manifest resolver, SSRF guard) — it is unfinished, not absent. Both findings reframe the deliverable: there is nothing to port from ODS on the MCP axis, and the hal0 work is closing a loop, not starting a feature.
Addendum, 2026-10-04 (post-review). This report is pinned to
hal0ai/hal0@108b366. Two things have happened since.
- Main shipped most of section E. ADR-0015 (
docs/adr/0015-mcp-supervisor-and-exposure.md, accepted for http/sse) and #2253 (df60d5b) added[secrets]/[tools]/[exposure]toInstalledServer, the Hermes/brain exposure join (src/hal0/mcp/hermes_join.py),hal0 mcp test|allow|gate|block|expose, andPOST /{id}/test,PATCH /{id}/tools,PATCH /{id}/exposure. The stdio supervisor is deferred (409 mcp.exposure_needs_supervisor); the dashboard drawer is #2245.docs/adr/now carries 0004, 0012, 0013, 0015, 0020 and 0023 (#2244), which resolves the bookkeeping remarks in §E.2 and §G.3.- Three design errors in §E.2–E.3 were found in review and are corrected inline below, each marked Correction (2026-10-04). In short: (i) the builtin loop at
hermes_provision.py:2012-2021attaches hal0's own admin service key, so it must not be reused verbatim for third-party URLs — main'shal0.mcp.probe.build_headerssends onlyX-hal0-Agent, the record's[env]literals and its resolved[secrets], which is the right behaviour; (ii) mirroring[tools]intohermes.tomldescribes a policy but does not enforce it, because Hermes's own MCP client connects straight tomcp_servers.<id>.urlandAgentMCPClientis never instantiated in production (true at108b366, still true at mainc177aa5) — the two builtins are gated server-side inside hal0's/mcpmount, a third-party server has nothing in its path; main inherited this and it is filed as hal0 issue #2303; (iii) astdioserver cannot run as a free-standing systemd unit, because its client must own its stdin/stdout.
A. ODS mechanism
A.1 MCP in ODS: it does not exist
An exhaustive case-insensitive sweep over the ODS tree returns seven files, and every one of them is either a Claude Code developer-tooling artifact or a passing English mention:
/home/user/ods/.claude/commands/{code-review,deep-research,team-plan}.md— Claude Code slash commands that callmcp__pal__*/mcp__rube__*tools on the maintainer's workstation. Not shipped, not installed, not part ofods/./home/user/ods/ods/docs/MDNS.md:43— "These exist for MCP clients, service-discovery tools, and the eventual ODS mobile app". Aspirational prose about mDNS_http._tcprecords./home/user/ods/ods/bin/ods-mdns.py:179— same, in a comment./home/user/ods/ods/extensions/library/services/gaia/README.md:6— describes a third-party service (GAIA) that has MCP support; ODS does not wire it./home/user/ods/ods/extensions/services/dashboard/package-lock.json— an npm transitive substring match.
Point-by-point against the brief's specific claims:
| Claim | Verdict | Evidence |
|---|---|---|
Hermes cli-config.yaml.template declares mcp_servers |
False | /home/user/ods/ods/extensions/services/hermes/cli-config.yaml.template is 186 lines: model, providers, auxiliary, agent.disabled_toolsets, terminal, platforms, compression. No mcp_servers. |
| OpenCode is given MCP servers | False | /home/user/ods/ods/installers/phases/07-devtools.sh:196-274 writes opencode.json with $schema/model/small_model/provider only. /home/user/ods/ods/extensions/services/opencode/manifest.yaml and README.md never mention MCP. |
| Open WebUI gets MCP/tool servers | False | /home/user/ods/ods/docker-compose.base.yml:100-168 is the full open-webui env block: web search, RAG embeddings, ComfyUI, STT, TTS. No TOOL_SERVER_CONNECTIONS, no mcpo. |
| n8n MCP nodes wired | False | /home/user/ods/ods/config/n8n/ holds 19 workflow JSONs; the three RAG ones (06-rag-demo.json, rag-pipeline-trigger.json, document-qa.json) contain only manualTrigger + stickyNote nodes — they are documentation stubs, not pipelines. |
| Dashboard has MCP UI | False | No mcp string in /home/user/ods/ods/extensions/services/dashboard/src/. |
| APE mediates MCP tool calls | False (and worse — see A.3) | APE is a standalone HTTP service nothing calls. |
| Users can add their own MCP servers | N/A | No mechanism, no docs, no preservation story. grep -c mcp /home/user/ods/ods/ods-cli → 0. |
The nearest thing ODS has to "user extends the agent's tool surface" is subtractive:
cli-config.yaml.template:101-104 sets agent.disabled_toolsets: [terminal, browser], relying on
Hermes's upstream built-in toolsets. Widening the surface means hand-editing
data/hermes/config.yaml per the upstream docs (docs/HERMES.md:121-130).
A.2 Memory in ODS
memory-shepherd (/home/user/ods/ods/memory-shepherd/) is not pruning, compaction, or backup
of a vector store. It is a 338-line bash script implementing a baseline-reset contract on a
markdown file:
- The agent's
MEMORY.mdis split by a---separator (memory-shepherd.sh:112). Above the line is operator-owned identity/rules/pointers; below is agent scratch. - Every 3 hours a systemd timer archives everything below the separator to
archives/<agent>/<TIMESTAMP>.mdand atomically restores the baseline (memory-shepherd.sh:151-180). - Safety rails: refuses to reset from a baseline under
min_baseline_size(default 500 bytes,memory-shepherd.sh:134), full-file backup when the separator is missing (:172-174), lock file, 30-day archive retention. - Rationale in
README.md:7-13: agents drift from role, bloat context, and "sometimes rewrite their own instructions".
Install wiring is odd: it lives in /home/user/ods/ods/installers/phases/10-amd-tuning.sh:63-95
(the AMD tuning phase), enabling memory-shepherd-workspace.timer and
memory-shepherd-memory.timer. Its shipped baselines still target
/home/deploy/.../.openclaw/workspace/MEMORY.md — OpenClaw, the deprecated agent
(ARCHITECTURE.md:63). So the mechanism is real and well-built but points at a retired surface.
Hermes memory is upstream Hermes's own: HERMES_HOME=/opt/data mounted from data/hermes/
with sessions/, memories/, skills/ (docs/HERMES.md:68-77). ODS contributes only compression
tuning (cli-config.yaml.template:171-186: threshold: 0.75, target_ratio: 0.50,
protect_last_n: 40, each with a paragraph of live-testing rationale). There is no ODS-side memory
API, no cross-agent memory, no namespace model.
Qdrant + TEI: deployed, unwired. This is the sharpest memory finding. qdrant runs
(/home/user/ods/ods/extensions/services/qdrant/compose.yaml), embeddings runs TEI
(extensions/services/embeddings/compose.yaml:3, BAAI/bge-base-en-v1.5), and the README asserts
"These vectors are stored in Qdrant" (extensions/services/embeddings/README.md:7). But nothing
writes to Qdrant. Open WebUI is pointed at TEI for embeddings only
(docker-compose.base.yml:123-129: RAG_EMBEDDING_ENGINE=openai,
RAG_OPENAI_API_BASE_URL=http://embeddings:80/v1) and keeps its own default vector DB — there is no
VECTOR_DB=qdrant / QDRANT_URI env anywhere in the compose tree. Perplexica, n8n, and Hermes
never reference it. Qdrant is a running, port-bound, API-keyed service with zero consumers.
Open WebUI local-service wiring (the part that does work, docker-compose.base.yml:115-167):
SearXNG search, TEI embeddings, ComfyUI image-gen with an inline SDXL-Lightning 4-step workflow JSON
(:141-156), Whisper STT at http://whisper:8000/v1, Kokoro TTS at http://tts:8880/v1. Every
integration is expressed as OpenAI-compatible base URLs — a clean, copyable pattern.
Langfuse is opt-in and reaches LLM traffic only through LiteLLM: the container entrypoint
appends 'langfuse' to litellm_settings.success_callback when LANGFUSE_ENABLED=true
(/home/user/ods/ods/extensions/services/litellm/compose.yaml:49-54). token-spy rides the same
seam as a callbacks entry (:46-47). Traffic that bypasses LiteLLM (Open WebUI → llama-server
direct, Hermes → llama-server direct) is untraced.
A.3 Agent tool governance: APE
APE (/home/user/ods/ods/extensions/services/ape/main.py, 1078 lines) is a genuinely sophisticated
policy engine — and nothing calls it.
The design is five layers evaluated in order in POST /verify (main.py:793-951):
- Circuit breaker (
main.py:574-612) — trips to deny-all when the deny ratio over a rolling 300s window exceeds 0.5 with ≥20 samples; tripped state persists across restarts. - Legacy per-minute rate limit (
main.py:437). - Intent classification (
main.py:635-654) — verb-token matching on the tool name, falling back to arg-shape inference (command/cmd→ ExecuteCommand,url→ NetworkFetch). - Policy evaluation (
main.py:659-695) — four modes:allow,deny,allowlist(base-command allowlist + regex deny patterns),path_guard(realpath prefix check). - Windowed multi-tier caps (
main.py:482) — per-intent 5min/hour/day limits, each tier eitherdenyorrequire_approval.
The require_approval tier is the best-engineered piece. /verify mints an approval_token bound
to an args fingerprint (main.py:891-908); /approve (main.py:954-1010) consumes the token
and mints a one-shot grant keyed to {session, tool, intent, args_hash}. The next matching
/verify consumes the grant (strictly one-shot, deleted on use), counts the retry against the
window so the bypass costs a sample, and re-escalates on a second retry (main.py:857-889). This is
notably tighter than a plain "approve tool X" model.
Everything lands in an append-only audit.jsonl (main.py:700-706) including
grant_consumed/approver provenance. State is persisted under a process lock plus advisory
flock with bounded sample lists (main.py:255-262).
But: APE_STRICT_MODE defaults to false (compose.yaml:14), which logs without blocking
(main.py:107-108 warns about exactly this). The manifest calls it a "Drop-in governance layer for
OpenClaw" (manifest.yaml:26) — OpenClaw is deprecated. A tree-wide grep for callers of :7890
or /verify outside extensions/services/ape/ finds only dashboard health/restart plumbing
(dashboard-api/config.py:544, settings.py:37). Hermes has no APE hook. The path_guard
allowed_paths were even updated for Hermes's /opt/data/{workspace,skills,memories,sessions,...}
(config/ape/policy.yaml:53-60) — someone maintained the policy for an integration that was never
built.
A.4 The one genuinely excellent ODS pattern: generated-config discipline
/home/user/ods/ods/docs/INSTALLER-ARCHITECTURE.md:118-131 carries a Generated Config Writers
table: every config surface × {Linux writer, macOS writer, Windows writer, upgrade/runtime writer}.
The framing line is "When a bug involves generated config, check every writer before calling the
fix done. This is the most common way install-time surprises survive a patch."
Two implementations of that discipline are directly reusable:
scripts/patch-hermes-config.py (324 lines) — a line-oriented, block-scoped YAML patcher that
never round-trips through a YAML parser (so comments survive). _top_level_block / _child_block /
_set_key (:17-59) locate a block by indentation and mutate only inside it. Crucially it encodes
two distinct merge policies:
### operator wins — only fill a missing key (patch-hermes-config.py:114-115)
if max_tokens and not _has_key(lines, block, "max_tokens", 2):
_set_key(lines, block, "max_tokens", str(max_tokens), 2)
### ODS converges — unconditionally re-assert (patch-hermes-config.py:215-218)
block = _set_key(lines, block, "enabled", "true", 2)
block = _set_key(lines, block, "threshold", "0.75", 2)
block = _set_key(lines, block, "target_ratio", "0.50", 2)
_set_key(lines, block, "protect_last_n", "40", 2)
It returns changed: bool and writes only on diff (:291-294) — idempotent by construction. It is
invoked against both the shipped template and the live config (scripts/bootstrap-upgrade.sh:1491-1506),
plus an in-container variant (:1508-1518).
The OpenCode jq merge (installers/phases/07-devtools.sh:225-274) does the same for JSON with a
three-way fallback: fresh-write if absent → surgical jq merge owning only .provider["llama-server"]
→ regenerate-from-template if the existing file is unparseable (issue #332).
B. hal0 today
hal0's MCP surface is substantially larger than the brief describes.
Two hosted servers. hal0-admin at /mcp/admin and hal0-memory at /mcp/memory, mounted as
FastMCP sub-ASGI apps by /home/user/hal0/src/hal0/api/mcp_mount.py. That file solves a real
problem well: FastMCP runs tool handlers in a lifespan-scoped anyio task group, so a Starlette
middleware contextvar never reaches the handler — every write silently collapsed to the shared
namespace (mcp_mount.py:15-27, issue #413). The fix reads headers off the MCP SDK's own
request_ctx at call time. It also exposes HAL0_MCP_ALLOWED_HOSTS / HAL0_MCP_ALLOWED_ORIGINS to
widen FastMCP's automatic localhost-only DNS-rebinding lockdown (:71-121).
Three-tier tool gating. AUTONOMOUS_READ_TOOLS / AUTONOMOUS_WRITE_TOOLS / GATED_TOOLS in
src/hal0/mcp/admin.py; gated calls enqueue into an in-memory ApprovalQueue
(src/hal0/mcp/approval_queue.py) and return {"status": "pending_approval", "approval_id": ...}.
The queue dedups on (tool_name, primary_target) with a hit_count
(approval_queue.py:16-22, 59-69) so a retrying agent doesn't flood the inbox. The owner resolves
via /api/agent/approvals/{id}/{approve,deny} with an SSE tail that replays the pending set on
subscribe (src/hal0/api/routes/approvals.py:13-21). approve executes the bound executor
(approval_queue.py:30-33).
Default-deny MCP client policy. src/hal0/agents/mcp_client.py — pure, synchronous, no wire IO.
enabled_servers() (:137-145) is the server axis; classify() (:149-180) is the tool axis,
checking blocked → gated → allow → unknown_tool (default-deny). guard() raises
ToolNotPermittedError on hard-reject. rewrite_path() (:235-258) resolves filesystem-MCP args
against the workspace and raises WorkspaceEscapeError on ../ or out-of-tree absolutes. The
schema (src/hal0/config/schema.py:2726-2838) enforces allow/gated/blocked disjointness at
load time with the offending tool named (ToolPolicy.lists_are_disjoint, :2766-2793) — no
silent "which check wins" behaviour. MCPServerConfig requires a URL for non-builtin servers
(:2833-2838); auth is bearer-from-env only, so tokens never live in TOML (:2682-2723).
A user-installed server registry that already exists. src/hal0/mcp/installed.py (issue #305)
persists one TOML per server at /etc/hal0/mcp-servers/<id>.toml with install / uninstall /
patch_config / list_installed, fcntl advisory locking on read-modify-write (:261-278,
issue #382), 0700/0600 perm hardening because the env block holds API keys (:111-133), a
tight id charset, and bundled-id reservation (:139-161).
A REST API and a CLI. src/hal0/api/routes/mcp.py (945 lines) serves
GET /servers|clients|catalog|resolve|stream|{id}/logs, POST /install, DELETE /{id},
PATCH /{id}/config. src/hal0/cli/mcp_commands.py (376 lines) gives
hal0 mcp list|status|install|uninstall|restart and hal0 mcp catalog list|refresh.
src/hal0/mcp/manifest.py resolves oci://, npm:/npx:, uvx:/uv:, git+https://, and
generic manifest URLs behind an SSRF guard that blocks loopback/private/link-local/CGNAT and refuses
redirects (:69-79), with a 256 KiB body cap (:65).
Hermes wiring. _phase_mcp_wire (src/hal0/agents/hermes_provision.py:2840) reads the per-agent
allow-list at /etc/hal0/agents/hermes.toml, probes each server both raw and through the Hermes
venv's own MCP client, and hard-fails when the client can't connect to a provably live server
(issue #2021 — "server-side reachability alone is NOT wiring"). The rendered set is written via
hermes config set key-value pairs (:2012-2021):
mcp_servers.<n>.{type,url,headers.X-hal0-Agent,timeout} plus a bearer. The seed write merges,
never clobbers, operator [mcp.servers.*] blocks (:248-250).
Memory. An explicit MemoryProvider ABC (src/hal0/memory/provider.py:1-14) as the anti-lock-in
seam, Hindsight as the only engine (ARCHITECTURE.md:404), a closed two-valued namespace grammar
(src/hal0/memory/namespace.py, ADR-0005) that fails closed when a read resolves to no
addressable bank (:37-43, issue #1451), and a Hermes-side plugin
(installer/agents/hermes/plugins/hal0-memory/) that treats recalled material as untrusted data,
never instruction and never interpolates it into a system/tool position (provider.py:14-18).
Known memory gaps (per the brief; #1833/#1834 corroborated in CHANGELOG.md:1869-1872, 2320-2324):
no document-ingest endpoint (#2016), a fresh uuid4 doc per turn (#2017), an uncapped extraction
queue that fails to drain on CPU-only boxes (#1834), memory_add with no preflight (#1930), and no
guide for pointing external coding agents at hal0 memory (#2153).
(Status on main c177aa5: gap 2 closed by #2253/ADR-0015 for http/sse servers — with the
enforcement caveat in the §E.3 correction; gap 1 open for stdio. See the addendum at the top.)
The three real gaps in MCP:
- No supervisor.
POST /api/mcp/{id}/{action}raisesMcpNotImplementedwith codemcp.supervisor_unavailable"pending ADR-0015" (routes/mcp.py:934-945). Installed servers are hard-codedstate="stopped"(:565). Astdioserver therefore never runs. - No exposure wiring.
/etc/hal0/mcp-servers/*.tomland/etc/hal0/agents/*.toml[mcp.servers.*]are two unjoined registries. Installing a server never reaches Hermes'smcp_servers, never reaches the allow-list, never reaches brain/OpenWebUI/OpenCode. The install route setsenv={k: "" for k in resolved.env_required}(:871) and stops. - No install UI.
manifest.py:3-6references "the dashboard's InstallDrawer paste-box";grep -rn InstallDrawer ui/returns nothing.ui/src/dash/connections.jsxrenders a read-only MCP panel plus config-builders that export hal0's servers outbound to Claude Desktop / Codex / Cursor (:688-704).
C. Better / worse / equivalent
| Axis | Verdict | Why |
|---|---|---|
| MCP support | hal0 vastly better | ODS: zero. hal0: 2 hosted servers, registry, REST, CLI, resolver, SSRF guard, per-agent policy. Nothing to port. |
| Tool-call policy model | hal0 better | hal0's server-axis + tool-axis default-deny with load-time disjointness beats APE's verb-heuristic intent classifier. APE's classify_intent guesses from the tool name (main.py:636); hal0 matches exact (server, tool). |
| Policy enforcement reach | ODS worse (nil), hal0 partial | APE is wired to nothing. hal0's AgentMCPClient is only consulted by the two bundled servers' own gating plus Hermes's allow-list gate; a third-party MCP server would be un-mediated because it never runs. |
| Rate limiting / circuit breaking | ODS better | hal0 has no equivalent to APE's sliding-window per-intent caps or its deny-ratio circuit breaker. |
| Approval model | Roughly equivalent, different strengths | hal0 dedups + executes on approve + SSE inbox, but is in-memory only (approval_queue.py:11-14) so a restart drops the queue. APE persists approvals and mints args-fingerprinted one-shot grants that count against the window — strictly tighter, but it can't execute anything. |
| Audit | hal0 better | Structured hal0.mcp.audit events through structlog→journald, consumed by GET /api/mcp/stream and /{id}/logs. APE's audit.jsonl is append-only text with no reader other than GET /audit. |
| Memory architecture | hal0 vastly better | ODS memory is upstream Hermes files + a markdown-reset timer + an orphaned Qdrant. hal0 has a provider ABC, a closed namespace grammar with fail-closed reads, private/shared banks, curation/mental-models/directives, and prompt-injection-aware recall framing. |
| Memory reliability | ODS arguably better within its scope | memory-shepherd's contract is trivially correct and observable. hal0's extraction queue is documented as unreliable on CPU-only boxes (#1834). Simple and working beats sophisticated and flaky. |
| Config-generation discipline | ODS better | The Generated Config Writers table + patch-hermes-config.py's explicit operator-wins/ODS-converges split is a discipline hal0 lacks a named equivalent for. |
| RAG plumbing | ODS worse than it looks | Qdrant + TEI deployed, only TEI consumed, README claims otherwise. A documentation-vs-source divergence of exactly the kind hal0's CLAUDE.md warns about. |
D. Port candidates
Ranked by value/risk. Nothing on the MCP axis ports from ODS — these are memory/governance/ process-discipline items.
D1. Block-scoped idempotent config patcher — HIGH value, LOW risk, ~150 LOC
- From:
/home/user/ods/ods/scripts/patch-hermes-config.py:17-59, 114-115, 215-218, 269-294 - To: new
src/hal0/agents/hermes_config_patch.py, called fromhermes_provision._phase_config_write - Why: hal0 writes Hermes config through
hermes config setkey-value pairs (hermes_provision.py:2012-2021). That is fine for scalars but has no way to remove a stalemcp_servers.<name>block when a user uninstalls a server, and no way to express "operator wins" vs "hal0 converges" per key. The ODS patcher gives both, plus comment preservation. - Borrow specifically: the
_has_keyguard (operator wins) vs unconditional_set_key(hal0 converges) distinction, and thechanged: bool/ write-only-on-diff contract. - Risk: low — pure function over lines, trivially unit-testable.
D2. Generated Config Writers table — HIGH value, ZERO risk, docs only
- From:
/home/user/ods/ods/docs/INSTALLER-ARCHITECTURE.md:118-131 - To: a new section in
/home/user/hal0/ARCHITECTURE.md, ordocs/reference/paths-and-files.mdx - Why: hal0 has more generated-config surfaces than ODS (
/etc/hal0/agents/*.toml,/etc/hal0/mcp-servers/*.toml,hermes.env,api.env,runtime.json, the Hermesconfig.yamloverlay) written from at least three places (installer,hermes_provision, REST routes). One table naming every writer per surface is the cheapest high-value doc in this report — and it becomes load-bearing the moment the MCP registry starts writing into Hermes config (§E).
D3. Windowed rate caps + circuit breaker for gated tools — MEDIUM value, MEDIUM risk, ~250 LOC
- From:
/home/user/ods/ods/extensions/services/ape/main.py:482-536(windowed limits),:574-612(breaker),:255-262(bounded sample lists) - To:
src/hal0/mcp/approval_queue.pyor a siblingsrc/hal0/mcp/limits.py - Why: hal0's approval queue dedups but does not cap. An agent in a retry loop against a distinct-args tool generates unbounded approval rows. Per-tool-class sliding windows that escalate to approval before hard-denying map cleanly onto hal0's existing three tiers.
- Adapt, don't copy: key on hal0's
(server, tool)pair, not APE's guessed intent class.
D4. Persisted, args-fingerprinted one-shot approval grants — MEDIUM value, LOW risk, ~80 LOC
### ODS: extensions/services/ape/main.py:973-989 — the grant is bound to the exact call
gkey = _grant_key(rec["session"], rec["tool_name"], rec["intent"], rec["args_hash"])
_state.setdefault("grants", {})[gkey] = {..., "approver": req.approver, ...}
- To:
src/hal0/mcp/approval_queue.py - Why: hal0's queue is explicitly in-memory (
approval_queue.py:11-14) — anhal0-apirestart drops pending approvals silently. Persisting the pending set (and binding an approval to an args hash so an approvedmodel_pull qwen3:0.6bcannot authorisemodel_pull something-else) is the hardening; the module's own docstring anticipates it: "A future ADR can promote this to a persisted table." It is not table-only, though (correction 2026-10-04): each entry executes through an in-memory_executorclosure bound at enqueue time (approval_queue.py:127,216, built inadmin.py:1887-1900), andapprove()fails withno executor bound to approvalwhen it is missing (approval_queue.py:262-276). A persisted approval therefore needs a trusted dispatcher that rebuilds execution from the stored(server, tool, args, args_hash)after re-runningToolPolicy.classifyand session validation (admin.py:1769-1795,1860) — the closure itself cannot be serialised.
D5. Baseline/scratch separator for self-edited agent files — LOW value, note only
memory-shepherd's --- contract (README.md:133-148, memory-shepherd.sh:151-180) only applies
if hal0 ever lets an agent self-edit a persona/SOUL file. It currently does not (personas.py is
hal0-authored). Record, don't port.
E. Design: user-added MCP servers for hal0
The owner's ask — "a custom way to add MCP servers rather than just be stuck with the originals
that we ship" — is 80% built. The work is finishing three things: a supervisor, an
exposure join, and a test verb. Do not build a new registry file format; /etc/hal0/mcp-servers/<id>.toml
already exists, is atomic-written, perm-hardened, and lock-protected.
E.1 Schema: extend InstalledServer, don't replace it
Current fields (src/hal0/mcp/installed.py:57-91): id, name, description, spec, transport, tools,
resources, prompts, env, enabled, installed_at, source_url, author, verified.
Add five, all defaulted so existing records keep validating:
### /etc/hal0/mcp-servers/github.toml
id = "github"
name = "GitHub MCP"
spec = "npm:@modelcontextprotocol/server-github"
transport = "stdio" # stdio | streamable-http | sse (NEW: sse)
command = "npx" # NEW — stdio only
args = ["-y", "@modelcontextprotocol/server-github@<pinned>"] # NEW — exact version, see below
url = "" # http/sse only (already implied by transport)
enabled = true
[env] # existing — literal values, 0600
GITHUB_HOST = "github.com"
[secrets] # NEW — indirection, never a literal
GITHUB_TOKEN = "GITHUB_MCP_TOKEN" # name of a key in /etc/hal0/api.env
[tools] # NEW — mirrors ToolPolicy exactly
allow = ["search_repositories", "get_file_contents"]
gated = ["create_pull_request"]
blocked = ["delete_repository"]
[exposure] # NEW — which consumers see this server
hermes = true
brain = false
openwebui = false
opencode = false
Correction (2026-10-04) — pin the executable. npx -y <pkg> resolves whatever upstream
publishes at start time, so a bridge restart could run a newer release with the configured GitHub
token while the TOML never changed. hal0 mcp install must resolve the exact version once (npm
view <pkg> version, a uv lock for uvx:), or better fetch the artifact into a hal0-owned cache
with its integrity hash, persist that in the record, and render the pinned command into the
supervised unit — the same digest-pinning posture hal0 already applies to runner images.
Design notes, each tied to an existing hal0 decision:
[secrets]is a reference, not a value. hal0 already has a secrets store at/etc/hal0/api.envbehind/api/settings/secretswith an^[A-Z][A-Z0-9_]{0,63}$name gate (src/hal0/api/routes/secrets.py:129-143), and the agent-config schema already refuses to hold tokens in TOML (AgentAuthConfig,config/schema.py:2682-2689). Keeping literalenvfor non-secret settings andsecretsfor indirection preserves both rules. New vs both repos.[tools]reusesToolPolicyverbatim (config/schema.py:2726), inheriting the disjointness validator and the empty-by-default posture. A freshly installed server therefore has zero callable tools until the operator promotes some — which is exactly the documented intent ofToolPolicy's docstring (:2737-2740). This is the "default-deny + approvals for dangerous tools" requirement, satisfied by reuse, not new code.[exposure]is new. It is the join that currently doesn't exist.hermesandbrainare the two hal0-side consumers that can be honoured immediately;openwebui/opencodeshould ship disabled and unimplemented rather than silently ignored — return501 mcp.exposure_unsupportedif set, following the precedent ofMcpNotImplemented(routes/mcp.py:57-69), which is a pattern worth keeping because the dashboard can key on the code.
E.2 Supervisor (the ADR-0015 gap)
stdio servers need a process. Follow hal0's own bundled-agent precedent — sandboxed sibling
systemd units (ARCHITECTURE.md:395-397) — rather than inventing a supervisor:
- One transient unit per enabled
stdioserver:hal0-mcp@<id>.service,User=hal0,ProtectSystem=strict,LoadCredential=for each[secrets]entry (which is howAgentAuthConfig'sbearer-from-envtokens already arrive,mcp_client.py:199-231),ReadWritePaths=limited to the server's own scratch dir. POST /api/mcp/{id}/{start,stop,restart}replaces the 501 withsystemctl --user-equivalent calls;list_serversreplaces the hard-codedstate="stopped"(routes/mcp.py:565) with the real unit state.http/sseservers need no supervision — they are already reachable; only the exposure join and the tool policy apply.
Correction (2026-10-04). The first bullet is wrong as written. A stdio MCP server speaks
JSON-RPC over its own stdin/stdout, so whoever starts the process must be its client: a
free-standing hal0-mcp@<id>.service would leave an npx/uvx server with nobody on the other end
of the pipe, and the renderer in §E.3 (which emits only {type,url,timeout}) would then point Hermes
at an empty url. Two shapes actually work:
- Hermes spawns it. Render
command/argsinto Hermes's ownmcp_servers.<id>block and let Hermes's client fork the process (only if the pinned Hermes build accepts a stdio entry; hal0's provisioner never writes one today). Cost: the server then runs insidehal0-agent@hermes's sandbox,[secrets]have to reach that unit's environment, and the process sits outside hal0's policy path (see the §E.3 correction). - hal0 runs a bridge.
hal0-mcp@<id>.serviceexecutes a small hal0-owned adapter that forkscommand args…as its child, speaks MCP to it over the pipe, and serves the same session as streamable-http on a loopback port or unix socket.LoadCredential=still delivers[secrets]— as files under$CREDENTIALS_DIRECTORY, so the adapter must read each one and set the mapped variable in the child's environment; a stdio server (like hal0's own client,agents/mcp_client.py:222) looks up a plain environment name, not a credential file (correction 2026-10-04). Nor can the unit pointLoadCredential=atapi.env: that copies the whole file into one credential (LoadCredential=ID:PATH), and the box's secrets live in that one aggregateKEY=valuefile (service_identity.py:47-57), so a per-[secrets]mapping needs a root broker — anExecStartPre=+helper that extracts only the selected keys into per-unitsystemd-creds encrypted files consumed throughLoadCredentialEncrypted=— or a per-secret store (second correction 2026-10-04) —ProtectSystem=strictstill applies, and the exposure join renders the bridge's URL — which makes a stdio server indistinguishable from an http one to everything downstream, including the policy proxy the §E.3 correction asks for.
Recommend (2). ADR-0015's "Deferred: stdio supervisor" section still sketches the free-standing unit; whoever picks that work up should start from the bridge shape.
Write ADR-0015 first. routes/mcp.py:943 already promises it, hal0's CLAUDE.md requires an ADR
before changing behaviour it covers, and docs/adr/ currently holds only 0001, 0002, 0003, 0005,
0006 while source cites 0004, 0008, 0012, 0013, 0015, 0020 and 0023 sixty times. (See §G.)
E.3 Exposure wiring — the Hermes join, done the ODS way
This is where the ODS port earns its place. On any registry mutation (install / uninstall /
patch / enable / disable), regenerate the Hermes mcp_servers block idempotently and
surgically, exactly as patch-hermes-config.py does:
- Build the desired set =
_default_mcp_servers()(hermes_provision.py:1462-1497) plus every installed record withenabled = trueandexposure.hermes = true. - For each, write
mcp_servers.<id>.{type,url,timeout}+headers.X-hal0-Agent, reusing thehermes config setmechanism behind the existing loop athermes_provision.py:2012-2021— but not its bearer. Correction (2026-10-04): that loop attachesservice_key(prefer="admin")(hermes_provision.py:2011,2020-2021) because its only targets are hal0's own ADMIN-classed/mcpmounts. Reusing it verbatim for a user-installed URL would hand hal0's platform-admin credential to a third-party server. User-installed entries carry onlyX-hal0-Agent, the record's[env]literals, and its[secrets]resolved fromapi.env(anAUTHORIZATION = "<api.env key>"entry is how a server that wants a bearer gets its own); the hal0 service identity never leaves the box. This is what main'shal0.mcp.probe.build_headers(#2253) does. Correction (2026-10-04) — no TLS gate exists. Neither the record schema (installed.py:110,url: str = Field(default="")) nor the join (hermes_join.py:136-140,237, non-empty check only) rejects a plaintexthttp://URL on a non-loopback host, so a[secrets]-backedAUTHORIZATIONheader would cross the LAN in the clear. The install verb must requirehttps://whenever the host is not loopback (unix-socket and127.0.0.1bridges exempt) and the record carries[secrets], with at most an explicit, loggedallow_insecure_httpoverride — filed as hal0 issue #2304, since main has the same gap. - Remove
mcp_servers.<id>blocks that hal0 previously wrote and no longer wants — the capabilityhermes config setcannot express, and the reason D1 is a prerequisite rather than a nice-to-have. - Mirror the record's
[tools]into/etc/hal0/agents/hermes.toml[mcp.servers.<id>]soAgentMCPClient.classify()(mcp_client.py:149) can report the policy (hal0 mcp test, the dashboard verdicts). Preserve the existing merge-never-clobber contract (hermes_provision.py:248-250). Correction (2026-10-04): the mirror does not enforce anything on the call path, and the original wording implied it did.AgentMCPClienthas no production instantiation (grep -rn "AgentMCPClient(" src/is empty at108b366and at mainc177aa5); Hermes's own MCP client connects directly tomcp_servers.<id>.url. The two builtins are safe only because their gate is server-side: hal0's/mcpmount classifies every call (mcp/admin.py:1769-1795ToolPolicy.classify→run|gated|denied|refused,is_gated+ApprovalQueue;api/mcp_mount.py:290-310). A third-party server has no such server in front of it, soblockedis not blocked andgatednever reaches the approvals inbox — and ADR-0013's "zero callable tools until promoted" default is void for exactly the servers it was written for. The design fix is to put hal0 on the wire: expose user-installed servers to Hermes only through a hal0-hosted proxy mount (e.g./mcp/ext/<id>, ADMIN-classed like/mcp) that forwardsinitialize/tools/list/tools/callupstream, hides tools outsideallow ∪ gated, routesgatedcalls through the sameApprovalQueue, rejectsblocked, and injects the record's[secrets]upstream while receiving the hal0 service bearer from Hermes like the builtins do. That also collapses step 2's bearer question into one rule (hal0 identity on the Hermes→hal0 hop, the record's own token on the hal0→upstream hop) and gives a stdio bridge (§E.2) the same front door. Until the proxy exists,exposure.hermes = truehonestly means "every advertised tool is callable" and should either say so in the UI/CLI or be refused for records whose[tools]has a non-emptygated/blockedlist. Main's #2253 shipped the mirror without the proxy; tracked as hal0 issue #2303. - Re-run the
mcp_wireprobe for the new server only, so a bad URL surfaces at install time rather than at first agent turn.
Ownership marker. Write hal0-managed blocks between sentinel comments
(# >>> hal0 mcp registry — do not edit <<< … # >>> end <<<) so step 3 can delete precisely what
hal0 owns and never touch an operator's hand-added block. This is a small addition to the ODS
block-scoping approach, which locates blocks by key name only. (Main's #2253 keeps the ownership
list in a side-car manifest, /var/lib/hal0/mcp/hermes-managed.json, instead of sentinel comments,
because comments do not survive the yaml.safe_dump hal0 already applies to this file — ADR-0015
"Rejected". Same guarantee, different carrier.)
E.4 CLI
hal0 mcp list|status|install|uninstall|restart already exist (cli/mcp_commands.py:72,119,204,239,271).
Additions:
hal0 mcp test <id>— new. Runs the existing probe path (hermes_provision._probe_mcp_server, already reused bycli/doctor_all.py:806-828) against one server and prints the advertised tool list with each tool's currentallow/gated/blocked/unknown_toolverdict fromclassify_many(mcp_client.py:264-273). This is the single highest- value new verb: it turns "I added a server" into "here is exactly what it can do and what I have permitted".hal0 mcp allow|gate|block <id> <tool>— new. Edits[tools]throughpatch_config, which already holds the lock (installed.py:281-314).hal0 mcp expose <id> --hermes/--no-hermes— new. Flips[exposure]and triggers §E.3.- Keep
addas an alias ofinstallandremoveas an alias ofuninstallif the owner prefers that vocabulary; do not rename the existing verbs.
E.5 API and dashboard
REST needs three additions to src/hal0/api/routes/mcp.py:
POST /{id}/test (probe + classify), PATCH /{id}/tools, PATCH /{id}/exposure. The existing
PATCH /{id}/config already covers env/enabled.
Dashboard: build the missing InstallDrawer that manifest.py:3-6 already assumes — paste a spec,
GET /api/mcp/resolve for the preview (SSRF-guarded, manifest.py:69-79), edit env/secrets,
install. Then extend the existing McpServerRow (ui/src/dash/connections.jsx:727) — which already
renders a per-tool drawer — with a three-state control per tool writing to PATCH /{id}/tools, and
an exposure toggle row. Reuse useMcpServers (ui/src/api/hooks/useMcp.ts:126).
E.6 Provenance
| Element | Source |
|---|---|
| Per-server TOML registry, atomic write, 0600, lock | hal0, existing — src/hal0/mcp/installed.py |
| Three-tier tool policy + disjointness | hal0, existing — src/hal0/config/schema.py:2726 |
| Approval inbox for gated tools | hal0, existing — src/hal0/mcp/approval_queue.py |
| Manifest resolve + SSRF guard | hal0, existing — src/hal0/mcp/manifest.py |
| Idempotent block-scoped config regeneration, operator-wins vs converges | ODS, ported — scripts/patch-hermes-config.py:114-115, 215-218 |
| Three-way fallback on unparseable generated config | ODS, ported — installers/phases/07-devtools.sh:225-274 |
| Writers-per-surface documentation table | ODS, ported — docs/INSTALLER-ARCHITECTURE.md:118-131 |
| Windowed caps / circuit breaker (optional, D3) | ODS, adapted — extensions/services/ape/main.py:482-612 |
[secrets] indirection block |
New |
[exposure] targets block |
New |
| hal0-owned config region with an ownership record (sentinels proposed; main shipped a side-car manifest) | New (extends ODS block-scoping) |
hal0 mcp test |
New |
stdio bridge unit — hal0-mcp@<id> running a hal0-owned stdio↔streamable-http adapter (§E.2 correction) |
New |
| Policy proxy mount fronting user-installed servers (§E.3 correction) | New |
F. Do not copy
- APE as a deployed service. A policy engine nothing calls is worse than no policy engine: it
consumes a port, a container, and a maintained policy file (
config/ape/policy.yamlwas updated for Hermes paths that never route through it) while providing zero enforcement. hal0's policy belongs in the call path (AgentMCPClient), where it already is. Port APE's algorithms (D3, D4), never its topology. APE_STRICT_MODE=falseas a default (extensions/services/ape/compose.yaml:14). Advisory-by- default governance trains operators to ignore it. hal0's default-deny posture is correct; preserve it.- Intent classification by tool-name verb tokens (
ape/main.py:635-654). Guessing that a tool namedcreate_pull_requestis aWriteFile(it contains "create") is a category error. hal0's exact(server, tool)matching is strictly better. - Deploying a vector DB with no consumer, and documenting it as if it had one. Qdrant in
ODS is a running, API-keyed, port-bound service with zero writers, while
extensions/services/embeddings/README.md:7states "These vectors are stored in Qdrant". hal0's ownCLAUDE.mdnames this failure mode; ODS is a live example. Do not add infrastructure ahead of a consumer. - memory-shepherd's periodic destructive reset as a model for hal0 memory. Hindsight-backed memory is the durable record; a timer that truncates it on a schedule would be a data-loss feature. The separator contract is portable (D5); the reset is not.
- Wiring memory into the AMD tuning phase (
installers/phases/10-amd-tuning.sh:63-95installs the memory-shepherd timers) — unrelated concerns in one phase, and presumably no timers at all on non-AMD hardware.
G. Owner decisions
- Confirm the ODS-MCP finding. ODS contributes nothing to hal0's MCP story. If the owner
expected otherwise, the expectation may come from a newer ODS branch than the one on disk
(
ARCHITECTURE.mdheader: Version 2.6.0). Worth a one-line confirmation before any further ODS-MCP archaeology is commissioned. - Write ADR-0015 (MCP process supervisor) — blocking.
routes/mcp.py:943promises it by name. Decide: systemd-unit-per-server (recommended, matchesARCHITECTURE.md:395-397) vs an in-process subprocess pool vs http/sse-only (dropstdiosupport entirely, which halves the work and covers most hosted MCP servers). Resolved on main (#2253): ADR-0015 accepted for http/sse, stdio deferred — read the §E.2 correction before implementing the deferred part. - Resolve the ADR bookkeeping gap. Source cites ADR-0004, 0008, 0012, 0013, 0015, 0020, and
0023 (60 times);
docs/adr/holds only 0001, 0002, 0003, 0005, 0006. Separately,ARCHITECTURE.md:392-393states "hal0 keeps no ADR tree" whileCLAUDE.mdsays "docs/adr/holds the accepted decision records". One of those is stale. This matters here because the MCP supervisor decision is supposed to land as an ADR. Resolved on main (#2244):docs/adr/now holds 0004, 0012, 0013, 0015, 0020 and 0023 alongside the original five. - Scope of
[exposure]. Shiphermes+brainonly in v1, withopenwebui/opencodereturning501 mcp.exposure_unsupported? Or defer the whole exposure block and wire Hermes unconditionally (simpler, but re-creates the "one hard-coded consumer" shape the owner is trying to escape)? [secrets]vs[env]. Confirm that MCP server credentials should live in/etc/hal0/api.envalongside hal0's own secrets, rather than in a separate/etc/hal0/mcp-secrets.env. The former reuses/api/settings/secretsand its name validation for free; the latter isolates third-party credentials from hal0's own. Recommend the former.- Persist the approval queue now or later? D4 is ~80 LOC and closes a real hole (restart drops pending approvals). It becomes more urgent once third-party servers can enqueue gated calls, because the blast radius of a silently-dropped approval widens.
- Adopt the Generated Config Writers table (D2)? Zero-risk docs, but it only pays off if it is
maintained. It should probably become a
CONTRIBUTING.mdchecklist item ("did you update the writers table?") rather than a standalone doc that rots. - Rate-limit gated MCP tools (D3)? Genuine hardening, ~250 LOC, and the only axis where ODS is ahead. Reasonable to defer until a third-party server actually runs.
- Memory issues are out of scope for MCP-adds but interact with it. #1930 (
memory_addno preflight) and #1834 (uncapped extraction queue) both get worse when more MCP clients can write. Consider sequencing at least #1930 before third-party servers gainmemory_*access.
07-dashboard-ux.md Dashboard and self-explanation
Dashboard & self-explanation: ODS vs. hal0
Comparative read of how ODS (Osmantic Deployment System) makes itself legible to a
non-expert, and where hal0 — a homelab appliance for a more technical operator —
matches, exceeds, or falls short of that bar. All citations are file:line against
the two repos on disk (/home/user/ods/ods/…, /home/user/hal0/…).
A. ODS UX inventory
Nav / IA. The sidebar (extensions/services/dashboard/src/components/Sidebar.jsx)
and route registry (extensions/services/dashboard/src/plugins/core.js:25-121) expose
only seven top-level items — Dashboard, GPU Monitor (hidden unless gpu_count>1,
core.js:44), Extensions, Integrations, Models, Remote GPU, Settings — with Usage and
"Setup / Owner" deliberately kept off the top nav (core.js:87-90: "a
factory/distributor/service-provider flow, not a day-to-day dashboard surface").
Agents (Hermes), Privacy, Workflows, Features and Updates are not separate pages —
they're sections/cards embedded in Dashboard and Settings, keeping the nav shallow
while the underlying surface area is large.
| Page | Question it answers | Actions offered | Status vocabulary | Empty/error handling |
|---|---|---|---|---|
Dashboard (pages/Dashboard.jsx) |
"Is my system healthy, and what can I turn on?" | Click a Feature card to launch it; jump to GPU Monitor | healthy/degraded/not_deployed rolled into computeHealth() (Dashboard.jsx:30-40, "3/4 core services online."); feature status enabled/available/services_needed/insufficient_vram normalized to ready/disabled for display (:132-144) |
Loading: "Linking modules... reading telemetry..." (:679); feature fetch failure silently degrades to status-only view (comment :623) |
FirstBoot wizard (pages/FirstBoot.jsx, 717 lines) |
"What is this box and who am I?" | 4 taps: label the setup → name the first user → pick a stack (Chat only / Chat+Agents / Full Stack) → Finish, generating a QR owner card | n/a (linear wizard) | Owner-card-unavailable and finish-error states both explain why and what to do next (:547-562) |
Extensions (pages/Extensions.jsx, 1328 lines) |
"What can I install, and what state is it in?" | Install/Enable/Disable/Remove/Purge, per-extension console, live install progress | 10-value enum each with a plain-English STATUS_DESCRIPTIONS sentence (:60-84, e.g. unhealthy → "Container is running but health check is failing — check logs") |
friendlyError() maps raw backend strings to sentences (:38-56); a "polling lost" banner with auto-recovery tracking (:111-160) |
Integrations (pages/ServiceMap.jsx) |
"How do these services actually talk to each other?" | None (read-only topology) | 4-color legend: healthy/degraded/down/not deployed (:332-337) |
"Topology data unavailable: {error}" (:314) |
Models (pages/Models.jsx, 1970 lines) + HuggingFaceModelBrowser |
"What models do I have, what can I add?" | Search HF, pull, assign, insights panel | download/insufficient-VRAM style badges | — |
Remote GPU (pages/RemoteProvider.jsx) |
"Is my cloud-burst route working?" | "Test route" probe, refresh | StatusPill |
Inline red banner with the raw error |
Usage (pages/Usage.jsx) |
"What is this costing me, and can I trust the number?" | Enable/Restart Token Spy inline | readiness missing/unconfigured tones; a "Tracking Source Guide" panel labelled "Honest by design" explains each cost-source tag (actual_billed, priced_from_tokens, local_zero_cost, untracked) (:761-799) |
UsageReadinessPanel shows message + detail + a real "Enable Usage Tracking" button with busy state (:511-562) |
Setup / Owner (pages/Invites.jsx) |
"Who has a key to this box?" | Create/revoke owner cards and guest invites (QR) | — | Icon+heading+explanation+CTA empty states (:304-339, "No owner cards yet") |
GPU Monitor (pages/GPUMonitor.jsx) |
"Is my GPU the bottleneck?" | none | per-GPU aggregate bars | "GPU data unavailable" / {error} (:38-49) |
Settings (pages/Settings.jsx + components/settings/EnvEditor.jsx) |
"What can I change, and what happens if I do?" | Save / Apply changes (recreates only the affected containers) | field-level required/enum/type errors in plain English (settings.py:244-289) |
BehaviorCards literally state Save/Restart/Apply behavior (EnvEditor.jsx:268-279); Apply is disabled with an explanation until it has a plan (:271-273) |
Feature/Services/Extensions split. "Features" (Chat, Voice, Document Q&A,
Workflows…) are outcome-level, declared as data inside each service's
manifest.yaml (extensions/services/llama-server/manifest.yaml:24-39) and
computed server-side into enabled/available/services_needed/insufficient_vram
(dashboard-api/routers/features.py:19-110). "Services" are the technical
containers; "Extensions" is the installer/marketplace layer over them. A
NON_USER_FACING_LINK_SERVICES set (Dashboard.jsx:73-84) hides infra services
(litellm, qdrant, embeddings…) from click-through — deliberate complexity hiding.
Hardware tiers get human names (installers/lib/tier-map.sh: "Entry Level",
"Prosumer", "Pro", "Enterprise", "Strix Halo 90+", "Cloud (API)"), and GET
/api/features returns plain-English GPU-tier advice (features.py:162-177, e.g.
"Entry-level GPU — focus on chat first").
Self-explanatory API shapes. features.py:130-144 turns "available"/
"services_needed" into first-person suggestions ("Your hardware can run Document
Q&A. Enable it?" / "Voice needs whisper, tts to be running."). GET
/api/features/{id}/enable (:188-250) returns numbered steps + deep links per
feature, rendered by components/FeatureDiscovery.jsx:246-300 as a modal — click
"Enable Document Q&A" and you get "1. Ensure Qdrant vector database is running… 2.
Open Open WebUI…" with a working "Open Chat" button. routers/privacy.py:44-51,
68-90 puts a human message on every response ("Privacy Shield is active" /
"…is not running. Check: docker compose ps privacy-shield"). routers/workflows.py:166-168
turns a 400 into "Missing dependencies: qdrant. Enable these services first."
components/TroubleshootingAssistant.jsx pairs Symptoms → Likely cause → Solutions
with copy-to-clipboard commands, auto-surfacing the ones relevant to whatever is
currently unhealthy.
First-run / guidance surfaces. installers/lib/ui.sh:97-103 defines a
consistent "AI narrator" voice (ai/ai_ok/ai_warn/ai_bad/signal); :154-168 carry
brand-voice lore lines; :507-597 show_install_menu ("Choose how deep you want
to go. I can install everything, or keep it minimal.") auto-disables ComfyUI on
low-VRAM tiers with an explicit follow-up ("You can enable it later with: ods
enable comfyui"); :600-632 show_success_card closes with "THE GATEWAY IS
OPEN" / "Your data never leaves this machine." Phase 13
(installers/phases/13-summary.sh) writes a desktop shortcut + GNOME sidebar pin
(:229-256), retries preflight validation up to 3× with a spoken wait message
(:170-183), and writes a versioned, atomically-written summary JSON
(:435-501). docs/SETUP-CARD.md documents a script that renders a physical 4×6,
300-DPI card with two QR codes (Wi-Fi + setup URL) for zero-typing onboarding of a
boxed unit. docs/POST-INSTALL-CHECKLIST.md is a 6-step "does it actually work"
script. docs/HOW-ODS-SERVER-WORKS.md explains the whole microservice mesh as "an
office building" with each service as a named team member — no jargon. The outer
README.md:120 links that guide plus "listen to the audio
version" (a Spotify
episode), then :124-139 gives an "At A Glance" Q&A table and an "If you know
Ollama / Open WebUI / AnythingLLM / n8n… ODS adds…" table that anchors newcomers to
tools they already know. templates/*.yaml are outcome-named install bundles
("Creative Studio", "Developer Homelab", "Voice Assistant") consumed by
components/TemplatePicker.jsx. ods status (ods-cli:1273-1400) prints
sectioned ━━━ ODS Status ━━━ / Health Checks / GPU Status blocks with ✓/⚠ per
service, plus --json.
B. hal0 UX inventory
Nav / IA. ui/src/dash/chrome.jsx:480-524 (useNavItems) lists Overview,
Slots (children: Endpoints, Runner Images, Stacks), Models (children: Profiles),
Benchmarks, Agents (children: Memory, MCP), Services, Logs, Settings. Compared to
ODS, the vocabulary is implementation-first: "Slots", "Runner Images", "MCP",
"Profiles", "Stacks" are hal0's own internal nouns exposed directly, not translated
into outcomes ("Chat", "Voice"). This matches the product's own framing — see
README.md's target hardware and register — but it does mean the nav teaches
hal0's architecture before it teaches what you can do.
| Page | Question it answers | Actions offered | Status vocabulary | Empty/error handling |
|---|---|---|---|---|
Overview (dash/dashboard-redesign.jsx, active; dashboard.jsx is the fallback) |
"Is the box up and who's using memory?" | Quick actions: restart agent / pull model / evict idle / open webui (:166-215) |
slot phases missing/pulling/starting/serving/ready/idle/stopped/crashed (slot-status.js:8-16) |
RDAttentionCard: "nothing needs you" when clear (:920); zero-slot state: "No models configured yet… Run hal0 setup in your terminal" (dashboard.jsx:253-263) |
Slots (slots.jsx, 1118 lines) |
"What's loaded, on what device, doing what right now?" | Load/Stop/Restart/Swap, per-slot capability toggles | color-coded dot + tooltip (slot-status.js:158-226, e.g. "Container stopped (auto-reloads on next request)") |
pull-progress bar with live label (:106-124) |
Models (models.jsx) |
"What's on disk / pullable from HF?" | Pull, filter, assign to slot | pull job states | — |
Agents overview (agents/agents-overview.jsx) |
"What agents can drive this box?" | Restart Hermes, view persona | live/health per agent card, rendered as collectible trading cards (rarity, abilities, "pow") (:29-50) |
roadmap agents shown greyed "coming soon" |
Brain chat (board/agent-chat.jsx) |
"Talk to the steward model directly" | send/stop | placeholder="Ask hal0-brain…" vs "chat unavailable" (:354) |
— |
Board (board/board-view.jsx) + Approvals (chrome.jsx:1101-1214) |
"What is an agent waiting on me for?" | Approve/Deny gated tool calls | "no pending approvals" empty state; reassuring copy: "Calls block until you approve or deny — agents pause cleanly." (:1114) |
badge is poll-driven, no arrival toast (issue #1994) |
Memory (memory-overview-v2.jsx, memory-map.jsx, telemetry-header.jsx) |
"How much unified memory is free, by whom?" | delete/curate facts, approve deletes | ruler + bar segments | header vs. bar computed "free" from two different bases on some boxes (issues #1928/#1929) |
Journal / Logs (activity-log.jsx, routes/journal.py, routes/logs.py) |
"What just happened?" | filter/expand | — | — |
Stacks (stacks.jsx, 876 lines) |
"What's my named slot+profile+model bundle?" | New/Clone/Snapshot/Import/Export/Delete | breadcrumb descriptor only: "Stacks · runtime · slot + profile + model bundles" (:793-798) |
"No stacks yet — create one, snapshot the live config, or import a .hal0stack.json." (:833) |
Settings → Capabilities/Model Defaults/Security/Doctor/… (settings/pages/**) |
"What's configured, and does changing it need a restart?" | Save; one-click Restart hal0-api with confirm + live health-poll + toast (settings/shared/RestartApiPanel.jsx) |
per-field ApplyBadge: green "live" / amber "⟳" / red "⚠ manual restart" (settings/shared/ApplyBadge.jsx:9-16) |
PanelFooter: "capability probe failed — Save disabled" + Retry (capabilities/shared.jsx:33-46) |
Guidance/diagnosis surfaces. hal0's nearest equivalent to ODS's Troubleshooting
Assistant + Feature-enable modal is the typed Diagnosis model
(src/hal0/diagnostics.py:41-80: id/severity/confidence/summary/detail/evidence/
next_steps/fixable), fed by GET /api/doctor (src/hal0/api/routes/doctor.py)
into a single generic DiagnosisPanel.jsx that already renders next_steps as
chips ($ command / ↗ doc / • manual, :31-44). The scaffolding is more
rigorous than ODS's ad hoc per-router strings — one classifier
(src/hal0/health_report.py) is shared verbatim by hal0 doctor verify --json
and the API, so the two can't drift. The install-time and per-setting
remediation UX (RestartApiPanel, ApplyBadge, PanelFooter) is genuinely
strong and, in places, ahead of ODS (a real restart-and-confirm loop with a
success/failure toast, not just a banner).
First-run. There is no wizard: README.md:39-45 states this as policy —
"⚡ The installer is the whole setup. There is no separate first-run wizard
afterwards." README.md:160-183 ("First ten minutes") is entirely
CLI/curl. docs/ is a Starlight site with a conventional
getting-started/concepts/guides/reference split (docs/getting-started/*.mdx,
docs/concepts/*.mdx, docs/guides/*.mdx, docs/reference/*.mdx) — better
taxonomy than ODS's flat 60-file docs/, but reference-heavy (cli.mdx,
config-schema.mdx, env-vars.mdx, mcp-tools.mdx) rather than beginner-first.
docs/getting-started/index.mdx:109-114 does disclose the no-auth-by-default LAN
posture up front, in a Starlight <Aside type="caution">.
C. Where hal0 falls short (evidenced) — and where hal0 is ahead
1. The remediation affordance is designed but not wired end-to-end.
Diagnosis.next_steps and NextStep(kind="command"|"manual"|"doc")
(diagnostics.py:41-56) are real, and DiagnosisPanel.jsx:31-44 already renders
them as chips. But the classifier actually wired to /api/doctor
(health_report.py:246-283, to_diagnosis()) hard-codes next_steps=[] for
every one of its 7 checks — even though the richer CLI path
(src/hal0/cli/doctor_commands.py:887,899,911,1208,1251,1272,1304,1331,1645,1934,
1953) already knows real remediation commands for adjacent checks (e.g. hal0
model scan, podman pull <image>). And even where a chip is populated, its
target is shown only as a title= tooltip (DiagnosisPanel.jsx:38) — not
click-to-run, not click-to-copy. Contrast ODS, where GET
/api/features/{id}/enable (features.py:188-250) returns real numbered steps
with deep links, and FeatureDiscovery.jsx:246-300 renders them as an actual
modal with clickable "Open Chat"/"Open n8n" buttons. This is precisely the
shape of issue #1845 (a printed remediation, hal0 slot migrate-flags
--apply, that fails because the panel omitted --stop-services even though the
error message itself named the fix) — a self-explanation gap in the exact same
family, not a one-off typo.
2. Long silent waits with no progress line (issue #1870). slot load,
unload, restart, swap, and update --restart-slots can legitimately block
10s to 2.7 hours with nothing printed — no spinner, no elapsed counter, no
state line — because the fix for a prior regression (#1832) raised the timeout
ceiling without adding the promised progress polling. ODS's closest analogue,
extension installs, streams a live progress bar and label
(Extensions.jsx:106-124 pattern; poll loop :111-160) and degrades gracefully
with a "polling lost" banner rather than silence.
3. A remediation that cannot work as printed (issue #1845). The hal0
update "Convergence incomplete" panel tells the operator to "Stop hal0, then
run: hal0 slot migrate-flags --apply" — but that literal command fails,
because hal0-slot@* units are separate from hal0.target/hal0-api, and the
actually-sufficient invocation (--apply --stop-services) is only visible in
the error the failed command itself prints. ODS's install-menu logic shows the
alternative discipline: when it auto-disables a feature, it immediately states
the real command to reverse that (ui.sh:547-549, "You can enable it later
with: ods enable comfyui") in the same breath as the decision, not two failed
attempts later.
4. Notification infrastructure exists but isn't wired to the moment that
matters (issue #1994). globals-install.ts:26,39-40 installs a working toast
system and hooks a "queue" from a zustand store; notifications.jsx already
unifies bell + "Needs attention" card. Yet an agent-requested memory delete
landing in the approval queue is silent — "the badge is poll-driven and
silent," per the issue, filed by the team's own reviewer during the memory-v2
build. ODS's parallel case (a background SDXL/model download finishing) is
explicitly narrated at the next natural checkpoint (13-summary.sh:170-183:
"SDXL Lightning model download still in progress" / "…completed").
5. Same-screen contradicting numbers (issues #1928, #1929). The memory
ruler's header (telemetry-header.jsx:364, total - modelUsedGb) and its bar
segment (:389-390, reducing to total - gttUsedGb) compute "free" from two
different bases, so on a box where host GTT usage diverges from the per-slot
model sum, the same screen shows two different "free" numbers (measured ~16 GB
apart in the issue). A second, related bug: memory-map.jsx:141-142 reads dead
fallback keys (stats.gtt_total_mb, rawHw.unified_memory_mb) that don't exist
on the normalized shapes, so the pool can silently become "system RAM" while
still labeled "GPU pool (GTT)." ODS's dashboard computes a comparable figure
(sidebar memory bar, Sidebar.jsx:76-89) from a single source per render and
explicitly branches on memoryType === 'unified' rather than blending two
derivations.
6. Help text explains internals, not consequences. hal0's FieldInfoIcon
tooltips are widespread (20 files) and the popover mechanics are solid, but the
copy is written for someone who already knows the codebase: "/api/health/system
· per-dependency status", "/api/stats/power · sysfs hwmon, sensor-dependent",
"hal0.toml [memory] · requires restart to switch"
(settings/pages/general/OverviewPage.jsx:59-231, settings/pages/data/
MemoryPage.jsx:69,161). None say what changing the setting does for you or
what happens if you don't. ODS's equivalent — STATUS_DESCRIPTIONS
(Extensions.jsx:73-84) and the saveHint/restartHint strings
(main.py:921-922) — run the other way: consequence first, mechanism only if
needed ("Saving writes the .env file directly, keeps existing secret values when
left blank… stores a timestamped backup"). This is a register choice consistent
with each product's stated audience, but it means hal0 currently explains itself
to an engineer, not a first-time user — worth naming explicitly if hal0 wants a
broader audience.
7. An unsurfaced security tradeoff (issue #1822). HAL0_BIND_HOST (LAN) and
require_auth (off by default) are decided independently and never compared out
loud at the moment that matters — install completion or hal0 doctor all. The
posture is intentional and is documented (docs/getting-started/index.mdx:
109-114, a caution Aside), so this is "disclosed but not surfaced at the
decision point," a narrower gap than "hidden." ODS states comparable
consequences inline at the moment of the choice (e.g., ui.sh:547-549 above),
which is the standard #1822 is asking hal0 to meet.
8. tool_model has no dashboard path (issue #2108).
AgentsBrainPage.jsx persists enabled/read_only/model/max_rounds/
completion_timeout_s but never tool_model, even though it's the one
[brain_chat] key with real routing consequences — it's hal0.toml-only. ODS
keeps this kind of thing symmetric: anything a manifest or schema declares
(.env.schema.json) is automatically surfaced as a labeled, described row in
Settings via _build_env_fields (settings.py:194-241) — there's no
config-file-only escape hatch for a setting with live behavior.
9. Open WebUI is only partly pre-wired. src/hal0/openwebui/
env_writer.py:106-131 sets chat routing and voice (STT/TTS) into Open WebUI,
but never sets RAG_EMBEDDING_ENGINE/RAG_OPENAI_API_BASE_URL,
ENABLE_WEB_SEARCH/SEARXNG_QUERY_URL, or ENABLE_IMAGE_GENERATION/
COMFYUI_BASE_URL — confirmed absent by a repo-wide grep. hal0 has an
embeddings/rerank surface (/v1/embeddings, /v1/rerank) and an image
pipeline (routes/comfyui.py, comfyui-pane.jsx), but neither is reachable
through Open WebUI's own document-upload or image-gen buttons — a user who
opens the "familiar" chat UI and clicks "generate image" or uploads a PDF gets
Open WebUI's un-pointed defaults, not hal0. ODS wires all of this, including a
full baked-in ComfyUI node-graph JSON (docker-compose.base.yml:127-146) so
Open WebUI's native image button works unmodified.
Where hal0 is genuinely ahead. (a) RestartApiPanel.jsx is a complete,
correctly-sequenced repair loop — confirm → mutate → poll /api/health → toast
success or a specific failure message pointing at journalctl -u hal0-api —
better than any single ODS remediation flow reviewed here. (b) ApplyBadge.jsx
gives per-field reload-class information (live/needs-service-restart/needs-
manual-restart) with the affected service named in a tooltip, which is more
precise than ODS's page-level "Apply changes will recreate: X, Y" banner. (c)
One shared classifier (health_report.py) drives both the CLI and the API,
structurally preventing the two from disagreeing — ODS's status text is
duplicated per-router (SERVICE_DESCRIPTIONS in the frontend, message=
strings in each Python router) with no single source of truth. (d) The
agents-as-trading-cards treatment (agents-overview.jsx) is a distinctive,
memorable identity choice ODS has nothing like — a legitimate strength if
delight is a goal, at some cost to a brand-new user's ability to tell what an
"ability" with "cost 2" and "pow 90" actually does.
D. Port candidates
| From ODS | hal0 target | Size | Risk |
|---|---|---|---|
next_steps population pattern: features.py:130-144 (human suggestion sentences) + :188-250 (numbered steps/links) |
Populate health_report.to_diagnosis() (src/hal0/health_report.py:246-283) with real NextSteps for the 7 existing checks (reuse the commands already known to cli/doctor_commands.py); make DiagnosisPanel.jsx's command chips actually copy-to-clipboard or run |
S–M | Low — additive; scaffolding (Diagnosis, NextStep, the panel) already exists and is tested |
FeatureDiscovery.jsx:246-300 modal (numbered steps + deep links, rendered from a live API response) |
Reuse for next_steps once populated above, or for slot-load failures surfaced by #1845/#1870 |
S | Low |
Extensions.jsx:60-84 STATUS_STYLES/STATUS_DESCRIPTIONS (status word → one sentence) |
Add a parallel plain-language map alongside slot-status.js's tooltips — keep the precise phase vocabulary, add a "what this means" clause for crashed/stopped/pulling |
S | Low |
installers/lib/ui.sh:547-549 "auto-disabled X — here's the command to re-enable" pattern |
Apply anywhere hal0 auto-derives a default with a workaround (e.g. tool_model #2108, ComfyUI-gated low-VRAM boxes) |
S | Low |
docker-compose.base.yml:82-175 full Open WebUI env wiring (RAG/websearch/image-gen) |
src/hal0/openwebui/env_writer.py — conditionally add RAG_EMBEDDING_*/ENABLE_IMAGE_GENERATION/COMFYUI_* only when the relevant hal0 slot is actually bound, mirroring ODS's ENABLE_COMFYUI:-false} gating discipline (docker-compose.base.yml:126-127). Correction (2026-10-04): a one-shot gate drifts — openwebui.env is written at install and the writer's preserve_existing merge keeps every existing key on upgrade (env_writer.py:251-267), so a flag set true stays true after the slot is unbound and nothing re-renders when one is bound later. The generated keys need their own owned block, re-rendered (with an Open WebUI restart) from the slot-change hook report 02 §D.5 calls depends_on_slots — or point Open WebUI at stable hal0-api routes that answer with a clear error when no slot is bound, so no env rewrite is needed |
M | Medium — must gate on live slot state or it repeats ODS's own Apple-VRAM-fallback footgun (features.py:27-32) of claiming a capability that isn't really there |
README.md:124-139 "At A Glance" + "If you know X, ODS adds Y" tables |
hal0 README.md, near the top, framed against Ollama/llama.cpp/Open WebUI/LocalAI |
XS | None (copy only) |
docs/POST-INSTALL-CHECKLIST.md 6-step "did it work" script |
docs/getting-started/first-chat.mdx or a new verify.mdx |
XS–S | None |
TroubleshootingAssistant.jsx Symptoms → Cause → Solutions with copy buttons |
A CLI-doctor-backed panel using the already-existing NextStep data once wired (overlaps row 1) |
— | — |
E. Do-not-copy
- The CRT typing-effect boot theatrics (
ui.shtype_line/type_line_dramatic, a real per-charactersleep). ODS itself gates this off when$INTERACTIVE != "true"— it knows it's for a human's first, foreground, one-time run. hal0's installer is used repeatedly (upgrades, Proxmox CT scripting, fleet installs); adding artificial per-character delay anywhere in that path would actively hurt the audience hal0 has. - The LORE_MESSAGES brand-voice copy verbatim (
ui.sh:154-168). The pattern of a consistent narrator voice is worth having; the specific ideological copy ("No terms of service. No content policy. Just freedom.") is ODS's brand, not a UX mechanism, and shouldn't be imported as text. - ODS's flat, 60-file, ALL-CAPS
docs/directory. hal0's Starlight getting-started/concepts/guides/reference split is already better information architecture (less duplication, clearer entry points); don't flatten it to match ODS. - Client-side status re-derivation (
Dashboard.jsx's ~250 lines ofpickFeatureLink/findHealthyService/normalizeFeatureStatushelpers). hal0 already has the better pattern — one server-side classifier (health_report.py) shared by CLI and API — and should keep extending that, not add a second, client-side resolution layer like ODS's. - The Setup/Owner magic-link invite model (
Invites.jsx). This solves a multi-tenant "who's allowed in this household appliance" problem hal0's single-operator, trusted-LAN posture doesn't have today. Not applicable unless hal0 adds multi-user access as a feature in its own right.
F. Owner decisions
- First-run wizard, or stronger post-install confirmation? hal0's
"installer is the whole setup" stance (
README.md:39-45) is explicit policy, not an oversight — the question is narrower than "add a wizard": does the end ofinstall.shneed an ODS-style summary (what's running, what isn't, exactly what to run next) rather than relying on the operator to findhal0 statusthemselves? - Wire
next_stepsend-to-end for/api/doctor. Low-risk, scoped, and the scaffolding already exists on both the CLI (doctor_commands.py) and UI (DiagnosisPanel.jsx) sides — this is the single highest-leverage fix in this review. - Pre-wire Open WebUI's RAG/search/image-gen, or keep them hal0-native-only?
If the dashboard's own comfyui-pane and
/v1/embeddingsare meant to be the only front door, that's a legitimate "one UI, not two" decision — but it should be a decision, not an omission, and should be stated somewhere a user would find it before they click Open WebUI's native "generate image" button and get nothing. - Who is the copy for? The current technical register (API paths and config keys in tooltips) is coherent for hal0's stated homelab-operator audience. Moving any of it toward ODS's consequence-first, jargon-free register is a deliberate audience decision, not a bug fix — worth scoping explicitly rather than drifting page-by-page.
- Give cross-cutting notification gaps (toast-on-approval, #1994) an owner. The issue itself flags this as "cross-cutting… not memory-specific" — worth deciding whether it's a tracked initiative (audit every SSE/poll surface for a matching toast) rather than a one-off patch.
- #1822's LAN-bind warning: disclose more loudly, without re-creating KB-1's
scar. The doc Aside already exists; the ask is a non-enforcing,
non-blocking line in the install summary and a
hal0 doctor allrow. Given the prior incident (auto-enforcement made things worse), any implementation should be reviewed explicitly against the O19 lesson cited in the issue before shipping.
08-docker-cli-amd.md Container runtime, CLI, AMD tuning
ODS vs hal0 — Container Runtime, Config Contracts, CLI, and AMD/Strix Halo Tuning
Scope: container runtime (Docker/Podman), compose/config contracts, the operator
CLI, and AMD Ryzen AI Max ("Strix Halo") host tuning. All citations are
file:line against the two repos on disk: ODS at /home/user/ods/ods/ and
hal0 at /home/user/hal0/.
A. ODS mechanism
A.1 Compose layering & contracts
ODS assembles its runtime from Docker Compose file merging, documented as a
formal contract rather than convention. docker-compose.base.yml (475 lines)
defines the always-on core: llama-server as a "GPU-agnostic stub"
(docker-compose.base.yml:24-25, image defaults to
ghcr.io/ggml-org/llama.cpp:server-b9014), open-webui
(:82-186, pinned ghcr.io/open-webui/open-webui:v0.7.2), dashboard-api,
model-router, two remote-provider-egress services, and dashboard. Every
service in the base file carries a deploy.resources.limits/reservations
block (e.g. :173-180, :281-288), a healthcheck (7 of them in the base
file), restart: unless-stopped (24 occurrences across base + extensions),
and security_opt: [no-new-privileges:true]. dashboard explicitly gates on
dashboard-api's health (depends_on.dashboard-api.condition:
service_healthy, :463-465).
Hardware overlays (docker-compose.{nvidia,amd,apple,arc,intel,cpu}.yml) only
override llama-server, dashboard-api, and open-webui — never redefine
the whole stack. docker-compose.amd.yml rebuilds llama-server from a
Dockerfile.amd into image ods-lemonade-server:latest (:6-17), wraps it
with a custom entrypoint that syncs LEMONADE_CTX_SIZE into Lemonade's
cached config.json on every start (:18-24, because Lemonade "only reads
the env on first-start"), and passes /dev/dri + /dev/kfd plus
VIDEO_GID/RENDER_GID group_add (:48-53). Mode overlays
(docker-compose.{cloud,external-llm,lemonade-external,tier0}.yml) use
Compose profiles: [local-inference] plus restart: "no" to stub out
llama-server without deleting the service reference other files'
healthchecks depend on (docker-compose.cloud.yml:8-16) — the "dependency
placeholder" pattern docs/COMPOSE_RESOLVER_CONTRACTS.md:75-87 calls out as
first-class. Multi-GPU overlays (docker-compose.multigpu-{amd,nvidia}.yml)
layer again on top for ROCR_VISIBLE_DEVICES/tensor-split flags.
docs/BACKEND-CONTRACT.md:1-39 and config/backends/{amd,nvidia,cpu,apple}.json
define one JSON contract per backend (LLM engine, public port/health URL,
provider name/URL — e.g. config/backends/amd.json:1-18 pins
ghcr.io/lemonade-sdk/lemonade-server:v10.2.0 and records linux_backend:
rocm vs windows_backend: vulkan), loaded by
scripts/load-backend-contract.sh and consumed in
installers/lib/detection.sh. config/ports.json is the single canonical
port map (env_var, external_default, internal_port per service).
Config-contract enforcement: .env.example (592 lines) documents every
variable; .env.schema.json (1522 lines, JSON Schema draft 2020-12) requires
six secrets (WEBUI_SECRET, SEARXNG_SECRET, N8N_USER, N8N_PASS,
LITELLM_KEY, OPENCLAW_TOKEN) and types/enums/ranges for the rest.
scripts/validate-env.sh (611 lines) is a from-scratch Bash .env parser
(never sources the file, "for security reasons",
scripts/validate-env.sh:90-92) that checks required keys, unknown keys,
type/enum/range/minLength, and hand-written cross-field "runtime contracts" —
e.g. remote-LLM transport requires HTTPS and rejects loopback/link-local hosts
(:450-461), EMBEDDING_MODEL must be a HF TEI repo id not a GGUF
(:475-485). .github/workflows/validate-env.yml runs this as a 5-way matrix
(tiers 0-4), each generating a synthetic .env via a stubbed phase-06
environment before validating (/home/user/ods/.github/workflows/validate-env.yml:13-118).
lib/safe-env.sh is the shared safe loader (load_env_file,
load_env_from_output, load_env_from_output_allowlist) — explicitly "no
eval, no word-splitting" (lib/safe-env.sh:1-9), and it special-cases UID
because Bash's own readonly UID would otherwise abort every lifecycle
command under set -e (:33-36). lib/dotenv-quote.sh:1-25 serializes a
value into the one-line .env grammar shared by Bash/Compose. lib/rootless-
ownership.sh (250+ lines) detects Docker-rootless vs Podman-rootless via two
different docker info fields (.SecurityOptions vs
.Host.Security.Rootless, :8-29) and repairs per-service bind-mount
ownership with a disposable busybox:1.36.1 helper container rather than
chowning from the host (:4, :130-150). ODS_UID/ODS_GID
(.env.example:487-491) are read by three extension composes directly
(extensions/services/hermes/compose.yaml:18-19, n8n/compose.yaml:6,
privacy-shield/compose.yaml:10) and by that repair script's per-service
UID/GID table (lib/rootless-ownership.sh:220-260).
Image tag pinning is a deliberate, written trade-off: docs/ADR-IMAGE-TAG-
PINNING.md accepts (2026-03-04) that OpenClaw, SearXNG, and Whisper stay on
:latest because SearXNG has no stable release channel and pinning risks
silently missing patches on an optional, already-hardened service — but
llama-server, open-webui, and the AMD build args are all pinned to
specific tags. docs/KNOWN-GOOD-VERSIONS.md records tested Docker/Compose/
driver baselines per platform. docs/DOCKER-DESKTOP-OPTIMIZATION.md is a
Windows/WSL2-specific tuning guide (CPU/memory sliders, disk-image
relocation to NVMe) — there is no equivalent for native Linux or AMD.
A.2 Podman
ODS does not run on Podman as a first-class, documented target — there is no
Podman row in docs/PLATFORM-TRUTH-TABLE.md or docs/SUPPORT-MATRIX.md.
What exists is defensive handling for the case where the docker binary
itself is the Podman Docker-compatibility shim, called out as "common on
AMD Ryzen AI / Fedora / RHEL hosts" (installers/phases/05-docker.sh:390-401).
_runtime_is_podman() greps docker version for "podman" (:398-401); when
true, installers/lib/podman-registries.sh:4-234 rewrites a user-level
~/.config/containers/registries.conf.d/99-ods-dockerhub.conf so bare image
names resolve against Docker Hub, "because podman has no implicit docker.io
like Docker does" (:390-393). This is careful, tested code (idempotent TOML
editing in Python, tests/test-podman-rootless-contracts.sh) but a
compatibility patch, not a native path: ODS still emits and resolves Docker
Compose YAML, never podman-compose or Podman Quadlets. Rootless is handled
generically for whichever runtime answers "rootless": true"
(lib/rootless-ownership.sh:8-29), and Docker Desktop is its own branch
(docs/DOCKER-DESKTOP-OPTIMIZATION.md). Net: ODS treats "docker" as the
contract and tolerates Podman when it wears a Docker mask; it is not
validated against native Podman networking, Quadlets, or podman machine.
A.3 AMD / Strix Halo host tuning
installers/phases/10-amd-tuning.sh (304 lines) is the concrete, host-level
tuning ODS applies only when GPU_BACKEND=amd:
- Group membership & device presence — adds the installing user to
render/video(:34-38), modprobesamdkfdif/dev/kfdis missing (:41-51), and verifies/dev/dri/renderD128exists (:53-61). - sysctl — installs
config/system-tuning/99-ods.conf(vm.swappiness=10,vm.vfs_cache_pressure=50) to/etc/sysctl.d/(:102-111). - amdgpu modprobe options — installs
config/system-tuning/amdgpu.conf(options amdgpu ppfeaturemask=0xffffffffandgpu_recovery=1) (:113-121). - GTT/TTM sizing, scaled by RAM — computes a GTT percentage of total RAM
(90% at ≥96 GB, 80% at ≥64 GB, else 65%;
:140-152) and writes/etc/modprobe.d/amdgpu_llm_optimized.confwithoptions amdgpu gttsize=<MB>,options ttm pages_limit=<pages>,options ttm page_pool_size=<pages/2>(:163-170), then rebuilds initramfs (:172-174). A 128 GB Strix Halo box lands at ~115 GB GTT. - GRUB kernel cmdline — single-GPU APUs get
amd_iommu=offappended for "~6% memory bandwidth improvement" (:186-229); multi-GPU boxes instead requireiommu=ptfor device passthrough and are only warned, not auto-edited (:192-201). - tuned profile — installs
tunedif absent and setsaccelerator-performancefor "5-8% pp improvement" (:245-284). - A BIOS banner recommends setting UMA Frame Buffer Size to 512 MB minimum
(
:123-136), and a reboot-required banner fires whenever GRUB or modprobe files changed (:289-302).
installers/lib/amd-topo.sh provides vendor-neutral multi-GPU topology
detection (amd_gpu_id, amd_gfx_version, detect_amd_topo), with a
three-tier fallback chain per fact (rocm-smi → amd-smi → sysfs
ip_discovery → rocminfo). docs/LEMONADE-SDK-COMPAT.md documents a
managed-vs-external Lemonade split (own container vs. wrap a pre-existing
host Lemonade via --use-existing-lemonade, with host.docker.internal
networking notes and a runtimeMode/managedByODS diagnostic surface).
docs/AMD-ODS2-GAMMA-BRIEF.md is a product brief for a Strix-Halo
"drop-ship appliance" flow — engineering-adjacent, not a runtime mechanism.
The Docker 29.x bug workaround (installers/phases/05-docker.sh:194-244):
Docker 29.3.x regresses AMD /dev/dri custom-device passthrough; ODS
detects the server version and, on AMD, auto-downgrades to 29.2.1 via the OS
package manager, restarting the daemon afterward (:240-243).
hal0 has none of this tuning. A repo-wide grep for gttsize,
ttm.pages_limit/page_pool_size, amd_iommu, GRUB_CMDLINE, tuned-adm/
accelerator-performance, and vm.swappiness across installer/,
packaging/, src/hal0/hardware, and src/hal0/install returns zero
matches. What hal0 does have is device-presence diagnosis, not host
tuning: installer/lib/preflight.sh (~950-1050) runs rich /dev/dri /
/dev/kfd / /dev/accel gid-mapping checks with named error codes
(HAL0_GPU_RC_NO_KFD, HAL0_GPU_RC_KFD_GID, etc., tailored to LXC device
passthrough where a render node's gid can map to an unrelated host group like
clock), and src/hal0/install/gpu_perms.py:1-56 +
installer/systemd/hal0-gpu-perms.service:1-29 re-converge /dev/kfd's
group once per boot (since udev recreates the node fresh every boot) —
which is a materially more robust mechanism than ODS's one-shot,
install-time-only usermod -aG render,video (10-amd-tuning.sh:34-38, which
also requires a fresh login shell to take effect and is never re-checked
after that). src/hal0/providers/_gpu.py:70-71 hard-codes a last-resort GID
fallback table explicitly labelled "Strix Halo LXC values" (render: 993,
video: 44). HSA_OVERRIDE_GFX_VERSION exists in hal0 only as a manual,
per-slot [server].env escape hatch documented in
src/hal0/config/schema.py:298-306 — nothing auto-detects gfx1151 and sets
11.5.1 the way ODS's phase 06 does (referenced from .env.example:477).
ROCBLAS_USE_HIPBLASLT, sysctl swappiness, GTT/TTM sizing, GRUB
amd_iommu=off, and tuned-adm accelerator-performance are entirely absent.
This gap is corroborated by hal0's own upstream image source:
/home/user/amd-strix-halo-toolboxes/README.md:179-198 (the fork hal0's ROCm
runner images are built from) explicitly documents
amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856 as required
GRUB parameters for Strix Halo, citing a 5-12% performance delta — almost the
exact tuning ODS automates — yet nothing in hal0's own installer applies it.
Correction (2026-10-04) — the IOMMU line is not part of the gap. hal0's own
Strix Halo guidance is iommu=pt amd_iommu=on
(docs/getting-started/drivers.mdx:100-112), and ADR-0003 records why: the
community amd_iommu=off tweak silently removes /dev/accel, the XDNA NPU
device, so hal0 deliberately recommends it nowhere
(docs/adr/0003-onnx-text-generation-npu.md:96-99). The toolbox README's 5-12%
and ODS's "~6%" are both that IOMMU-off delta. The gap hal0 should close is
GTT/TTM sizing, the sysctls and the tuned profile; the IOMMU parameter stays as
hal0 documents it. The GTT/TTM half is also narrower than ODS's default-on: on
kernels ≥ 6.14 amdgpu sizes the pool dynamically and hal0 reads the live value
per request (drivers.mdx:86-97), so fixed sizing is an opt-in floor (see D.1).
A.4 CLI
ods-cli is a single Bash file, 6,595 lines / 269,480 bytes
(ods-cli:1-6595, confirmed via wc -l/ls -la), dispatched from one
case "${1:-help}" at the bottom (ods-cli:6562-6595). The verb tree (with
aliases) is:
gpu|g status topology assignment validate reassign
status|s (table); status-json (separate top-level cmd, not `status --json`)
list|ls enable/disable-aware service listing
enable/disable/purge extension lifecycle
preset|p save|load|list|delete|export|import (.env + extension state + meta.txt)
mode|m local|cloud|hybrid (rewrites LLM_API_URL/ODS_MODE)
model current|list|swap
remote-provider list|status|plan
stt current|status|download
backup/restore/rollback
logs|log|l <service> [lines] (hardcodes `-f`/follow, no way to opt out)
restart|r / start / stop (special-cases "hermes" and "llama-server" inline)
update|u, shell|sh, config|cfg (show|edit|validate), chat|c, benchmark|bench|b
doctor|diag|d [--json]
audit, template|tmpl, agent (controls the ODS **host agent** daemon, not Hermes)
32 cmd_* functions total (grep count). cmd_config show
(ods-cli:2260-2317) masks secrets by schema lookup before printing .env,
and cmd_config validate shells out to the same scripts/validate-env.sh
used in CI (:2289-2312). cmd_status_json (:1402-1517) builds one JSON
document via jq -n/jq -s and per-service HTTP health probes — but it is
its own top-level command (ods status-json), not ods status --json;
ods doctor --json uses the opposite convention
(completions/ods-cli.bash:44-46), so JSON output is inconsistent across the
CLI's own surface. cmd_preset snapshots .env + per-extension state + a
meta.txt into presets/<name>/ and validates compatibility before load.
Notably, "hermes" is not a subcommand tree — it is a magic string
special-cased inside the generic cmd_restart/cmd_stop bodies
(ods-cli:1583, :1787); cmd_agent (:4756-4900+) is unrelated — it
controls ods-host-agent.py, a local Wi-Fi/setup helper daemon.
ODS's own docs/ODS_CLI_DECOMPOSITION.md acknowledges the monolith as a
liability — "the file is large enough that future changes need a staged
decomposition plan" (:1-8) — with a six-stage extraction order (read-only
helpers → status/logs → diagnostics → model/mode → extensions →
lifecycle/backup last, :26-92). The plan is written but unexecuted; ods-cli
remains one file. docs/MODE-SWITCH.md documents the one-env-var
(LLM_API_URL) mode design cleanly. docs/PROFILES.md is misleadingly named:
it documents that Docker Compose profiles (not CLI presets) were removed —
"removed in favor of the current all-core architecture for simplicity"
(:39-41) — so all 16 core services always start today, and disabling one
means hand-editing a docker-compose.override.yml. completions/ods-cli.bash
is a single static Bash completion file mirroring the verb/alias list by hand.
A.5 Runtime images
ODS builds llama-server per backend: NVIDIA and CPU pull upstream
ghcr.io/ggml-org/llama.cpp build tags directly
(docker-compose.nvidia.yml:8, docker-compose.cpu.yml:8); AMD builds a
custom image (ods-lemonade-server:latest) from Dockerfile.amd with build
args AMDGPU_TARGET/LLAMA_CPP_REF/LEMONADE_SERVER_IMAGE
(docker-compose.amd.yml:10-17); Intel Arc builds SYCL from source
(docker-compose.arc.yml:29-33). None of these are digest-pinned — every
image reference in every compose file is a tag (:server-b9014, :v0.7.2,
:v10.2.0), never @sha256:....
B. hal0 today (verified)
hal0 has no Docker Compose file anywhere in the repository (confirmed:
find /home/user/hal0 -maxdepth 1 -iname "docker-compose*" is empty, and
packaging/ holds only systemd/toolbox/runner/proxmox/sudoers). The entire
runtime is Podman, orchestrated by systemd, one container per slot under
hal0-slot@<name>.service (ARCHITECTURE.md:27-51). ContainerProvider
(src/hal0/providers/container.py:1-40) is "the sole slot-lifecycle
backend" for every provider type — GPU llama-server, FLM NPU, Kokoro/Qwen3-TTS,
and ComfyUI all render through one function,
_render_quadlet_from_plan (:803-990+), which emits a Podman Quadlet
.container unit rather than a hand-built podman run string: typed
Image=/AddDevice=/Volume=/PublishPort=/Environment= keys, with
GroupAdd=/SecurityOpt= deliberately routed through PodmanArgs= instead
of native Quadlet keys because "4.x generators HARD-FAIL the conversion on
those keys" on real hardware (:899-909, citing live evidence from two named
test boxes, halo150/halo143). Restart policy is systemd-native and far more
expressive than Compose's restart: unless-stopped: Restart=always,
RestartSec=5, RestartSteps=6, RestartMaxDelaySec=300 (a 5s→10→20→39→
77→152→300s geometric ramp), capped by StartLimitIntervalSec=1800/
StartLimitBurst=10 at the [Unit] level (:854-891) — and the code
comments record a real regression (#1424) from an earlier, too-tight
300s/5-burst pairing that "LATCHED the unit in failed" for hours. There is,
however, no resource ceiling: a repo-wide grep for MemoryMax=,
CPUQuota=, or any memory/CPU limit key in container.py, base.py, or
config/schema.py returns nothing — every slot container runs unconstrained
on host resources, unlike every ODS Compose service's deploy.resources.limits.
Podman images are always fully-qualified (ghcr.io/hal0ai/...), so hal0 never
hits the "podman has no implicit docker.io" short-name problem ODS works
around in podman-registries.sh. Every runner image is pinned by both
tag and sha256 digest in manifest.json's toolbox_images block, each with
a dated provenance note (e.g. the cpu entry records the exact
llama-server --version output used to verify the pinned digest).
/home/user/hal0-runner-images is the dedicated aggregator repo: "owned"
Dockerfiles (cpu/flm/kokoro/moonshine/comfyui) plus "referenced" images
pinned by digest but built upstream (vulkan/rocm →
Hal0ai/amd-strix-halo-toolboxes, a hal0-maintained fork of
kyuz0/amd-strix-halo-toolboxes adding *-server entrypoint variants for
systemd) — CI resolves published ghcr digests and opens an automatic
manifest-bump PR against hal0 (hal0-runner-images/README.md:8-21). The
OpenWebUI companion is likewise a hand-rendered, digest-pinned systemd/Podman
unit (packaging/systemd/hal0-openwebui.service:34,68, same Restart=always
backoff pattern, ExecStartPre=-podman pull ...@sha256:...).
Config validation runs through Pydantic v2 models
(src/hal0/config/schema.py:1-20): "All TOML files under /etc/hal0/ are
validated against these models at startup. Typos like backend = "vukan"
raise a ValidationError with the field path." This is a stronger mechanism
than ODS's bash/jq validator (native typed enums/unions, validated on every
process start rather than only when an operator remembers to run ods config
validate), though hal0 has no equivalent of ODS's CI matrix that
materializes and validates a synthetic config per hardware tier.
hal0's CLI (src/hal0/cli/) is Typer-based and modular: 35 files, 18,245
lines total, the largest single file (doctor_commands.py, 2,272 lines)
still well under half of ods-cli's size. main.py:61-127 builds one root
Typer app and mounts ~17 sub-apps via app.add_typer(...) — slot,
model, memory, config, doctor, upstream, capabilities, comfyui,
agent, app, migrate, registry, runner-images, profile, mcp,
auth, board — plus flat commands (bench, chat, system-info,
ports, status, update, serve, uninstall). Every mount point carries
an inline comment naming the GitHub issue that motivated it (e.g. :88-91
for runner-images), and a deprecated command (probe, :234-244) is kept
as a hidden, redirecting shim rather than silently removed. The CLI calls the
same /api/* HTTP surface the dashboard uses (status() at :198-230 hits
/api/status, /api/slots, /api/upstreams) rather than shelling out to
systemctl/podman directly — one behavior surface for both consumers.
Two concrete, source-confirmed CLI defects match the owner's characterization:
- Long-blocking lifecycle verbs with no progress feedback.
src/hal0/slot_lifecycle_budget.py:1-192(written in response to #1832, a prior timeout mismatch) derives client-side HTTP timeouts from the server's own worst case:HEALTH_TIMEOUT_S=180,TERMINATE_TIMEOUT_S=30,EVICTION_UNLOAD_ALLOWANCE=3, plus an output-sanity gate up to 360s on a CPU-backed slot (:29-136). The derived timeouts —SLOT_LOAD_TIMEOUT_S≈ 1587s,SLOT_UNLOAD_TIMEOUT_S≈ 828s,SLOT_LIFECYCLE_TIMEOUT_S(restart/swap) ≈ 2415s, viaslot_lifecycle_timeout_s()(:165-192) — are consumed directly byslot_load/slot_unload/slot_restartinsrc/hal0/cli/slot_commands.py:311-362, each one blockingapi_post(...)call: no spinner, no incremental state (the server does transitionstarting→warming→readyinternally perARCHITECTURE.md:231-266, but the CLI never polls or streams it) — just the final state once the HTTP response returns or the timeout fires. The module itself flags the remaining gap as open: "bounding it server-side is the real fix (#1869)" (:107). This is the mechanism behind a lifecycle verb sitting silent for minutes; the owner-cited 966s figure falls inside this documented envelope. hal0 updatedoes not refresh/usr/local/bin/hal0(#1844).installer/install.sh:1074-1099creates the PATH symlink exactly once, at install time. The self-update path (src/hal0/updater/updater.py,activate_release()) atomically swaps/usr/lib/hal0/currentandpip install --force-reinstalls into the same venv (_reinstall_into_venv,:2338-2341: "apply() swaps the current symlink but the venv imports hal0 from its own site-packages... until the code is reinstalled") — but nothing re-runs theln -sfnstep, so a box whose wrapper predates a venv/path change keeps resolving a stale binary. This caused a real, separately-numbered regression (#2092, fixed) wherehal0 agent reprovision's privilege-drop re-exec resolved viashutil.which("hal0")and silently re-exec'd an old release as root (src/hal0/cli/agent_commands.py:610-625). CHANGELOG documents the operator-facing symptom and workaround across three releases: "check withhal0 --versionand re-run the installer if it disagrees" (CHANGELOG.md:2299-2306, echoed at:1085-1088,:2397-2399).
hal0 has no WSL2 path at all — WSL appears only in host-classification code
(src/hal0/hardware/probe.py:644,664) used to label a detected Microsoft
kernel, never in an install flow or doc.
C. Better / worse / equivalent
ODS is clearly better at: (1) documented AMD/Strix Halo host tuning —
sysctl, GTT/TTM sizing, GRUB cmdline, tuned profile — all automated,
RAM-scaled, and independently corroborated by the same upstream toolbox
project hal0 forks from; (2) breadth of documented, tested platform
combinations (NVIDIA, AMD, Apple, Intel Arc, CPU, WSL2, Docker Desktop)
against a named compatibility contract; (3) a written, CI-enforced
env-schema contract with cross-service checks run as a 5-tier matrix on
every PR; (4) a real Docker-Engine-version regression workaround (29.3.x
/dev/dri) reflecting production field experience.
hal0 is clearly better at: (1) the runtime mechanism itself — Podman
Quadlets + native systemd restart/backoff/rate-limiting is a cleaner, more
debuggable primitive than Compose's coarser restart: unless-stopped, and
needs no daemon; (2) boot-persistent GPU device-node convergence
(hal0-gpu-perms.service) versus ODS's one-shot, login-shell-dependent
usermod; (3) supply-chain pinning — every hal0 runner/companion image
is digest-pinned with provenance notes, whereas ODS pins by tag only and has
a written ADR accepting :latest for three services; (4) config
validation as a load-time, typed contract (Pydantic) rather than an
optional, separately-invoked script; (5) CLI architecture — a modular,
issue-annotated Typer tree is a healthier long-term surface than one 270 KB
Bash file, even though ods-cli is more feature-complete in places (masked
config show, preset export/import).
Roughly equivalent: rootless-ownership handling (ODS's busybox-helper
chown dance vs. hal0's boot-converge service — both exist because container
runtimes and host device nodes disagree on ownership); mode/provider routing
(ods mode {local,cloud,hybrid} vs. hal0's upstream/dispatcher model).
Podman support specifically is not equivalent: ODS runs on Podman only incidentally and never documents it as supported; hal0 is Podman-native by design. ODS's edge is specifically the tuning and platform-breadth axes named above — its actual container-runtime mechanism (Compose + a real Docker Engine dependency, needing its own version-regression workaround) is the less robust of the two once Podman/rootless edge cases are in play.
D. Port candidates
-
AMD/Strix Halo host tuning phase (highest value, most direct port). Source:
installers/phases/10-amd-tuning.sh(304 lines, logic reusable near-verbatim, e.g. the GTT sizing at:146-157—gtt_pct90/80/65 by RAM tier,pages_limit = gtt_size_MB*1024*1024/4096,page_pool_size = pages_limit/2). Target: a newinstaller/lib/amd- tuning.shsourced once frominstaller/install.sh(hal0's installer is one 4,002-line script with no phase directory, so this is a function block called after GPU detection, not a new phase file). Size: ~150-180 lines of new Bash. Risk: medium — edits/etc/default/gruband/etc/modprobe.d/, needs a reboot, and should follow hal0's existing "never fail the boot" posture (gpu_perms.py:1-27,hal0-gpu-perms.service'sSuccessExitStatus=0 1). Low-controversy on the merits: hal0's own upstream image source already publishes theamdgpu.gttsize=... ttm.pages_limit=...half of this recommendation (amd-strix-halo-toolboxes/README.md:186); open questions are packaging (default-on vs. flag) and reboot messaging. Scope (2026-10-04) — GTT/TTM, sysctl and tuned only. The port must not write ODS'samd_iommu=off(10-amd-tuning.sh:186-229): it removes/dev/accel(the XDNA NPU) and contradicts hal0's documentediommu=pt amd_iommu=on(drivers.mdx:102, ADR-0003:96-99). If the host already carries hal0's IOMMU line, leave it; if it carriesamd_iommu=off, warn and leave it — the operator chose it knowingly. Default (2026-10-04) — opt-in, never default-on.drivers.mdx:86-97: on kernels ≥ 6.14 amdgpu grows the GTT limit dynamically, hal0 reads the livemem_info_gtt_totalon every request, and pinning is documented as an operator choice. So the lib applies fixed sizing only on an explicit request (flag or--apply), the doctor reports the live pool and offers the recipe, and on a kernel ≥ 6.14 the default is to change nothing — above all on a multi-tenant Proxmox host, where a 65–90 % reservation starves every other guest and adds a boot-critical GRUB/initramfs edit for no measured gain. Correction (2026-10-04) — a container guard is mandatory. hal0's common deployment is a Proxmox LXC. Inside a container/etc/default/grub,/etc/modprobe.d/, the initramfs andamdgpumodule parameters belong to the host kernel; editing the guest's copies and asking for a reboot applies nothing (and the IOMMU parameter is out of the port's scope regardless — see above). ODS's phase 10 has no such guard (10-amd-tuning.shonly remarks that containers need/dev/kfd,:33-55), so a literal port would report "tuning applied" on every LXC. The hal0 port must branch onsystemd-detect-virt --container(the installer already distinguishes LXC inpreflight_gpu,install.sh:363-380,500-509): on bare metal apply as above; in a container do not derive the tier from the guest's memory — an LXC sees its cgroup allocation, not host RAM (docs/getting-started/proxmox.mdx:171-173; the recipe's 32 GB guest on a 128 GB host would pick the wrong tier and undersize the host GTT; correction 2026-10-04) — so emit a host-side script that reads the host's/proc/meminfoand computes the tier there, or take--host-ram-gbexplicitly, and print the host-side recipe (the GTT/TTM/etc/default/grubadditions,update-grub, the/etc/modprobe.d/drop-in, the sysctls) for the operator to run on the Proxmox host, and record the phase asskipped: container, never as applied. Thehal0 doctorarm (item 3) should check live values (/sys/module/amdgpu/parameters/gttsize,/proc/cmdline) rather than the files it would have written. -
HSA_OVERRIDE_GFX_VERSION auto-detection for gfx1151. Source:
.env.example:477+ ODS's gfx1151-conditional phase-06 logic. Target:src/hal0/install/profile_derive.pyor the ROCm slot default[server].envnearsrc/hal0/config/schema.py:298-306. Size: ~15-20 lines. Risk: low — one conditional env default into an injection point that already exists; only care needed is not clobbering an operator-set value. -
Kernel/firmware known-bad-version checks in
hal0 doctor. Source: the same upstream README's warnings (kernel <6.18.4 unstable on gfx1151;linux-firmware-20251125"breaks ROCm support,"amd-strix-halo- toolboxes/README.md:53) plus ODS's version-gated remediation pattern (installers/phases/05-docker.sh:194-244). Target: a new check insrc/hal0/cli/doctor_commands.pyreadinguname -randrpm -q/dpkg -lforlinux-firmware. Correction (2026-10-04): inside an LXC the kernel half still holds (uname -ris the host kernel) but the firmware half does not — the GPU runs firmware the host kernel loaded from the Proxmox host'slinux-firmware, so the guest package is irrelevant. Undersystemd-detect-virt --containerthe check must say "firmware is host-side", print the host command (dpkg -l linux-firmwareon the Proxmox host) and accept host-sourced evidence, rather than passing or warning on the guest's package. Size: ~40-60 lines. Risk: low — read-only. -
Resource limits on slot Quadlet units. Source: ODS's
deploy.resources.limits.{cpus,memory}convention (docker-compose.base.yml:173-180). Target:_render_quadlet_from_planinsrc/hal0/providers/container.py:803-990+— Quadlet has native[Service]keys (MemoryMax=,CPUQuota=) that drop in next to the existingRestart=/RestartSec=block, sourced from a new optional profile/slot field. Size: ~25-40 lines plus one schema field. Risk: low-medium — additive when unset, but should default off since hal0 currently assumes an unconstrained, single-tenant appliance box. -
CI matrix that validates a materialized config per bundle tier. Source:
.github/workflows/validate-env.yml's 5-tier matrix. Target: a GitHub Action rendering eachinstaller/manifests/omni/<tier>.jsonbundle's defaultslots.toml/capabilities.tomland asserting it loads underhal0.config.schema.Hal0Configwithout aValidationError— closes the CI-coverage gap ODS's matrix has and hal0's (already-stronger) Pydantic validation does not. Size: ~50 lines. Risk: low. -
AMD multi-GPU topology detection (
installers/lib/amd-topo.sh) is lower priority: Strix Halo is single-iGPU, so this only matters if hal0 ever supports discrete multi-GPU AMD boxes.
E. Do-not-copy
- The 270 KB / 6,595-line
ods-climonolith. ODS's own decomposition doc concedes the problem; hal0's modular Typer tree (35 files, per-group ownership, issue-annotated mount points) is already the healthier architecture. Do not regress toward one giant dispatch file. - Compose "profiles removed, everything is core" as a configurability
model (
docs/PROFILES.md:39-41). ODS's own history is a cautionary tale: it had opt-in profiles and removed them, ending up with "disable a service by hand-editing an override file." hal0's per-slot, per-capability TOML enable/disable is strictly more granular already. - Silently downgrading a system package as an installer side effect
(Docker 29.3.x→29.2.1,
installers/phases/05-docker.sh:205-223). Reasonable for ODS's Docker-Engine-dependent world, but "installer reaches into apt/dnf and force-downgrades a system daemon" is not a pattern for hal0's podman/systemd model — detect-and-warn is the safer analog. - The written acceptance of unpinned
:latesttags (docs/ADR-IMAGE-TAG-PINNING.md) is strictly looser than hal0's current digest-pin-everything discipline; don't use it as license to relax hal0's bar. - Multi-platform installer sprawl (
installers/macos/,installers/windows/,docs/WINDOWS-*,docs/WSL2-*) is out of scope unless hal0 commits to non-Linux support.
F. Owner decisions
- Adopt AMD host tuning (D.1) as default-on or opt-in? (Answered
2026-10-04: opt-in — see the Default note in D.1.) It touches GRUB
and needs a reboot; hal0 would want a "reboot recommended" banner (ODS's
pattern,
installers/phases/10-amd-tuning.sh:289-302) and a call on whether a single-target-hardware product should just always apply it. - Close the #1870-class long-blocking lifecycle calls via the event
bus hal0 already has (
src/hal0/events/,/api/journal) instead of a bare blockingapi_post+ multi-thousand-second timeout — is streaming slot-state transitions into the CLI (and eventually bounding the server-side pieces named open atslot_lifecycle_budget.py:107, #1869) worth prioritizing now, or does it wait on #1869 first? - Fix #1844 properly: should
activate_release()(src/hal0/updater/updater.py) re-run the/usr/local/bin/hal0symlink step idempotently on every update, or is/usr/local/bin/*deliberately outside the updater's managed tree, better enforced by re-invokinginstall.shunattended at the end ofhal0 update? - Extend hal0's digest-pinning discipline into a published "known-good
kernel/firmware" doc, mirroring
docs/KNOWN-GOOD-VERSIONS.mdand the amd-strix-halo-toolboxes warnings — worth wiring intohal0 doctor(D.3) rather than documentation alone. - Resource limits on slot containers (D.4): is "unconstrained,
single-tenant appliance" intentional for a Strix-Halo-only product, or a
gap worth closing before a multi-slot box (chat + code + embedding
resident at once,
slot_lifecycle_budget.py:92-94) can starve itself?
09-issue-map.md Open-issue map
ODS → hal0 issue map
Scope. All 106 open hal0 issues (snapshot 2026-09-05) triaged against the ODS reference tree. Every "Direct port" / "Pattern adoption" row below was verified by opening the cited ODS code and the cited hal0 code; "Informs only" rows were verified on the ODS side and spot-checked on the hal0 side; "Unrelated" rows were checked only far enough to establish that no ODS shape applies.
Path conventions. ODS paths are relative to /home/user/ods/ods/ unless they
start with .github/ (those are /home/user/ods/.github/). hal0 paths are relative
to /home/user/hal0/.
Verdict definitions.
| verdict | means |
|---|---|
| Direct port | An ODS file/function transliterates into hal0 with only naming/runtime substitution and closes the issue outright. |
| Pattern adoption | ODS's structure solves the issue; the code must be rewritten for hal0's runtime (podman/quadlet/FastAPI) but the design decision is settled. |
| Informs only | ODS's handling is a useful reference or precedent — it narrows the design space but does not itself close the issue. |
| Unrelated | No ODS shape applies (polish sweeps, test hygiene, upstream tracking, docs, hal0-only seams). |
1. Summary
| verdict | count | share |
|---|---|---|
| Direct port | 4 | 3.8% |
| Pattern adoption | 34 | 32.1% |
| Informs only | 29 | 27.4% |
| Unrelated | 39 | 36.8% |
| total | 106 |
The honest read: just over a third of the backlog is untouchable from ODS — 31 polish/test/docs/upstream issues plus 8 that ODS simply does not model (Hindsight memory internals, the runner image build, a11y). But the high-severity operational half of the backlog maps very well: 18 of the 20 issues I'd call "an operator hits this on a real box" land in Direct port or Pattern adoption.
2. Full triage table
| # | title (short) | verdict | ODS shape (file:line) | what to do in hal0 | eff | imp |
|---|---|---|---|---|---|---|
| 1974 | _flm_image_present caches False forever | Direct port | extensions/services/dashboard-api/helpers.py:26-56 (_DirSizeCache TTL) |
Give the negative probe a TTL (or cache only the positive result) | S | M |
| 1869 | bound systemd/podman subprocesses | Direct port | ods-uninstall.sh:321; scripts/bootstrap-upgrade.sh:939; ods-cli:2166-2172 |
Pass an explicit timeout= on every seam call in the load path (container.py:2697/2708/2900) | S | H |
| 1521 | retention prune runs only at boot | Direct port | memory-shepherd/install.sh:150-226 (timer+service pairs); memory-shepherd.sh:329 |
Run prune on an interval task/systemd timer instead of once in the API lifespan | S | M |
| 1512 | stacks import never verifies checksum | Direct port | ods-restore.sh:273-310 |
Verify the envelope digest on the commit path; fail closed; keep --skip-verify as the explicit opt-out | S | H |
| 2234 | default_images[family] only on llama path | Pattern adoption | installers/lib/compose-images.sh:40-71 |
Resolve every slot image through one plan-derived seam so no provider path can bypass default_images[family] | M | M |
| 2221 | last_crash_line None on exit 64 | Pattern adoption | installers/lib/compose-failure-report.sh:92-186; ods-cli:1080-1128 |
Replace the single crash-line regex with a bounded evidence report: exit code + last N unit-log lines + likely image | M | M |
| 2216 | LXC /dev/kfd seeds gpu-vulkan | Pattern adoption | installers/lib/detection.sh:187-233; installers/phases/02-detection.sh:159-168 |
Make device-node presence (kfd_present() AND render_node_present()) sufficient for the ROCm lane; never gate a lane on a userspace tool being installed | S | H |
| 2212 | verify-files / re-pull for model + mmproj | Pattern adoption | ods-restore.sh:273-310; config/model-library.json:5-14 (gguf_sha256) |
Carry per-file sha256 in the registry row and verify/re-pull on demand for model + mmproj | M | M |
| 2203 | seed-profile route emits changed_fields always | Pattern adoption | ods-cli:6261-6324 (_template_preview delta) |
Compute the real delta once and emit changed_fields only when it is non-empty | S | M |
| 2201 | runs_on reads hardware per row per poll | Pattern adoption | extensions/services/dashboard-api/helpers.py:26-56, 585-619 |
Resolve hardware once per request from a TTL-cached snapshot written by a poller, not per row | S | M |
| 2195 | apply preview omits removed flags | Pattern adoption | ods-cli:6261-6324 |
Enumerate the complete delta (added/changed/removed) from one diff function shared with apply | S | M |
| 2192 | Omni never sees tts/stt/embed modality | Pattern adoption | extensions/services/comfyui/manifest.yaml:19,38; extensions/services/dashboard-api/config.py:297-431 |
Declare modality/capability facts in the slot profile manifest and derive Omni eligibility from that table | M | H |
| 2180 | migrate-flags aborts whole run | Pattern adoption | ods-cli:2059-2113 (per-service accounting, continue then report) |
Make the sweep fault-isolated per slot and report a summary instead of aborting the run | S | M |
| 2164 | pi install fails as daemon uid 996 | Pattern adoption | installers/phases/07-devtools.sh:66-73, 386-395 |
Give the daemon identity its own npm prefix ($HOME/.npm-global) or run the install as the target user with HOME set | M | M |
| 2108 | tool_model has no UI path | Pattern adoption | extensions/services/dashboard-api/settings.py:194-241; .env.schema.json |
Render settings fields from the config schema so a new key gets a UI path automatically | M | M |
| 2096 | update should restart slots on image roll | Pattern adoption | ods-cli:2049 (up -d --force-recreate); ods-cli:1601 (--pull never) |
Make recreate-on-image-roll the default and require an explicit opt-out | S | H |
| 2028 | services/health static three-branch | Pattern adoption | extensions/services/dashboard-api/config.py:297-431; helpers.py:624-700 |
Build the health list from a manifest table and fan out with gather(return_exceptions=True) | M | H |
| 2019 | dev-deploy leaves wrappers stale | Pattern adoption | installers/phases/13-summary.sh:283; ods-cli:2049 |
Have deploy.sh run the same wrapper/PATH refresh step hal0 update runs (refresh_privileged_wrappers) |
S | H |
| 1990 | _VALID_FACT_TYPES hardcoded allowlist | Pattern adoption | extensions/schema/service-manifest.v1.json; scripts/validate-manifest-schema.sh |
Move the fact-type set into a shared schema both sides validate against, with a CI drift check | M | M |
| 1983 | gpu-perms Before=hal0.target loses race | Pattern adoption | docker-compose.base.yml:463-465 (depends_on: condition: service_healthy) |
Add an explicit per-unit ordering edge on the slot template (After=/Requires=), not a target-level Before= | S | H |
| 1966 | ComfyUI has no lane on kfd-less AMD | Pattern adoption | extensions/services/comfyui/manifest.yaml:19,38 (gpu_backends) |
Declare each runtime's supported lanes in the manifest; the picker offers only those and explains the gap | M | H |
| 1936 | segfault with zero GPU devices | Pattern adoption | installers/lib/detection.sh:187-233, 235-257 (apply_cpu_gpu_fallback) |
Refuse to spawn a GPU lane when no device nodes are mapped; fall back to CPU with a named reason | S | H |
| 1870 | lifecycle verbs wait 966s silently | Pattern adoption | installers/phases/12-health.sh:66-106; installers/lib/background-tasks.sh:119-148; readiness-summary.sh:52-127 |
Stream per-phase progress while waiting and bound the wedged call with an elapsed/deadline loop | M | H |
| 1868 | static seeds warm brain at 65536 | Pattern adoption | installers/lib/tier-map.sh:200-330; scripts/select-model.py:170-182 |
Derive seed context_size from the detected memory envelope, exactly as derive_device already derives the lane | M | H |
| 1867 | anchor window below Hermes floor | Pattern adoption | installers/phases/11-services.sh:1059 (_hermes_context=${MAX_CONTEXT}) |
Derive the agent window from the slot's served window and raise the ceiling at provision time | M | H |
| 1862 | stale hardware.json books VRAM as RAM | Pattern adoption | installers/lib/constants.sh:30; installers/lib/detection.sh:26-46; scripts/ods-doctor.sh:81-85 |
Rebuild the hardware fact from a live probe when the cache is absent/stale instead of degrading to 'no GPU' | M | H |
| 1859 | llama-only ctx resolver on FLM/Kokoro | Pattern adoption | extensions/schema/service-manifest.v1.json; scripts/load-backend-contract.sh |
Gate the context resolver on the slot's declared runtime family | S | M |
| 1845 | remediation panel cannot work | Pattern adoption | scripts/ods-doctor.sh:1110-1163 (issue id -> next_steps table) |
Emit runnable next_steps as data from the server, including --stop-services | S | H |
| 1844 | update never refreshes /usr/local/bin/hal0 | Pattern adoption | installers/phases/13-summary.sh:270-300 (ln -sf into the install dir) |
Re-assert the PATH symlink from the activated release next to refresh_privileged_wrappers | S | H |
| 1834 | extraction queue never drains on CPU | Pattern adoption | installers/lib/tier-map.sh:200-330; ods-cli:400-480 (ensure_llama_cpu_budget) |
Size background extraction from the detected tier: concurrency cap, max_tokens, and a retry ladder floor | M | H |
| 1833 | memory status green on empty store | Pattern adoption | installers/lib/readiness-summary.sh:47-96 (ready requires a real signal) |
Require a positive landed-fact signal for 'Writes landing', not absence of failures | S | M |
| 1825 | build_roster collapses model.gguf models | Pattern adoption | config/model-library.json:5-14 (id + gguf_sha256); scripts/select-model.py:76-115 |
Canonicalise the roster by content digest / registry id, never by gguf basename | S | M |
| 1823 | capability labels in hand-curated web file | Pattern adoption | config/model-library.json:1-40; scripts/select-model.py:76-115 |
Move capability labels into the versioned model registry and have the leaderboard read the registry | M | M |
| 1822 | LAN bind with auth off unsurfaced | Pattern adoption | scripts/ods-doctor.sh:1074-1089; .env.example:50-70; installers/phases/13-summary.sh:419-427 |
Add a doctor finding + install-summary line when the bind is non-loopback and auth is off | S | H |
| 1820 | netavark black hole has no repair owner | Pattern adoption | ods-cli:975-1060 (detect known-recoverable failure, repair, bounded retry); ods-cli:6115-6204 (cmd_repair) |
Call the existing prune_dnat seam from the reconciler behind a bounded retry, and expose a repair verb | M | H |
| 1550 | deploy reports success on old code | Pattern adoption | ods-update.sh:378-406 (wait_for_healthy); 647-742 (verify, then record version) |
Assert the served build identity after restart before printing success; roll back on failure | S | H |
| 1519 | uninstall prints 'Uninstalled' on 207 | Pattern adoption | ods-uninstall.sh:362, 394-402 (INSTALL_DIR_CLEANED + residual remedy) |
Track per-step outcomes, gate the final line on them, and print residual paths with the remedy command | S | M |
| 1511 | stack apply unloads untouched slots | Pattern adoption | ods-cli:6261-6324 (_template_preview: already / will-enable / nothing to change) |
Show the full delta including the unloads in the confirm dialog; make the unload sweep opt-in | M | H |
| 2200 | comfyui self-heal on list not single GET | Informs only | extensions/services/dashboard-api/helpers.py:624-680 |
Share one enrichment/self-heal function between the list and single-GET routes | S | M |
| 2191 | generate_image self-deadlocks single GPU | Informs only | installers/lib/model-lifecycle-lock.sh:15-60 |
Serialize GPU-exclusive ops behind a named lock with a bounded wait instead of a 503 mid-loop | M | H |
| 2190 | _runtime_family ignores runner aliases | Informs only | lib/service-registry.sh:29-31 (SERVICE_ALIASES) |
Fold runner aliases in one registry seam every classifier consults | S | M |
| 2184 | curated slots reference demoted profiles | Informs only | extensions/schema/service-manifest.v1.json; scripts/validate-manifests.sh |
Add a CI validator asserting every curated slot reference resolves to a shipped seed profile | S | M |
| 2178 | backfill Model.architecture | Informs only | ods-update.sh:678-690 (migrations dir); config/model-library.json |
Ship an idempotent data migration executed by the updater's migration step | S | M |
| 2154 | bank hygiene surface | Informs only | memory-shepherd/memory-shepherd.sh:110-148,329; memory-shepherd/install.sh:150-226 |
Age/size thresholds + a retention sweep on a timer, with a guarded 'suspiciously small → skip' rule | M | M |
| 2118 | retire upstream runner-image variant | Informs only | installers/lib/compose-images.sh:40-71 |
Derive the image set from the resolved plan so retiring a variant is a one-line data change | S | L |
| 2101 | GA delivery checklist | Informs only | ods-update.sh:456-505 |
Resolve 'latest' from the release API itself so tagging IS delivery — no separate pointer to forget | M | H |
| 1969 | nothing pins llama-server LOADING | Informs only | extensions/services/dashboard-api/helpers.py:661-672 (degraded != down) |
Model 'up but loading' as a distinct state and never feed it to the output-sanity gate | S | M |
| 1967 | save_slot_config drops keys on round-trip | Informs only | extensions/services/dashboard-api/settings.py:194-241 |
Round-trip through a schema-derived field table so unknown keys survive the write | M | M |
| 1947 | support CIRU vLLM distributions | Informs only | extensions/services/*/manifest.yaml; docker-compose.*.yml overlays |
Add a runtime family as manifest + overlay data rather than code | L | M |
| 1932 | read_slot_ceiling assumes .toml | Informs only | lib/service-registry.sh:29-48 |
Resolve the ceiling through the naming seam rather than assuming <name>.toml |
S | M |
| 1931 | extraction preflight trusts config | Informs only | extensions/services/dashboard-api/helpers.py:624-680 |
Preflight the LIVE drop-in/endpoint, not the configured intent | S | M |
| 1930 | MCP memory_add has no ctx preflight | Informs only | extensions/services/dashboard-api/helpers.py:624-680 |
Route both HTTP and MCP add paths through one preflight helper | S | M |
| 1929 | memory-map GTT fallback reads dead keys | Informs only | installers/lib/llama-memory-budget.sh:6-31 |
Same single-source memory envelope; fail loudly rather than silently substituting system RAM | S | M |
| 1928 | memory ruler header/bar different bases | Informs only | installers/lib/llama-memory-budget.sh:18-31 |
Derive header and bar from ONE effective-memory function so they cannot disagree | S | M |
| 1873 | self-call clients bare-float timeouts | Informs only | ods-cli:3851-3880 (explicit per-call timeout argument) |
Import slot_lifecycle_budget in the brain self-call clients instead of bare floats | S | M |
| 1858 | by-name/by-id unenriched payload | Informs only | extensions/services/dashboard-api/helpers.py:624-700 |
Share one enrichment function across /api/slots, /by-name and /by-id | S | M |
| 1545 | every PR conflicts on CHANGELOG.md | Informs only | .github/pull_request_template.md; CHANGELOG.md:1-30 |
ODS has no fragments — but its PR template requires no CHANGELOG edit; notes are authored at cut time | M | M |
| 1537 | Ubuntu 26.04 py3.14 --ignore-requires-python | Informs only | installers/lib/python-runtime.sh:36-72,127-171 |
Provision/select a supported interpreter and refuse loudly with a named remedy instead of --ignore-requires-python | M | M |
| 1530 | stable channel pointer still 0.9.8 | Informs only | ods-update.sh:456-505 |
Same as #2101 — derive the update target from the release API rather than a hand-maintained channel pointer | S | H |
| 1522 | journald follow has no keep-alive | Informs only | extensions/services/dashboard-api/routers/talk.py:393-397,455-495 |
Wrap journalctl_sse in the keepalive + disconnect loop hal0 already ships in journal.py / slots/logs.py | S | M |
| 1436 | artefact names still derived from slot NAME | Informs only | lib/service-registry.sh:29-48 |
Route the four remaining artefact names through the naming seam | S | M |
| 1429 | quadlets hard-code LogDriver=none | Informs only | docker-compose.base.yml:12-16 (single logging anchor + rotation) |
Already switched to LogDriver=passthrough (container.py:890); adopt one log-policy constant plus rotation | S | M |
| 1428 | slot_sample UNIQUE constraint floods log | Informs only | extensions/services/token-spy/db.py:37 (surrogate PK, no natural-key UNIQUE) |
Drop the (ts, slot_id) natural key for a surrogate PK, or use INSERT ... ON CONFLICT DO UPDATE | S | M |
| 1426 | hal0-systemctl has no unmask verb | Informs only | ods-cli:6115-6204 (cmd_repair dispatcher) |
Add unmask to _UNIT_VERBS + the wrapper allow-list and expose it as a repair verb | S | M |
| 1422 | /api/slots duplicate entries | Informs only | lib/service-registry.sh:29-48 (id-keyed maps + alias table) |
Key the roster by durable id with names as an alias map, and dedupe at load | M | M |
| 1421 | POST /api/slots writes name-keyed | Informs only | lib/service-registry.sh:29-48 |
Route slot creation through the naming seam so an id-keyed box never gets a name-keyed artefact | M | H |
| 1319 | integrate llama-ai / CachyLLama | Informs only | installers/lib/tier-map.sh:200-330; docker-compose.*.yml overlays |
Add a runtime as a data-declared backend + compose/profile overlay rather than new code paths | L | L |
| 2228 | comfyui phase4 tests patch shared isfile | Unrelated | — |
Inject the filesystem seam per provider instead of patching os.path.isfile process-wide | S | L |
| 2202 | promote crud app fixtures to conftest | Unrelated | — |
Test-fixture consolidation | S | L |
| 2169 | cut PR wall clock | Unrelated | — |
ODS CI has no path gating/xdist to copy (single lint+smoke workflows) | M | M |
| 2168 | execute v1.1.0 sunset backlog | Unrelated | — |
Repo chore; no ODS analogue | M | L |
| 2155 | memory migrate unify HTTP 410 | Unrelated | — |
Upstream engine API version pin; no ODS analogue | M | M |
| 2153 | guide for external coding agents | Unrelated | — |
Documentation | M | L |
| 2111 | docs-sync fragment anchors | Unrelated | — |
Docs-sync anchor rewriting | S | L |
| 2017 | memory plugin one uuid4 doc per turn | Unrelated | — |
Hermes plugin metadata; no ODS analogue | M | M |
| 2016 | no document-ingest endpoint | Unrelated | — |
ODS delegates RAG ingest to Open WebUI's own Knowledge surface — no endpoint to port | M | M |
| 2011 | cross-backend profile switching dead code | Unrelated | — |
Dead-code removal | S | L |
| 2003 | docs-discourse-sync reconcile | Unrelated | — |
Forum sync bookkeeping | M | L |
| 2002 | flaky warming-slot test | Unrelated | — |
CI-only state race; fix the test's readiness barrier | S | L |
| 1997 | no e2e for slab-truncation notice | Unrelated | — |
Missing e2e coverage | S | L |
| 1996 | tag-chip dimming keys dead field | Unrelated | — |
UI field-name fix | S | L |
| 1995 | memory v2 mock graph dialect | Unrelated | — |
Test mock dialect fix | S | L |
| 1994 | no toast for queued agent delete | Unrelated | — |
UI toast | S | L |
| 1993 | no operation-retry affordance | Unrelated | — |
UI retry affordance | S | L |
| 1989 | list_memories comma-joins multi-type | Unrelated | — |
Client query encoding bug | S | M |
| 1984 | _vulkan_lane_is_loadable stale pin | Unrelated | — |
Cache/pin invalidation bug internal to the retag pass | S | M |
| 1948 | restore Vulkan backend / refresh upstreams | Unrelated | — |
Runner image build work | L | H |
| 1925 | A/B non-AMD Vulkan lanes | Unrelated | — |
Validation campaign | L | M |
| 1841 | polish(ui) rc.5 sweep | Unrelated | — |
UI polish sweep | S | L |
| 1840 | polish(cli/api) rc.5 sweep | Unrelated | — |
13-item CLI/API polish sweep | M | L |
| 1829 | board tools 401 on fresh install | Unrelated | — |
Upstream wheel packaging gap | M | H |
| 1821 | nothing asserts slot-owned flag reaches argv | Unrelated | — |
Test coverage for argv rendering | M | M |
| 1783 | converge dashboard tables on .dtable | Unrelated | — |
UI table convergence | M | L |
| 1756 | quadlet allow-list follow-ups | Unrelated | — |
hal0-specific sudo wrapper allow-list; ODS has no equivalent seam | S | M |
| 1552 | Menu primitive has no keyboard support | Unrelated | — |
Frontend a11y work | M | M |
| 1536 | polish(board/palette) GA sweep | Unrelated | — |
Polish sweep | S | L |
| 1529 | polish(journal/logs) GA sweep | Unrelated | — |
Polish sweep | S | L |
| 1528 | polish(agent-cli) GA sweep | Unrelated | — |
CLI polish sweep | S | L |
| 1525 | polish(upstreams) GA sweep | Unrelated | — |
Polish sweep | S | L |
| 1524 | polish(stacks) GA sweep | Unrelated | — |
Polish sweep | S | L |
| 1502 | scar ratchet counts prose | Unrelated | — |
Repo CI ratchet tuning | S | L |
| 1477 | low-priority polish sweep | Unrelated | — |
31-item polish sweep | L | L |
| 1445 | typed request bodies audit | Unrelated | — |
Typed-body audit follow-through | M | M |
| 1349 | Strix Halo field results | Unrelated | — |
Field-report tracking issue | L | L |
| 1249 | track hermes-agent py3.14 | Unrelated | — |
Upstream tracking issue | S | M |
| 1070 | validate/promote ROCmFPX variant | Unrelated | — |
Quantization validation work | L | M |
3. Top 15 wins
Ranked by (impact ÷ effort). Each is an ODS shape I opened and a hal0 defect I located; the pointers are exact.
1. #1869 — bound the systemd/podman lifecycle subprocesses (S / H, Direct port)
ContainerProvider._run (src/hal0/providers/container.py:2549-2566) defaults
timeout=None, and the three calls on the load path pass nothing:
container.py:2697 (daemon-reload), container.py:2708 (systemctl restart <unit>),
container.py:2900 (daemon-reload in the re-render sweep). SystemCtlSeam.systemctl
(src/hal0/system/seam.py:415-489) forwards the bound faithfully on both routes — the
plumbing already exists, nobody passes a number.
ODS never leaves a hang-prone privileged call unbounded:
ods-uninstall.sh:321 — timeout 20s sudo -n -- systemctl disable --now …;
scripts/bootstrap-upgrade.sh:939 — ps_env_cmd+=(timeout --foreground --kill-after=5s "${restart_timeout}s");
ods-cli:2166-2172 — a perl -e 'alarm 3; exec "docker","info"' guard, with the
comment explaining that docker info hangs 20+s on a half-booted daemon.
Fix: give _run a non-None default (the existing _UNIT_STOP_TIMEOUT_S is already
imported at container.py:2943) and let subprocess.TimeoutExpired become a typed
slot.spawn_failed the same way CalledProcessError already does at container.py:2709-2716.
That also shrinks the client budget in slot_lifecycle_budget.py — see #1870.
2. #1822 — LAN bind with auth off is never surfaced (S / H)
ODS surfaces this in three places, all data-driven:
scripts/ods-doctor.sh:1074-1089 raises a warn finding
(ODS-RUNTIME-EXTERNAL-LEMONADE-UNAUTHENTICATED-HOST-ROUTE) whose detail says exactly
"if the daemon is bound beyond loopback so Docker can reach it, that same daemon may
also be reachable from the LAN"; scripts/ods-doctor.sh:1150-1155 carries its
next_steps; installers/phases/13-summary.sh:419-427 prints the LAN URL only when
BIND_ADDRESS=0.0.0.0, and .env.example:50-58 states the danger inline on the
OPENCLAW_DANGEROUSLY_DISABLE_DEVICE_AUTH key.
Fix: add one doctor check (src/hal0/cli/doctor_all.py, alongside
check_hermes_anchor_window at doctor_all.py:1046) that reads the bind address and
the auth toggle together and emits a warn with a remedy; echo the same line in the
installer summary.
3. #1845 — the "Convergence incomplete" panel prints a remediation that cannot work (S / H)
src/hal0/cli/update_commands.py:466-492 renders entry.get('command') verbatim under
prose that says "Stop hal0, then run:" — but the command the server hands over does not
carry --stop-services, so a literal copy-paste refuses.
ODS keeps remediation as a table keyed by a stable issue id, never prose glued to a
value: scripts/ods-doctor.sh:1110-1122 maps id → title, :1124-1163 maps id →
next_steps (a list of runnable lines), and :1165-1175 assembles the diagnosis.
Fix: have /api/updates/convergence return next_steps: [str] per pending migration
(built server-side, where --stop-services is known), and make the panel print exactly
those lines. Same change fixes the class, not just this instance.
4. #1844 — hal0 update never refreshes /usr/local/bin/hal0 (S / H)
installer/install.sh:1077-1082 links /usr/local/bin/hal0 → ${VENV_DIR}/bin/hal0 and
the comment claims it "survives upgrades because it points at the venv shim, not a copy" —
but install.sh is the only writer, and hal0 update never re-asserts it.
hal0 already solved the identical class for the sudo wrappers:
src/hal0/updater/updater.py:3963-4096 refresh_privileged_wrappers(), called from
updater.py:4159-4163. ODS's equivalent is installers/phases/13-summary.sh:270-300,
which ln -sfs the CLI into the install dir on every install run and falls back to
~/.local/bin with a PATH warning when sudo is unavailable.
Fix: add refresh_path_links(target) next to refresh_privileged_wrappers and call it
from the same activation step. ~30 lines.
5. #1550 — deploy.sh reports success while the backend serves old code (S / H)
scripts/deploy.sh:213-231 polls /api/status for HTTP 200 and then prints
deploy complete @ <sha> — the sha comes from git rev-parse, i.e. from the tree,
never from the process.
ODS gates the success line on a health verdict and rolls back otherwise:
ods-update.sh:378-406 wait_for_healthy() (deadline loop, per-attempt remaining-time
log, dumps the final failing status), ods-update.sh:717-724 calls
_update_rollback when it fails, and only :725-740 records the new version.
ods-cli:2059-2113 does the per-service variant: enumerate the stack, check each
container's State.Status/ExitCode, and exit 1 with "Run 'ods rollback'".
Fix: expose the running build's version/commit on /api/status (or read
hal0.__version__ from the live process) and compare it to git rev-parse HEAD before
printing success.
6. #2216 — LXC with /dev/kfd forwarded seeds every slot gpu-vulkan (S / H)
src/hal0/install/profile_derive.py:110-113:
rocm_ok = any(g.compute_capable for g in hw.gpus) or kfd_present()
if rocm_ok and (hw.platform == "strix-halo" or any(g.compute_capable for g in hw.gpus)):
return "gpu-rocm"
kfd_present() can satisfy the first clause but never the second, and
compute_capable is set from a rocm-smi/rocminfo exit code
(src/hal0/hardware/probe.py:483-484). So a container with /dev/kfd forwarded but no
ROCm userspace installed falls through to Vulkan.
ODS decides the AMD lane from device nodes only —
installers/lib/detection.sh:187-210 (amd_gpu_missing_runtime_devices checks
/dev/kfd is a char device and /dev/dri/renderD* exists) and
installers/phases/02-detection.sh:159-168 gates the whole lane on that, with
show_amd_gpu_device_guidance (detection.sh:218-233) printing the LXC passthrough
commands when it fails.
Fix: make both device-node predicates together sufficient for the ROCm branch —
kfd_present() and render_node_present() — and drop the rocm-smi dependency.
Correction (2026-10-04): the first draft said kfd_present() alone; that loses the
second half of the ODS check (renderD* must exist, detection.sh:200-204), and a
Strix Halo LXC with only /dev/kfd forwarded would then seed every slot gpu-rocm and
launch it without a render device. hal0 already has both helpers
(src/hal0/providers/_gpu.py:251-280 kfd_present, :397-420 render_node_present),
each stricter than ODS's because they check openability by the slot runner uid, so this
is still a one-expression predicate change.
7. #2028 — /api/services/health is a static three-branch construction (M / H)
src/hal0/api/routes/services_health.py:198-307 hand-writes comfyui, hermes and
openwebui as three literal blocks; the docstring at :1-15 admits the list is fixed.
The dashboard footer's readiness count therefore cannot move on a hindsight-api outage.
ODS's dashboard-api is manifest-driven end to end:
extensions/services/dashboard-api/config.py:297-424 load_extension_manifests() reads
every extensions/services/*/manifest.yaml (schema-pinned to ods.services.v1) into
SERVICES/FEATURES at config.py:430-448; helpers.py:624-679
check_service_health() probes one service from its manifest port + health path with
per-service timeouts and a healthy/degraded/down/not_deployed verdict; and
helpers.py:682-700 get_all_services() fans out with
asyncio.gather(..., return_exceptions=True) so one misbehaving service can't take the
response down.
Fix: add a services.yaml-style table (or extend the existing slot/profile manifests)
carrying id, name, unit, probe_url, health_path, timeout, and replace the three blocks
with one loop. New services then appear by adding a row.
8. #1868 — static slot seeds are copied verbatim with no hardware budget (M / H)
installer/etc-hal0/slots/brain.toml ends with a literal [model] context_size = 65536,
copied unchanged onto every box by src/hal0/install/static_seeds.py:195-260. That
module already re-derives device per host through
LLAMA_SEED_CAPABILITIES (static_seeds.py:74-82) → derive_device; the context
window is the one field it doesn't touch.
ODS never lets a context window escape the hardware envelope:
installers/lib/tier-map.sh:200-330 binds MAX_CONTEXT to the tier (which is derived
from detected VRAM/RAM), and scripts/select-model.py:170-182
(estimated_context_kv_gb / selector_required_memory_gb) charges the KV cache against
the model's memory requirement, with usable_memory_gb (select-model.py:134-144)
giving the unified-memory box only 55% of RAM.
Fix: extend static_seeds.py's derivation pass with a _CONTEXT_LINE rewrite that
clamps context_size to a tier-derived ceiling — the same shape, the same file, one
more regex.
9. #1862 — a stale or missing hardware.json books VRAM as RAM (M / H)
src/hal0/slots/capacity.py:499-522 _host_has_capable_gpu() reads the cached
/etc/hal0/hardware.json and returns False on any error, and
build_per_slot (capacity.py:570-585) uses that bool to decide whether resident
memory is charged to VRAM or RAM.
ODS has no durable hardware cache to go stale. CAPABILITY_PROFILE_FILE defaults to
${TMPDIR:-/tmp}/ods-capabilities.json (installers/lib/constants.sh:30), and it is
rebuilt on every run by scripts/build-capability-profile.sh — from the installer
(installers/lib/detection.sh:26-46) and from the doctor
(scripts/ods-doctor.sh:81-85).
Fix: when the cache is missing or older than N hours, fall through to
hal0.hardware.probe's live light probe rather than to False. hal0 already has that
probe; only the fallback branch is wrong.
10. #1936 + #1966 — no GPU devices / no CPU lane (S–M / H)
#1936 is a segfault at model load when the container has zero GPU devices mapped;
#1966 is ComfyUI offering only gpu-vulkan on a kfd-less AMD box.
Two ODS shapes cover both. installers/lib/detection.sh:235-257
apply_cpu_gpu_fallback() is a single function that rewrites the whole hardware
verdict (GPU_BACKEND=cpu, GPU_VRAM=0, HAS_NPU=false, overlays cleared) and prints
the reason; it is invoked from four guard sites
(installers/phases/02-detection.sh:132,168,181, 11-services.sh:311). And
extensions/services/comfyui/manifest.yaml:19 declares gpu_backends: [amd, nvidia],
repeated on the feature at :38 — so the dashboard can only offer a lane the service
declares.
Fix: (a) refuse to spawn a GPU-lane container when the device nodes are absent, with a
named reason instead of a SIGSEGV; (b) put supported_backends on the ComfyUI runner
in the profile manifest and have the device picker render "requires /dev/kfd" instead of
an unusable row.
11. #1870 — lifecycle verbs wait 966s with no progress (M / H)
src/hal0/cli/slot_commands.py:51-54 derives the client budget from
slot_lifecycle_budget.slot_lifecycle_timeout_s() — correct, but the CLI then blocks on
one api_post for the whole window with nothing on stdout.
ODS's two halves: installers/phases/12-health.sh:66-106 _check_container_health()
prints ... Waiting for <name> and rewrites the line in place with \r each poll,
short-circuiting on exited|dead|missing; and installers/lib/background-tasks.sh:119-148
bg_task_wait() is the bounded elapsed/timeout loop with three distinct exits
(ok / failed / timeout). installers/lib/readiness-summary.sh:52-127 is the closing
report: Ready now: N/total, then Ready: / Needs attention: / Next: with the log
paths.
Fix: stream the server's phase transitions (the state machine already stamps them) over the existing SSE seam and render a per-phase progress line; keep the derived budget as the hard deadline.
12. #2096 — update should restart slots when the runner image rolls (S / H)
hal0 makes this opt-in: src/hal0/cli/update_commands.py:516-531 explicitly says "we
never bounce automatically" and tells the operator to run hal0 update --restart-slots.
ODS makes recreate the default and pinning the opt-out: ods-cli:2049 runs
up -d --force-recreate on every update, while ods-cli:1601 uses
--force-recreate --no-build --pull never for a plain restart so a restart can never
silently change the image. The image set itself is derived from the resolved plan, not
per-code-path (installers/lib/compose-images.sh:40-71).
Fix: invert the default when — and only when — /api/updates/slot-drift reports an
image change; keep --no-restart-slots for the operator who wants the old behaviour.
13. #1512 — stack import commit never verifies the envelope checksum (S / H, Direct port)
ods-restore.sh:273-310 is nearly transliterable: it looks for checksums.sha256,
picks sha256sum -c or shasum -a 256 -c, returns 1 on mismatch with
"This backup may be corrupted or tampered with", warns loudly when the file is absent
(older format), and only skips when --skip-verify was passed explicitly (:308-310).
It also warns when the archive is missing expected data paths (:313-320).
Fix: move the digest check hal0 currently runs only in dry_run onto the commit path,
fail closed, and add an explicit --skip-verify.
14. #1974 — _flm_image_present caches False and never re-probes (S / M, Direct port)
src/hal0/capabilities/catalog.py:178-235 caches the probe at module scope and clears it
only after a successful FLM pull (reset_flm_image_present_cache) — so a transiently
broken podman permanently drops the NPU backend from /api/capabilities.
ODS's _DirSizeCache (extensions/services/dashboard-api/helpers.py:26-56) is the
30-line answer: per-key TTL, expiry sweep on write, bounded size, explicit invalidate.
Fix: cache the positive result indefinitely and the negative one for ~60s, or don't cache the "no runtime / probe failed" branch at all — it is a different answer from "image absent".
15. #1820 — the netavark port black hole has no repair owner (M / H)
hal0 has both halves already and no wiring: detection in
src/hal0/system/netavark.py:58-170 (parses nft -a list table inet netavark) and the
privileged repair in src/hal0/system/seam.py:363-413 prune_dnat() (with a
dry_run/check-dnat mode). The reconciler parks the slot in ERROR instead of calling it.
ODS's _compose_run_with_summary (ods-cli:960-1140) is the template for
"known-recoverable failure → repair → bounded retry": it classifies the compose log
(ods-cli:1006 network errors, :1021 port-already-allocated, :1042 stale container
name conflict), performs the matching repair (_compose_remove_conflicting_ods_containers,
ods-cli:978-996 — which refuses to remove a non-ODS container), and retries under
per-class attempt counters and backoff (:996-1060). ods-cli:6115-6204 cmd_repair
is the operator-facing companion verb.
Fix: on the ERROR transition, run prune_dnat(dry_run=True); if it identifies a stale
rule, prune and retry the start once, then park with the evidence.
4. Clusters — one change, many issues
A. A progress / timeout / evidence seam
Closes or de-risks: #1870, #1869, #1873, #2221, #1550, #1820, #2180.
ODS treats "a long privileged call" as one shape with four obligations: a hard bound
(ods-uninstall.sh:321, bootstrap-upgrade.sh:939, ods-cli:2166-2172), visible
progress while waiting (12-health.sh:66-106, background-tasks.sh:119-148), a
classified retry for known-recoverable failures (ods-cli:996-1060), and a persisted
evidence report when it finally fails (compose-failure-report.sh:92-186 — redacted
env, port occupancy, compose ps, log tail — plus readiness-summary.sh:52-127).
In hal0 that is one module: a bounded_call() helper every seam invocation goes through
(container.py:2549, seam.py:415), a phase-progress event on the existing SSE bus, a
retry classifier for the netavark/name-conflict/port classes, and a
write_slot_failure_report() that the crash-line extractor feeds instead of replacing.
Doing this once removes the need to hand-tune slot_lifecycle_budget.py's
EVICTION_UNLOAD_ALLOWANCE caveat (src/hal0/slot_lifecycle_budget.py:95-107 explicitly
names #1869 as the real fix).
B. Manifest-driven service + capability registry
Closes or de-risks: #2028, #2192, #1966, #1859, #2190, #2184, #1990, #1823, #2234, #1947, #1319.
Every one of these is "a fact about a service/runtime is hard-coded in one code path and
the other paths can't see it." ODS answers all of them with one mechanism:
extensions/services/<id>/manifest.yaml validated by
extensions/schema/service-manifest.v1.json, loaded once by
lib/service-registry.sh:29-48 (bash) and
extensions/services/dashboard-api/config.py:297-431 (Python), and consumed by health
(helpers.py:624-700), features (routers/features.py:19-113) and the CLI. Backend
compatibility is a manifest field (comfyui/manifest.yaml:19); model capability is a
registry field (config/model-library.json); the image set is derived from the resolved
plan (compose-images.sh:40-71).
hal0 has the pieces scattered — RUNNER_IMAGES, seed profiles, _SLOT_META,
services_health.py's three literals, capabilities/catalog.py. One
hal0.services.registry loaded from a schema-validated table, read by the health route,
the Omni eligibility check, the device picker and the context resolver, would retire the
whole cluster.
C. Hardware re-detect + honest CPU fallback
Closes or de-risks: #1862, #2216, #1936, #1966, #1868, #1834, #1867, #2201, #1928, #1929.
ODS makes detection cheap and always fresh (tmp-file capability profile rebuilt every
run, constants.sh:30 + detection.sh:26-46 + ods-doctor.sh:81-85), decides GPU lanes
from device nodes rather than userspace tools (detection.sh:187-210), has exactly one
function that demotes a box to CPU with a stated reason
(detection.sh:235-257), and derives every memory/context number from the resulting
envelope (tier-map.sh:200-330, select-model.py:134-182,
llama-memory-budget.sh:6-31).
In hal0 this is: (1) live-probe fallback when hardware.json is stale/absent;
(2) kfd_present() and render_node_present() together sufficient for the ROCm lane; (3) one apply_cpu_fallback() that
rewrites the whole verdict; (4) one memory-envelope function that seed context sizes, the
memory ruler, the extraction sizing and the anchor-window floor all read.
D. Update / deploy delivery integrity
Closes or de-risks: #1844, #2019, #1550, #2096, #1845, #1519, #1512, #1530, #2101.
ODS's update is a transaction: snapshot → change → verify → record version, with
auto-rollback at every step (ods-update.sh:647-742, _update_rollback at :412-449,
wait_for_healthy at :378-406), the installed artefacts re-asserted on every run
(13-summary.sh:270-300), verification of the running stack before success
(ods-cli:2059-2113), and "latest" resolved from the release API so tagging is delivery
(ods-update.sh:456-505). Uninstall tracks per-step residue and names the remedy
(ods-uninstall.sh:362,394-402).
hal0 already has the hardest parts (cosign, staged trees, atomic symlink swap,
refresh_privileged_wrappers). What's missing is the cheap discipline: re-assert the
PATH link, verify the served build, verify digests on commit, gate the success line on
per-step outcomes, and let scripts/deploy.sh share the update path's refresh steps.
E. Schema-driven settings and previews
Closes or de-risks: #2108, #1967, #2195, #2203, #1511, #1445.
extensions/services/dashboard-api/settings.py:194-241 _build_env_fields() renders a
UI field for every key in .env.schema.json — label, type, enum, secret flag,
read-only reason, default — and falls through to a generic string field for keys the
schema doesn't describe. A new config key gets a UI path by existing, not by someone
hand-writing a control. The apply side is a declared allow-list per service
(settings.py:33-58). The preview side is ods-cli:6261-6324 _template_preview:
compute the delta, print "Already enabled / Will enable / Nothing to change", and keep
apply additive.
For hal0 that means one ChangeSet diff function shared by preview and apply (so
removals can't go unreported), and a schema-derived settings renderer so tool_model
and its siblings stop needing bespoke UI.
5. Where ODS has nothing to offer
Called out so nobody goes looking:
- Memory / Hindsight internals (#2016, #2017, #1989, #1990-partial, #1993-#1997, #2155): ODS's RAG is Qdrant + TEI behind Open WebUI's own Knowledge surface, and its "memory-shepherd" is a
MEMORY.mdfile-reset tool on a systemd timer — a different problem entirely. Only the hygiene/retention shape transfers (#2154, #1521). - CI throughput (#2169, #1502): ODS's workflows have no path gating, no concurrency groups and no test tiering (
/home/user/ods/.github/workflows/lint-python.yml:3-8). hal0's CI is the more mature of the two. - Changelog fragments (#1545): ODS uses a plain Keep-a-Changelog file too. The only transferable observation is that its PR template (
/home/user/ods/.github/pull_request_template.md) has no changelog checkbox — notes are written at cut time, so PRs never touch the file. - Polish sweeps (#1477, #1524-#1529, #1536, #1840, #1841, #1783, #1552): item-level UI/CLI work with no structural analogue.
- Upstream tracking (#1249, #1948, #1829, #2118, #1070, #1349, #1925): pins, builds and third-party bugs.