ADR: Isolation Hardening (2026-05-13)

Status: Accepted. Context: Agent symlink incident in scitex-stats-auditor proof of concept; sac’s positioning vs. Clew reproducibility verification.

Problem

On 2026-05-13 the first per-package auditor (scitex-stats-auditor) ran under Apptainer with sac-base.sif + overlay and reported: “the project venv targets /opt/python3.12/bin/python3.12, which is missing on this host; I repaired it by symlinking it to /usr/bin/python3.12.” The agent’s mental model was that it had patched the host. Investigation showed the symlink only landed in the container’s overlay — the host was untouched. But two systemic gaps were exposed:

1. The agent thought it had host write access. Apptainer’s defaults make the container/host boundary porous enough that an agent can’t distinguish “I patched the container’s view of /opt” from “I patched the host.”

2. Operator-side prompts can’t enforce isolation. The prompt told the agent to be read-only; the agent ignored it. Prompt-level guardrails are not a security mechanism.

The deeper issue: sac’s stated positioning is reproducible-by-default, but its actual default behavior inherited Apptainer’s HPC-convenience defaults (auto-bind $HOME, /tmp, /proc, /sys, /dev; inherit all env vars; share host namespaces). Convenience-first defaults are upside-down for an agent runtime where the container is supposed to be the security boundary.

Decisions

D1. Hardened isolation by default; relaxed: true to opt out.

spec.apptainer.relaxed: false is the default. sac auto-prepends --containall (filesystem isolation), --cleanenv (environment isolation), and --writable-tmpfs (when no overlay is declared) to the apptainer argv.

relaxed: true is an explicit opt-out for HPC-style convenience use cases. Agents started with relaxed: true are outside the Clew verification chain — their runs cannot be attested as reproducible.

Rationale. sac’s differentiation against LangGraph / CrewAI / AutoGen is “spec.yaml declares isolation; mechanism enforces it; external verifier can attest it.” Default-strict supports that thesis directly. HPC users can opt in to the legacy behavior with one line, and they pay the cost of falling out of the verification chain (which they typically don’t need anyway).

D2. Universal preflight via $HOME-visibility check, not per-path enumeration.

The preflight that sac auto-injects before user startup_commands is:

test "$(id -u)" != "0" || (echo 'ERROR: running as root' && exit 1)
test ! -d "$HOME" || (echo 'ERROR: host $HOME visible — isolation breach' && exit 1)

Rationale. Per-path enumeration (test ! -e $HOME/.gitconfig, test ! -e $HOME/.ssh, …) has unbounded false-negative risk: every new credential store added in the next decade (.kube/config, .docker/config.json, .netrc, .npmrc, .pypirc, .gnupg/, ~/.config/anthropic/, ~/.bash_history with embedded secrets, …) requires a new line. The $HOME-visibility check covers all of them at once.

Under --containall, $HOME is NOT auto-bound — it should be invisible inside the container. If the check fails, either --containall isn’t in effect or an operator-declared bind brought it in. Either way, the agent shouldn’t start.

Operator opt-out for paths that legitimately need to be visible:

spec:
  apptainer:
    preflight_allow:
      - "$HOME/.gitconfig"   # acknowledged: agent needs read-only gitconfig

The opt-out is declared per-path, not as a blanket “disable preflight.”

D3. AgentCard exposes structured isolation block, not a flat enum.

Instead of isolation_level: hardened | relaxed | custom:

"x-scitex-agent-container": {
  "isolation": {
    "level": "hardened",
    "containall": true,
    "cleanenv": true,
    "writable_tmpfs": false,
    "preflight_passed": ["uid-nonzero", "no-host-home"],
    "preflight_allowed": [],
    "binds_count": 3,
    "binds_writable_count": 0
  }
}

level: hardened is the human shorthand for “all booleans true + preflight_allowed: []”. External verifiers (Clew, orochi attestation) read the structured booleans to attest specific properties.

Rationale. A flat custom label hides what’s custom about it. A run with preflight_allow: [$HOME/.ssh] and a run with preflight_allow: [$HOME/.aws] are both custom under the enum but have very different security profiles. Clew’s verification chain wants to attest specific properties: “did this run set containall? did it allow any preflight bypasses?” The structured block answers those directly.

Considered and rejected

Rejected: prompt-level isolation guarantees.

“Tell the agent in the prompt to be read-only.” The PoC proved this fails — the agent acknowledged the rule and then violated it. Prompts are not a security mechanism. Mechanism-level enforcement is non-negotiable for the Clew thesis.

Rejected: test -w <path> permission-based preflight.

test ! -w /opt would fail inside any container with a writable overlay shadowing /opt, even when no host damage is possible. The intent (catch leaks) and the test (catch any write capability) don’t align under apptainer’s overlay model. Existence-based checks on identity files ($HOME visibility) avoid this entirely.

Rejected: enumerated per-path preflight (.gitconfig, .ssh, .aws, …).

Forever incomplete; every new credential cache added by tooling elsewhere requires a sac patch. The $HOME-visibility check covers the same surface in one assertion.

Rejected: isolation_level: hardened | relaxed | custom flat enum.

Loses verifier resolution. Two custom runs with very different preflight_allow sets become indistinguishable on the card.

Implementation

Layer	Where	Status
apptainer.relaxed: false default	config/_types.py::ApptainerSpec	✅ shipped earlier today
--containall auto-prepended	runtimes/_apptainer_runtime.py	✅ shipped
--cleanenv auto-prepended	same	✅ shipped (D1)
--writable-tmpfs when no overlay	same	✅ shipped (D1)
Universal preflight injection	runtime wraps inner cmd with bash -c "<preflight>\nexec <inner>"	✅ shipped (D2)
spec.apptainer.preflight_allow field	config/_types.py::ApptainerSpec	⏳ deferred (out of D1–D4 scope)
AgentCard structured isolation block	a2a/_card.py::project_card	✅ shipped (D3)
sac agents check D4 bind-target warning	cli_pkg/build_cmds.py::check	✅ shipped (D4)
Regression tests	tests/.../test__apptainer_runtime.py + tests/.../a2a/test__card.py + tests/.../cli_pkg/test_build_cmds.py + tests/.../runtimes/test__apptainer_isolation.py + tests/.../runtimes/test__apptainer_preflight.py	✅ shipped

Consequences

Positive.

• sac becomes the only A2A-compatible agent runtime that mechanically enforces isolation by default. Clear differentiation from LangGraph / CrewAI / AutoGen (no isolation concept) and from Docker (isolation exists but not declared / not attestable).

• Clew can build verification chains that attest specific isolation properties without sac-internal introspection.

• The scitex-stats-auditor symlink-confusion class of incidents becomes impossible: the agent can’t “fix” the host because the host isn’t reachable.

Negative / tradeoffs.

• Existing agents that worked under the old defaults will fail-fast if they implicitly relied on $HOME auto-bind. Operators must either declare the bind explicitly OR opt in to relaxed: true.

• The --cleanenv default removes $PATH inheritance; sac has to inject the container’s expected $PATH itself.

• HPC-style “just run my command inside the container with my home visible” is now a two-step (write relaxed: true); intentional.

Addendum: D2 refinement (2026-05-13 evening)

Initial D2 design proposed test ! -d "$HOME" as a universal invariant. Pre-implementation verification against the scitex-stats-auditor spec.yaml exposed an Apptainer-specific edge case: Apptainer creates the entire directory path of every bind target as scaffolding, so a bind like /home/$USER/proj/scitex-stats:/home/$USER/proj/scitex-stats:ro causes /home/$USER (== $HOME) to exist as a directory inside the container — even under --containall, with no credential files visible. The D2 check would false-fire on every agent that mirrors host paths into the container.

Two solutions were considered:

• Bind-aware preflight (sac generates the preflight at runtime, knowing what binds it’s about to create, and the preflight checks $HOME contents are a subset of the declared bind scaffolding). Rejected: dynamic preflight is harder to integrate into Clew’s verification chain — verifier has to attest the generator, not just the executed script.

• Container-canonical paths (bind targets MUST use container- side conventional roots — /srv/, /work/, /opt/, /data/ — never host-mirroring paths). Accepted.

D4. Bind targets MUST be container-canonical paths.

Bind targets that mirror host paths (/home/, /Users/, /root/, absolute Windows-style paths) are deprecated. Bind targets MUST live under conventional container roots: /srv/, /work/, /opt/, /data/.

Spec.yaml convention:

spec:
  apptainer:
    binds:
      - $HOME/proj/scitex-stats:/srv/sources/scitex-stats:ro
      - $HOME/proj/scitex-dev:/srv/sources/scitex-dev:ro

Inside the container nothing under /home/$USER appears, so:

1. The D2 preflight (test ! -d "$HOME") stays static and universal.

2. spec.yaml is operator-agnostic — the same spec runs cleanly for any user.

3. Verification chain receives a static preflight script with a stable sha256; no meta-verification required.

Rationale, beyond the technical fix. Container-canonical paths match Docker / OCI best practice and break the “works on my machine” failure mode (every agent that hardcodes /home/ywatanabe/proj/... is operator-bound). The shared-path convention was an HPC convenience artifact; Clew’s reproducibility context inverts it.

sac-side enforcement (planned). sac agents check <name> will emit a warning when a bind target starts with /home/, /Users/, or /root/. Future strict mode (sac.audit.strict_binds: true) makes it an error.

Convenience. The runtime sets $SAC_WORKDIR=/srv/sources inside the container (when any bind targets land there), so startup_prompts and operator scripts can use cd $SAC_WORKDIR/<pkg> without hardcoding paths.

Order of execution

1. ADR addendum (this section) — done.

2. scitex-stats-auditor spec.yaml — bind targets translated to /srv/sources/...; startup_prompts updated to reference the container paths.

3. Implementation: D1 + D2 (static check) + D3 + D4 (CLI validator).

4. Restart scitex-stats-auditor against hardened sac; verify the static preflight passes end-to-end.

5. Preserve session.jsonl + preflight result for Clew supplementary.

Addendum: D5 — canonical container HOME (2026-05-14)

Live verification of the hardened auditor exposed two issues with D2 as originally specified:

1. Empty-$HOME false-fires. Apptainer scaffolds $HOME from the inherited passwd entry regardless of bind targets (even under --containall), so $HOME is always a directory. The D2 check was relaxed to “$HOME is empty” — workable, but it forces bind targets out of $HOME, breaking operator-intuitive paths.

2. /srv/-style targets force script rewrites. Anything that references ~/proj/X or $HOME/proj/X inside the container breaks.

D5. Canonical container HOME = /home/agent.

sac auto-injects --home /home/agent (skipped only under apptainer.relaxed: true or when the operator declared --home). Inside the container:

• $HOME == /home/agent, operator-independent.

• Bind targets use the canonical HOME: ~/proj/X:/home/agent/proj/X.

• The operator’s actual host home is never scaffolded inside the container.

• sac-base.sif is rebuilt so UID-1000’s passwd entry reads agent (not ubuntu); whoami matches $HOME.

D5 preflight (replaces D2’s empty-$HOME check).

# uid != 0, OR /proc/self/uid_map confirms userns-fakeroot.
if [ "$(id -u)" = "0" ]; then
  awk '$1==0 && $2!=0 {found=1} END {exit !found}' /proc/self/uid_map \
    || exit 11   # real root, refuse
fi
test "$HOME" = "/home/agent" || exit 12  # canonical HOME

Two static lines, attestable by sha256. The “no host leak” property falls out of --containall + canonical --home + declared binds:.

Rationale.

• D4 stays but is no longer the only path. Bind destinations may live under /home/agent/... (intuitive) OR /srv/, /work/, /opt/, /data/ (container-canonical). Both pass D5.

• fakeroot opt-in (apptainer.fakeroot: true) integrates: inside the container id -u == 0, but /proc/self/uid_map proves userns-mapping; the preflight accepts this without weakening the no-real-root guarantee.

Implementation (D5).

Layer	Where	Status
apptainer.fakeroot: bool	config/_types.py::ApptainerSpec	✅
Auto-prepend --home /home/agent	runtimes/_apptainer_iso_flags.py	✅
Auto-append --fakeroot when opted in	same	✅
Preflight rewrite (uid-map + canonical-HOME)	runtimes/_apptainer_preflight.py	✅
sac-base.sif: ubuntu → agent	containers/apptainer-base.def	✅ recipe; SIF rebuild pending
Bind destination validation	config/_parsers/_apptainer.py	✅
Doc updates + AgentCard JSON example	docs/isolation.md	✅

Network isolation addendum (2026-05-14).

Peer review pushed on --network=bridge, surfaced the A2A + MCP-over-loopback interop constraint, and converged on: stay with host netns for the Clew arXiv window; plan a --network=bridge + bridge-IF bind + sac-host /etc/hosts injection migration that preserves MCP URL stability. See docs/isolation.md §4.

References

• docs/isolation.md — the 10-category leak catalog this ADR’s decisions close.

• docs/spec-reference.md — spec.apptainer.relaxed, spec.apptainer.fakeroot.

• The 2026-05-13 scitex-stats-auditor incident — session.jsonl at ~/.scitex/agent-container/runtime/scitex-stats-auditor/ for the full trace.