# Prague Deployment Runbook

**Status:** Operational playbook — pre-flight for summer 2026 course
**Target date:** 2026-07 course start
**Author:** Benjamin Booth (with Claude Code)
**Last updated:** 2026-04-20

Consolidates the infrastructure work needed to run the Prague
classroom end-to-end. Pairs with the companion runbooks:

- [langfuse-setup.md](langfuse-setup.md) — observability
- [canvas-setup.md](canvas-setup.md) — LMS integration
- [design-instructor-prep-workflow.md](design-instructor-prep-workflow.md) — instructor UX

Memory context:
- `project_prague_deployment_requirements` — scope, cohort, EC gating
- `project_infra_notes_staging_production` — raw checklist (this doc supersedes)

---

## 0) Decisions still open (block progress)

| # | Decision | Owner | Needed by |
|---|---|---|---|
| D1 | Prague EU node hosting — UT-owned VM vs partner institution | Ben | 2026-05 |
| D2 | GDPR data-residency — does EU student trace data stay in EU? | Ben + UT Legal | 2026-05 |
| D3 | Part 810 review — which course modules are EC-gated? | Ben (instructor) | 2026-06 |
| D4 | Foreign-national list — which Prague students are non-US? | Ben + registrar | 2026-06 |

D1 + D2 together decide the LangFuse topology:
- Single LangFuse on Rascal (D1 any, D2 permits US storage) — simplest.
- Dual LangFuse (D2 requires EU residency) — one per node, no cross-flow.

D3 + D4 decide the nationality gating rollout cadence.

---

## 1) Rascal (Austin hub)

Status: operational at v0.10.10.

### 1.1 LangFuse self-hosted

```bash
# On Rascal
git clone https://github.com/langfuse/langfuse.git /opt/langfuse
cd /opt/langfuse
docker compose up -d
# Confirm: curl http://localhost:3000
```

In the LangFuse UI → create project → copy public + secret keys.
Then on every Axiom node:

```bash
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_HOST="https://langfuse.rascal.internal:3000"
```

Axiom auto-wires. Verify with the one-liner in
[langfuse-setup.md](langfuse-setup.md).

### 1.2 Open WebUI

```bash
docker run -d --name open-webui \
    -p 3001:8080 \
    -v open-webui-data:/app/backend/data \
    --add-host=host.docker.internal:host-gateway \
    ghcr.io/open-webui/open-webui:main
```

Configure the Axiom-gateway endpoint as the OpenAI-compatible
backend. Create an instructor admin account. Student accounts
are provisioned by `axi classroom create` (see FW-1 P5).

### 1.3 Cloudflare Tunnel

```bash
cloudflared tunnel login
cloudflared tunnel create rascal-classroom
# In /etc/cloudflared/config.yml:
#   tunnel: rascal-classroom
#   credentials-file: ...
#   ingress:
#     - hostname: classroom.axiom.example.edu
#       service: http://localhost:3001
#     - service: http_status:404
cloudflared tunnel route dns rascal-classroom classroom.axiom.example.edu
sudo systemctl enable --now cloudflared
```

CF handles geo-routing automatically once the Prague node is
registered against the same hostname.

### 1.4 Canvas credentials

Request developer key from UT IT Canvas admin with scopes listed in
[canvas-setup.md](canvas-setup.md). Set env vars on Rascal:

```bash
export AXIOM_CANVAS_API_URL="https://utexas.instructure.com"
export AXIOM_CANVAS_API_TOKEN="..."
```

---

## 2) Prague EU Node (new — provisioning)

Depends on D1.

### 2.1 VM provisioning (UT-owned path)

Hetzner is the cheapest compliant EU option:

- **Shape:** CPX31 (4 vCPU, 8GB RAM) — upgradeable to CPX41 if
  Bonsai inference needs more headroom
- **Region:** Falkenstein (DE, EU)
- **OS:** Ubuntu 24.04 LTS
- **Disk:** 100GB SSD

Install docker, axi, create a federation identity, bind to Rascal:

```bash
curl -fsSL https://get.docker.com | sh
usermod -aG docker $USER
python3 -m venv /opt/axiom-venv
/opt/axiom-venv/bin/pip install axi-platform
/opt/axiom-venv/bin/axi install-shim
/opt/axiom-venv/bin/axi federation init --owner "ben@utexas.edu" \
    --name prague --profile provider
# Add Rascal as a peer + bind identity
/opt/axiom-venv/bin/axi nodes add rascal ben@rascal.example.edu
```

Then from Rascal:

```bash
axi federation invite --ttl 24
# → copy the token
# → on Prague:
axi federation join <token> --confirm
```

### 2.2 Per-node env

```bash
# On Prague
export LANGFUSE_PUBLIC_KEY=...
export LANGFUSE_SECRET_KEY=...
export LANGFUSE_HOST=https://langfuse.rascal.example.edu  # or local EU instance
export AXIOM_CANVAS_API_URL=https://utexas.instructure.com
export AXIOM_CANVAS_API_TOKEN=...
```

### 2.3 Open WebUI on Prague

Same docker run as Rascal (step 1.2). Register the Prague hostname
with the Cloudflare tunnel so CF geo-routes EU students here
automatically.

### 2.4 Corpus replication (Austin → Prague)

Once joined, Prague pulls classroom knowledge packs from Rascal's
cohort registry on the default 60s cadence (ADR-023). No manual
step — verify with:

```bash
axi federation status
# → Prague should show as peer with verified=true
axi knowledge status
# → corpus doc count should converge with Rascal's
```

EC-gated packs replicate ONLY if Prague is same-org (UT-owned VM,
D1=a). If partner-hosted (D1=b), update the federation policy to
exclude EC packs.

---

## 3) Federation validation

Before opening to students, run the federation lifecycle harness in
a classroom-specific scenario:

```bash
cd axiom
pytest tests/federation_lifecycle/ -m federation_lifecycle
# → should still be green (7 scenarios, see v0.10.10 checkpoint)
```

Prague-specific scenario (to add, not yet written): hub-spoke with
simulated Austin↔Prague latency + round-trip of a promoted finding.
Tracked as follow-up work post-T1c.

---

## 4) Go-live checklist

Day-of (2026-07-01 or whenever course starts):

- [ ] Rascal health dashboard green (`axi status`)
- [ ] Prague node federated + verified (`axi federation status`)
- [ ] LangFuse receiving traces from both nodes (check a test turn)
- [ ] Canvas roster synced (`axi classroom prep lms-setup canvas-configure ...`)
- [ ] Demo classroom runs clean on both nodes
- [ ] Published real classroom (`axi classroom publish <id> --approver ...`)
- [ ] Students can hit `classroom.axiom.example.edu` from both
      continents and see the right node's latency profile
- [ ] Student onboarding rail (P3b) captures consent before first chat
- [ ] Instructor brief (CHALKE) shows signals during the first
      check-in window

---

## 5) Rollback plan

If a problem surfaces post-launch:

- **LangFuse trace volume crushes Rascal** → set
  ``AXIOM_TRACE_BACKEND=null`` on the Prague node and restart. Trace
  emission stops; chat path is unaffected (per the defensive
  defaults shipped in T1a).
- **Canvas API rate limit** → set ``AXIOM_LMS_PROVIDER=none`` on both
  nodes, switch to manual ``students.yaml`` for the rest of the
  course. Re-enable when UT IT raises the limit.
- **Prague node unreachable** → CF tunnel health check fails over to
  Rascal automatically; EU students see higher latency but
  everything works. Restart Prague VM.
- **Federation drift (Prague's corpus falls behind)** → cadence is
  pull-based, so `axi federation status` should self-heal within
  60s. If it doesn't, `axi federation leave --confirm` + re-join
  from invite to reset.

---

## 6) Post-course wind-down

See [FW-4 course-conclusion workflow](design-instructor-prep-workflow.md)
once it ships — covers harvest, archive, grade finalization.

---

## Appendix A — Env-var quick reference

```bash
# Tracing (T1a)
LANGFUSE_PUBLIC_KEY=pk-...
LANGFUSE_SECRET_KEY=sk-...
LANGFUSE_HOST=https://langfuse.rascal.internal:3000
AXIOM_TRACE_BACKEND=null|in_memory|langfuse  # explicit override

# LMS (T1b)
AXIOM_LMS_PROVIDER=canvas|none
AXIOM_LMS_NAME=canvas-utexas
AXIOM_CANVAS_API_URL=https://utexas.instructure.com
AXIOM_CANVAS_API_TOKEN=...

# Runtime paths
AXIOM_RUNTIME_ROOT=/var/axiom/runtime  # per-node state
AXIOM_ROOT=/opt/axiom-src               # source checkout (dev)
```

All are auto-detected with defensive defaults. A missing variable,
an empty variable, or a typo in a variable name never takes the
node down — worst case the feature it guards becomes a no-op until
corrected.
_Copyright (c) 2026 The University of Texas at Austin and B-Tree Labs. Apache-2.0 licensed._
