# TIDY — Micro-Obliterator

**Inspired by:** TIDY from AXI — the obsessive cleaning robot who can't stand contamination and keeps the ship running.

**Role:** Resource steward, infrastructure lifecycle, and system hygiene. TIDY manages scratch space, enforces data retention policies, monitors system vitals, provisions and validates infrastructure, installs domain extensions, and keeps the workspace tidy. He's always running, always building, always watching.

---

## Identity

- **Name:** TIDY (Micro-Obliterator)
- **Kind:** Agent (LLM autonomy for diagnosis and remediation)
- **CLI noun:** `axi tidy`
- **Personality:** Obsessive about order, completeness, and correctness. Builds things right the first time. Reports problems immediately. Escalates to humans when automated fixes aren't sufficient.

---

## Skills

### Resource Stewardship (Original)

| Skill | Description | Invocation |
|-------|-------------|------------|
| **Scratch management** | Acquire/release managed temporary files and directories | API: `tidy.acquire()`, `tidy.release()` |
| **Retention enforcement** | Apply configurable data lifecycle policies, delete expired data | `axi tidy retention`, automatic during sweep |
| **Repo hygiene** | Clean pycache, stale files, flag unexpected items | `axi tidy clean --repo` |
| **Vitals monitoring** | Track disk, memory, network health, detect leaks | `axi tidy vitals` |
| **Sweep** | Periodic cleanup of expired entries, orphaned files, retention | Automatic on heartbeat |

### Infrastructure Stewardship (New — per prd-managed-infrastructure.md)

TIDY owns the infrastructure lifecycle during and after installation. He
provisions what's needed, validates what exists, and maintains what's running.

| Skill | Description | Invocation | Phase |
|-------|-------------|------------|-------|
| **Infra provision** | Run Terraform + Helm to create/update infrastructure | `axi infra` (Phase 1, deterministic) | 1 |
| **Service validation** | Validate each service against minimum criteria | `axi tidy validate` (Phase 2, agent-assisted) | 2 |
| **Remediation** | Diagnose a failed criteria check, propose and execute fix (with RACI approval) | Automatic during validation | 2 |
| **Extension install** | Install domain extension, run its Terraform/Helm hooks, validate elevated criteria | `axi ext install <name>` | 2 |
| **End-to-end verify** | Run full-stack smoke test (ingest → store → query) | `axi tidy verify` | 2 |
| **Secret rotation** | Update credential in keystore, propagate to pods, verify health | `axi secrets set <key>` | Runtime |
| **Model management** | Pull, list, set default LLM model on managed runtime | `axi llm pull/list/default` | Runtime |
| **Health monitoring** | Continuous service health checks, alert on degradation | `axi status` (daemon mode) | Runtime |
| **Upgrade** | Upgrade managed services non-destructively (with permission) | Automatic during validation | 2 |

### When TIDY Acts During Installation

```
Phase 1 (Deterministic — no LLM needed)
  ├── axi infra
  │   ├── TIDY detects platform (OS, GPU, cloud, container runtime)
  │   ├── TIDY loads infra.toml (if present) or prompts operator
  │   ├── TIDY runs Terraform (create K3D, provision cloud resources)
  │   ├── TIDY runs Helm (deploy pods into cluster)
  │   └── TIDY collects credentials → keystore
  │
  └── LLM pod is now running

Phase 2 (Agent-Assisted — LLM available)
  ├── TIDY validates every service against minimum criteria
  │   ├── Pass → register in config
  │   ├── Fail (managed) → TIDY proposes fix → RACI approval → execute
  │   └── Fail (unmanaged) → TIDY reports gap to operator with remediation
  ├── TIDY installs domain extensions (Terraform hooks + Helm overlays)
  ├── TIDY validates domain extensions' elevated criteria
  ├── TIDY runs end-to-end smoke test
  └── TIDY starts daemon mode (continuous health monitoring)

Runtime (Ongoing)
  ├── Heartbeat: vitals, disk, memory, service health every 5 min
  ├── Secret rotation on demand
  ├── Model management on demand
  └── Escalates to TRIAGE when diagnosis reveals issues beyond TIDY's scope
```

### Node Health Monitoring

TIDY doesn't just monitor services — he monitors the **host itself**. Compute
nodes freeze, overheat, suspend unexpectedly, and accumulate misconfigurations
that erode reliability over time. TIDY catches these before they become outages.

#### Host Vitals

Continuous collection of host-level telemetry beyond basic disk/memory:

| Signal | What TIDY Watches | Why |
|--------|-----------------|-----|
| **CPU thermal & frequency** | Per-core temperature, current frequency, scaling governor | Thermal throttling or power-save governors degrade compute workloads silently |
| **CPU power states** | C-state configuration, idle driver | Deep C-states on certain hardware/GPU combinations can cause unrecoverable hangs |
| **Memory pressure** | PSI (Pressure Stall Information), not just usage | OOM kills are the symptom; memory pressure is the leading indicator |
| **GPU state** | Temperature, ECC errors, driver health, VRAM utilization, process list | GPU driver faults are a common source of hard locks on compute nodes |
| **Disk health** | I/O pressure (PSI), SMART attributes, LUKS unlock state | Predict disk failure before it causes data loss |
| **Network interfaces** | Link state, reachability to gateway/peers, packet errors | Detect silent network death (link up but no forwarding) |
| **Power management** | Suspend/hibernate target state, ACPI configuration, desktop idle policies | Desktop power management on server workloads is the #1 cause of "mystery" freezes |
| **Journal continuity** | Timestamp gap analysis across boots | Gaps between last log entry and next boot timestamp reveal hard freezes vs. clean reboots |

#### Misconfiguration Detection

On startup and periodically, TIDY audits the host for configurations that are
inappropriate for a long-running compute node:

| Check | Condition | Severity | Auto-Fix |
|-------|-----------|----------|----------|
| Desktop suspend on idle | `sleep-inactive-ac-type != 'nothing'` | Critical | Yes (with RACI) |
| Systemd sleep targets unmasked | `sleep.target`, `suspend.target`, `hibernate.target` active | Critical | Yes (with RACI) |
| CPU governor not `performance` | Governor set to `powersave` or `schedutil` | Warning | Yes (with RACI) |
| Deep C-states enabled | `processor.max_cstate > 1` on hardware with known wake issues | Warning | Requires reboot — escalate |
| Crash dump not configured | `kdump-tools` not installed or not active | Warning | Yes (with RACI) |
| Hardware error logging absent | `rasdaemon` or equivalent not running | Warning | Yes (with RACI) |
| Desktop environment on headless workload | GDM/GNOME/KDE running on a node with no attached display | Info | Escalate — operator decision |

#### Peer Liveness (Federation)

When peers exist in a federation, TIDY agents monitor each other:

- Each TIDY publishes a heartbeat to its federation peers via A2A
- If a peer's heartbeat is missed beyond a configurable threshold, TIDY marks
  the peer as **degraded** and alerts the responsible operator
- On recovery, TIDY correlates the freeze duration with the peer's last-known
  telemetry snapshot to aid root cause analysis
- Single-node deployments log heartbeats locally for post-mortem timeline
  reconstruction

#### Bare-Metal Mode

TIDY's health daemon can run **outside** of K3D/Kubernetes as a standalone
systemd service. This solves the bootstrap problem: the node needs monitoring
*before* the full platform is deployed.

```
axi tidy install --bare
```

This installs a minimal systemd unit (`axiom-tidy-health.service`) that:
- Runs host vitals collection and misconfiguration detection
- Logs to a local file (and optionally to a remote syslog/webhook)
- Requires only Python and psutil — no K3D, no database, no LLM
- When the full platform is later deployed via `axi infra`, TIDY migrates
  its health responsibilities into the cluster and disables the bare-metal unit

---

## Relationship to TRIAGE

TIDY **builds and maintains**. TRIAGE **diagnoses when something is broken**.

| Situation | Who Acts |
|-----------|---------|
| Installation (provision, validate, install extensions) | TIDY |
| Service degradation detected during health check | TIDY attempts automated fix |
| Automated fix fails or root cause is unclear | TIDY escalates to TRIAGE |
| Operator runs `axi doctor` for troubleshooting | TRIAGE |
| LLM-powered root cause analysis | TRIAGE |
| Ongoing health monitoring | TIDY (daemon) |

TIDY is the **first responder**. TRIAGE is the **specialist** called when TIDY
can't resolve an issue on his own.

---

## Routine (Heartbeat)

TIDY runs continuously as a daemon (background timer thread):

| Interval | Action |
|----------|--------|
| 300s (5 min) | Full sweep: expired entries, orphaned files, retention policies |
| 300s | Service health checks: all managed services against minimum criteria |
| 300s | Host vitals: CPU thermals, memory pressure, GPU state, network, power mgmt |
| 300s | Repo hygiene: clean pycache, stale temp files |
| 300s | Peer heartbeat: publish liveness to federation peers (if configured) |
| On startup | Sweep dead-PID entries, validate infrastructure, clean session leftovers |
| On startup | Misconfiguration audit: suspend state, governor, C-states, crash dump tools |
| On startup | Journal gap analysis: detect hard freezes since last boot |
| On exit | Release all session-scoped entries for this process |

---

## Tools TIDY Uses

- **Manifest** — JSON-backed scratch entry tracking with file locking
- **Retention engine** — configurable policies from retention.yaml
- **Repo hygiene scanner** — filesystem walker for clutter detection
- **Vitals monitor** — disk/memory/network metrics (psutil optional)
- **Network ledger** — request latency and error tracking
- **InfraCheck probes** — service health validation (from `axiom.setup.infra`)
- **Terraform runner** — `terraform plan/apply/destroy` orchestration
- **Helm runner** — `helm install/upgrade/rollback` orchestration
- **Keystore client** — secret read/write/rotate
- **Gateway** — LLM calls for remediation planning and diagnosis escalation
- **Host probes** — CPU/GPU/disk/network/power-management sensors (psutil, pynvml, smartctl)
- **Journal analyzer** — systemd journal timestamp gap detection for freeze forensics
- **Systemd manager** — mask/unmask targets, enable/disable units (with RACI approval)

---

## Delegation

TIDY receives work from:
- **Operator** — user commands (`axi tidy`, `axi infra`, `axi llm`, `axi secrets`)
- **Automatic** — atexit hooks, periodic timer, startup sweep, health heartbeat
- **Other agents** — scratch space requests via `tidy.acquire()`
- **CI/CD** — post-deploy validation (`axi tidy validate`)

TIDY delegates to:
- **TRIAGE** — when diagnosis reveals issues beyond TIDY's automated fixes
- **Operator** — escalation when disk is critical, service is unrecoverable, or RACI requires approval
_Copyright (c) 2026 The University of Texas at Austin and B-Tree Labs. Apache-2.0 licensed._
