---
phase: 04-installation-fleet-status
plan: 03
type: execute
wave: 2
depends_on: [04-01]
files_modified:
  - src/clawrium/core/hosts.py
  - src/clawrium/core/install.py
  - src/clawrium/core/health.py
  - tests/test_hosts.py
  - tests/test_health.py
autonomous: true
requirements: [INST-04, STAT-01]

must_haves:
  truths:
    - "Install state is tracked in host records (installed claws, status)"
    - "Partial/failed installations are recorded and visible"
    - "Health check determines if a claw process is running on host"
    - "Health check uses SSH to query process state, not cached data"
  artifacts:
    - path: "src/clawrium/core/hosts.py"
      provides: "Extended host records with claw installation tracking"
      contains: "claws"
    - path: "src/clawrium/core/health.py"
      provides: "Live health checking via SSH"
      exports: ["check_claw_health", "ClawStatus"]
    - path: "tests/test_health.py"
      provides: "Health check tests"
      min_lines: 40
  key_links:
    - from: "src/clawrium/core/install.py"
      to: "src/clawrium/core/hosts.py"
      via: "update_host for install state"
      pattern: "update_host.*claws"
    - from: "src/clawrium/core/health.py"
      to: "ansible_runner"
      via: "remote process check"
      pattern: "ansible_runner.run"
---

<objective>
Extend host records to track installed claws with their state, and add live health checking to determine if claw processes are running. This completes INST-04 (error tracking) and provides foundation for STAT-01 (fleet status).

Purpose: Track installation state for error visibility and enable live health checks for status display.
Output: Updated hosts.py with claw tracking, new health.py module for process checking.
</objective>

<execution_context>
@/home/devashish/workspace/ric03uec/clawrium/.claude/get-shit-done/workflows/execute-plan.md
@/home/devashish/workspace/ric03uec/clawrium/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/04-installation-fleet-status/04-CONTEXT.md

<interfaces>
<!-- From existing hosts.py -->
```python
def load_hosts() -> list[dict]
def save_hosts(hosts: list[dict]) -> None
def update_host(hostname: str, updater: Callable[[dict], dict]) -> bool
def get_host(identifier: str) -> dict | None

# Current host record structure:
{
    "hostname": "192.168.1.100",
    "key_id": "myhost",
    "port": 22,
    "user": "xclm",
    "auth_method": "key",
    "hardware": {...},
    "metadata": {"added_at": "...", "last_seen": "...", "tags": []}
}
```

<!-- From install.py (Plan 01) -->
```python
class InstallResult(TypedDict):
    success: bool
    claw: str
    version: str
    host: str
    playbooks_run: list[str]
    error: str | None
```

<!-- From hardware.py for ansible pattern -->
```python
def gather_hardware(hostname: str, user: str, port: int, ssh_key: str | None) -> HardwareInfo
# Uses ansible_runner.run with module="shell" for command execution
```
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add claw installation tracking to host records</name>
  <files>src/clawrium/core/hosts.py, src/clawrium/core/install.py, tests/test_hosts.py</files>
  <read_first>
    - src/clawrium/core/hosts.py (current host record structure, update_host function)
    - src/clawrium/core/install.py (run_installation function to modify)
    - tests/test_hosts.py (existing test patterns)
  </read_first>
  <behavior>
    - Test: After successful install, host record contains claws[claw_name] with status="installed"
    - Test: After failed install, host record contains claws[claw_name] with status="failed" and error message
    - Test: install.py updates host record on success via update_host
    - Test: install.py updates host record on failure via update_host
  </behavior>
  <action>
Update `src/clawrium/core/hosts.py` to document the extended host schema (no code changes needed - update_host handles arbitrary dict updates):

Add docstring update to document the claw tracking schema:
```python
# Host record schema (extended):
# {
#     "hostname": str,
#     "claws": {
#         "openclaw": {
#             "version": "0.1.0",
#             "status": "installed" | "failed" | "installing",
#             "installed_at": "ISO timestamp",
#             "error": str | None,
#             "user": "opc-hostname"  # per D-07
#         }
#     },
#     ...existing fields...
# }
```

Update `src/clawrium/core/install.py` run_installation to track state:

1. Before running playbooks, update host with status="installing":
```python
def set_installing(h: dict) -> dict:
    if "claws" not in h:
        h["claws"] = {}
    h["claws"][claw_name] = {
        "version": matched_version,
        "status": "installing",
        "installed_at": None,
        "error": None,
        "user": f"opc-{h['hostname'].split('.')[0]}"  # Simple hostname
    }
    return h

update_host(host["hostname"], set_installing)
```

2. On success, update status to "installed" with timestamp:
```python
from datetime import datetime, timezone

def set_installed(h: dict) -> dict:
    h["claws"][claw_name]["status"] = "installed"
    h["claws"][claw_name]["installed_at"] = datetime.now(timezone.utc).isoformat()
    return h

update_host(host["hostname"], set_installed)
```

3. On failure (in except block), update status to "failed" with error:
```python
def set_failed(h: dict) -> dict:
    if "claws" not in h:
        h["claws"] = {}
    if claw_name not in h["claws"]:
        h["claws"][claw_name] = {"version": matched_version, "user": None}
    h["claws"][claw_name]["status"] = "failed"
    h["claws"][claw_name]["error"] = str(e)
    h["claws"][claw_name]["installed_at"] = datetime.now(timezone.utc).isoformat()
    return h

update_host(host["hostname"], set_failed)
```

Add tests to `tests/test_hosts.py`:
1. test_host_claw_tracking_installed - verify claws dict structure after install
2. test_host_claw_tracking_failed - verify error is recorded on failure

Update `tests/test_install.py` to verify state tracking:
1. test_install_updates_host_on_success - check update_host called with installed status
2. test_install_updates_host_on_failure - check update_host called with failed status
  </action>
  <verify>
    <automated>cd /home/devashish/workspace/ric03uec/clawrium && python -m pytest tests/test_hosts.py tests/test_install.py -v -k "claw_tracking or updates_host" --tb=short 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - src/clawrium/core/install.py contains "update_host" calls for state tracking
    - install.py contains status assignment: "installing", "installed", "failed"
    - install.py contains "claws" dict manipulation in update callbacks
    - install.py imports datetime for timestamp
    - tests/test_hosts.py contains test_host_claw_tracking tests (or tests in test_install.py)
    - Tests verify claws dict contains version, status, installed_at, error fields
  </acceptance_criteria>
  <done>Host records track claw installation state with success/failure status</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: Create health check module for live claw status</name>
  <files>src/clawrium/core/health.py, tests/test_health.py</files>
  <read_first>
    - src/clawrium/core/hardware.py (ansible_runner pattern for remote shell commands)
    - src/clawrium/core/hosts.py (host record structure with claws field)
    - src/clawrium/core/keys.py (get_host_private_key for SSH key)
  </read_first>
  <behavior>
    - Test: check_claw_health returns "running" if process found
    - Test: check_claw_health returns "stopped" if process not found
    - Test: check_claw_health returns "unknown" if SSH fails
    - Test: check_claw_health uses SSH to query remote host (not cached)
  </behavior>
  <action>
Create `src/clawrium/core/health.py`:

```python
"""Live health checking for claw instances.

This module provides functions to check if claw processes are running
on remote hosts via SSH. Per D-13, this performs live checks, not cached data.
"""

import logging
import os
import tempfile
from enum import Enum
from typing import TypedDict

import ansible_runner

from clawrium.core.keys import get_host_private_key

logger = logging.getLogger(__name__)


class ClawStatus(str, Enum):
    """Status of a claw instance."""
    RUNNING = "running"
    STOPPED = "stopped"
    UNKNOWN = "unknown"
    NOT_INSTALLED = "not_installed"


class HealthResult(TypedDict):
    """Result of health check for a claw on a host."""
    claw: str
    host: str
    status: ClawStatus
    user: str | None
    error: str | None


def check_claw_health(
    claw_name: str,
    host: dict,
) -> HealthResult:
    """Check if a claw process is running on a host.

    Performs live SSH check per D-13. Does not use cached data.

    Args:
        claw_name: Name of claw to check (e.g., "openclaw")
        host: Host record dict with hostname, port, user, key_id, claws

    Returns:
        HealthResult with status and any error message
    """
    hostname = host["hostname"]
    port = host.get("port", 22)
    user = host.get("user", "xclm")

    # Get claw record from host
    claws = host.get("claws", {})
    claw_record = claws.get(claw_name)

    if not claw_record:
        return {
            "claw": claw_name,
            "host": hostname,
            "status": ClawStatus.NOT_INSTALLED,
            "user": None,
            "error": None,
        }

    claw_user = claw_record.get("user")
    if not claw_user:
        # No user set - can't check
        return {
            "claw": claw_name,
            "host": hostname,
            "status": ClawStatus.UNKNOWN,
            "user": None,
            "error": "No claw user recorded",
        }

    # Get SSH key
    key_id = host.get("key_id") or hostname
    ssh_key = get_host_private_key(key_id)
    if not ssh_key:
        return {
            "claw": claw_name,
            "host": hostname,
            "status": ClawStatus.UNKNOWN,
            "user": claw_user,
            "error": "SSH key not found",
        }

    # Build inventory
    inventory = {
        "all": {
            "hosts": {
                hostname: {
                    "ansible_user": user,
                    "ansible_port": port,
                    "ansible_ssh_private_key_file": str(ssh_key),
                }
            }
        }
    }

    # Check for node process owned by claw user
    # OpenClaw runs as a Node.js process
    check_cmd = f"pgrep -u {claw_user} node >/dev/null 2>&1 && echo RUNNING || echo STOPPED"

    with tempfile.TemporaryDirectory() as tmpdir:
        os.chmod(tmpdir, 0o700)

        result = ansible_runner.run(
            private_data_dir=tmpdir,
            inventory=inventory,
            host_pattern=hostname,
            module="shell",
            module_args=check_cmd,
            quiet=True,
            timeout=15,
        )

        if result.status == "timeout":
            return {
                "claw": claw_name,
                "host": hostname,
                "status": ClawStatus.UNKNOWN,
                "user": claw_user,
                "error": "Health check timed out",
            }

        if result.status != "successful":
            return {
                "claw": claw_name,
                "host": hostname,
                "status": ClawStatus.UNKNOWN,
                "user": claw_user,
                "error": f"SSH failed: {result.status}",
            }

        # Parse output from events
        output = ""
        for event in result.events:
            if event.get("event") == "runner_on_ok":
                output = event.get("event_data", {}).get("res", {}).get("stdout", "")
                break

        if "RUNNING" in output:
            return {
                "claw": claw_name,
                "host": hostname,
                "status": ClawStatus.RUNNING,
                "user": claw_user,
                "error": None,
            }
        else:
            return {
                "claw": claw_name,
                "host": hostname,
                "status": ClawStatus.STOPPED,
                "user": claw_user,
                "error": None,
            }


def check_all_claws_on_host(host: dict) -> list[HealthResult]:
    """Check health of all installed claws on a host.

    Args:
        host: Host record dict

    Returns:
        List of HealthResult for each installed claw
    """
    results = []
    claws = host.get("claws", {})

    for claw_name in claws:
        result = check_claw_health(claw_name, host)
        results.append(result)

    return results
```

Create `tests/test_health.py`:

```python
"""Tests for claw health checking."""

import pytest
from unittest.mock import patch, MagicMock

from clawrium.core.health import (
    check_claw_health,
    check_all_claws_on_host,
    ClawStatus,
)


@pytest.fixture
def mock_host():
    """Host record with installed claw."""
    return {
        "hostname": "192.168.1.100",
        "port": 22,
        "user": "xclm",
        "key_id": "testhost",
        "claws": {
            "openclaw": {
                "version": "0.1.0",
                "status": "installed",
                "user": "opc-testhost",
            }
        },
    }


def test_health_check_running(mock_host):
    """Process running returns RUNNING status."""
    mock_runner = MagicMock()
    mock_runner.status = "successful"
    mock_runner.events = [
        {"event": "runner_on_ok", "event_data": {"res": {"stdout": "RUNNING"}}}
    ]

    with patch("clawrium.core.health.get_host_private_key", return_value="/fake/key"):
        with patch("clawrium.core.health.ansible_runner.run", return_value=mock_runner):
            result = check_claw_health("openclaw", mock_host)

    assert result["status"] == ClawStatus.RUNNING
    assert result["claw"] == "openclaw"
    assert result["user"] == "opc-testhost"
    assert result["error"] is None


def test_health_check_stopped(mock_host):
    """Process not running returns STOPPED status."""
    mock_runner = MagicMock()
    mock_runner.status = "successful"
    mock_runner.events = [
        {"event": "runner_on_ok", "event_data": {"res": {"stdout": "STOPPED"}}}
    ]

    with patch("clawrium.core.health.get_host_private_key", return_value="/fake/key"):
        with patch("clawrium.core.health.ansible_runner.run", return_value=mock_runner):
            result = check_claw_health("openclaw", mock_host)

    assert result["status"] == ClawStatus.STOPPED


def test_health_check_ssh_fails(mock_host):
    """SSH failure returns UNKNOWN status with error."""
    mock_runner = MagicMock()
    mock_runner.status = "failed"
    mock_runner.events = []

    with patch("clawrium.core.health.get_host_private_key", return_value="/fake/key"):
        with patch("clawrium.core.health.ansible_runner.run", return_value=mock_runner):
            result = check_claw_health("openclaw", mock_host)

    assert result["status"] == ClawStatus.UNKNOWN
    assert "SSH failed" in result["error"]


def test_health_check_not_installed(mock_host):
    """Claw not in host record returns NOT_INSTALLED."""
    result = check_claw_health("zeroclaw", mock_host)

    assert result["status"] == ClawStatus.NOT_INSTALLED


def test_health_check_no_ssh_key(mock_host):
    """Missing SSH key returns UNKNOWN."""
    with patch("clawrium.core.health.get_host_private_key", return_value=None):
        result = check_claw_health("openclaw", mock_host)

    assert result["status"] == ClawStatus.UNKNOWN
    assert "SSH key not found" in result["error"]


def test_health_check_timeout(mock_host):
    """Timeout returns UNKNOWN status."""
    mock_runner = MagicMock()
    mock_runner.status = "timeout"
    mock_runner.events = []

    with patch("clawrium.core.health.get_host_private_key", return_value="/fake/key"):
        with patch("clawrium.core.health.ansible_runner.run", return_value=mock_runner):
            result = check_claw_health("openclaw", mock_host)

    assert result["status"] == ClawStatus.UNKNOWN
    assert "timed out" in result["error"]


def test_check_all_claws_on_host(mock_host):
    """check_all_claws_on_host returns results for each claw."""
    mock_runner = MagicMock()
    mock_runner.status = "successful"
    mock_runner.events = [
        {"event": "runner_on_ok", "event_data": {"res": {"stdout": "RUNNING"}}}
    ]

    with patch("clawrium.core.health.get_host_private_key", return_value="/fake/key"):
        with patch("clawrium.core.health.ansible_runner.run", return_value=mock_runner):
            results = check_all_claws_on_host(mock_host)

    assert len(results) == 1
    assert results[0]["claw"] == "openclaw"
    assert results[0]["status"] == ClawStatus.RUNNING
```
  </action>
  <verify>
    <automated>cd /home/devashish/workspace/ric03uec/clawrium && python -m pytest tests/test_health.py -v --tb=short 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - src/clawrium/core/health.py exists
    - health.py contains "class ClawStatus" enum with RUNNING, STOPPED, UNKNOWN, NOT_INSTALLED
    - health.py contains "def check_claw_health"
    - health.py contains "def check_all_claws_on_host"
    - health.py contains "ansible_runner.run" for SSH execution
    - health.py contains "pgrep" command for process detection
    - tests/test_health.py exists with at least 6 test functions
    - All tests in test_health.py pass
  </acceptance_criteria>
  <done>Health check module with live SSH-based process status verification</done>
</task>

</tasks>

<verification>
- Host records contain claws dict after installation
- Failed installations have status="failed" with error message
- check_claw_health performs live SSH check
- Health check returns correct status for running/stopped/unknown states
- All new tests pass
</verification>

<success_criteria>
- Installation state tracked in host records (D-11)
- Partial/failed states visible via claws dict
- Live health check via SSH (D-13)
- Tests cover all health check scenarios
</success_criteria>

<output>
After completion, create `.planning/phases/04-installation-fleet-status/04-03-SUMMARY.md`
</output>
