Metadata-Version: 2.4
Name: hpc-claw
Version: 0.2.2
Summary: HPC-specific intent modeling and policy-verifiable resource orchestration built on Agentivium Core.
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: agentivium-core==0.5.0
Requires-Dist: httpx>=0.27
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Requires-Dist: tomli>=2.0
Requires-Dist: typer>=0.12
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Description-Content-Type: text/markdown

# HPC-Claw

HPC-Claw is a user-space, pre-submission decision-support and verification layer for HPC scheduling. It normalizes structured or natural-language intent, captures time-bounded Slurm and policy snapshots, generates candidate decisions, builds evidence, and asks Agentivium Core's generic pipeline to classify relation and admissibility before approval or execution.

HPC-Claw is the first domain extension of `agentivium-core`. Agentivium Core defines the contracts: conversation state, context packs, memory, retrieval, clarification, intent IRs, parsers, LLM clients, policy validation results, planners, tool adapters, and execution traces. HPC-Claw implements HPC-specific behavior on top of those contracts.

HPC-Claw is not a scheduler replacement and does not modify `slurmctld`. Dry-run is the default. Explicit submission is protected by verification freshness, admissibility, script integrity, authorization, approval, an available audit sink, and a final exact-command confirmation.

## Installation

```bash
pip install -e ".[dev]"
```

The package targets Python 3.10 or newer and targets the public
`agentivium-core==0.5.0` contracts distributed through PyPI.

## Release Automation

Every push to `main` is treated as a release commit. The GitHub Actions pipeline
validates Python 3.10 through 3.13, builds and checks the wheel and source archive,
creates the `v<project.version>` GitHub Release, and then publishes the same
artifacts to PyPI through the `pypi` Trusted Publisher environment.

PyPI versions are immutable, so `project.version` in `pyproject.toml` must be
increased before every new commit pushed to `main`. A reused version fails before
publishing when its GitHub Release belongs to another commit.

## Conversational Shell

Normal use now starts from the application home and current workspace:

```bash
hpc-claw
hpc-claw start
hpc-claw submit
```

The first two commands initialize or migrate `~/.hpc-claw`, detect static HPC
capabilities, confirm workspace trust, create a persistent thread/run, and open
the terminal conversation. `hpc-claw submit` opens the submit-oriented shell in
dry-run mode. The shell exposes `/status`, `/context`, `/skills`, `/thread`,
`/new`, `/trace`, and `/exit`; it never invokes the CLI as a subprocess.

Global application state is owned by `~/.hpc-claw/`:

```text
config.toml        version.json       providers/       credentials/
clusters/          threads/           runs/            memory/
skills/            audit/             cache/           logs/
migrations/backups/
```

The home is created with mode `0700`; protected files use `0600`. Migrations are
versioned, backed up before mutation, and preserve legacy state for review.
Provider configuration is separate from credentials, and thread/run persistence
redacts secret-like fields and bearer or URL credentials.

Useful read-only diagnostics remain available without using the conversation:

```bash
hpc-claw detect --json
hpc-claw doctor --json
hpc-claw inspect ./job.sbatch --json
```

## Project-local Workspace Setup

Initialize a hidden assistant workspace before regular use:

```bash
hpc-claw init --role hpc_user \
  --llm-base-url http://localhost:11434/v1 \
  --llm-model qwen2.5:7b
```

For an administrator profile:

```bash
hpc-claw init --role admin \
  --llm-base-url https://api.openai.com/v1 \
  --llm-model gpt-4.1-mini \
  --llm-api-key-env OPENAI_API_KEY
```

This backward-compatible command creates `<workspace>/.hpc-claw/` as optional
project-local configuration. It is separate from the global application home and
contains:

- `config.yaml` for the active role, LLM endpoint, default policy, and paths
- `roles/` for admin or HPC-user permissions
- `policies/` with the provenance-aware Slurm policy copied into the workspace
- `prompts/` and `templates/` for internal assistant instructions and Slurm templates
- `contexts/`, `evidence/`, `cases/`, `approvals/`, `audit/`, `memory/`,
  `runs/`, `traces/`, `artifacts/`, and `logs/` for runtime state

Use the generated LLM config directly:

```bash
hpc-claw parse examples/requests/mpi_test.txt \
  --llm-config .hpc-claw/config.yaml \
  --out intent.json
```

## Verifiable Scheduling Path

Start with canonical YAML or JSON; natural-language parsing remains an optional,
untrusted adapter:

```bash
hpc-claw intent validate examples/intents/cpu_gpu_fallback.yaml --json
hpc-claw context collect --scheduler slurm --out .hpc-claw/contexts/latest.json
hpc-claw policy snapshot hpc_claw/policy/profiles/verifiable_slurm_v0_1.yaml \
  --out .hpc-claw/policies/site-policy-snapshot.json
hpc-claw recommend examples/intents/cpu_gpu_fallback.yaml \
  --context .hpc-claw/contexts/latest.json \
  --policy-snapshot .hpc-claw/policies/site-policy-snapshot.json \
  --dry-run --json
```

The dry-run result contains normalized intent, context/policy IDs and timestamps,
candidate decisions, observed/predicted/derived evidence, relation, admissibility,
blocking checks, deviations, re-verification deadline, and next action. It returns
the admissible set; the verifier does not choose a globally "best" decision.

The formal pipeline is:

```text
Structured intent
  -> Core Intent
  -> Slurm ContextSnapshot + PolicySnapshot
  -> rule and bounded-enumeration candidate Decisions
  -> HPC EvidenceBundle
  -> Core VerificationPipeline + HPC providers
  -> relation and admissible decision set
  -> explicit approval when required
  -> dry-run script or guarded sbatch
  -> append-only audit events
```

Machine-readable commands use `--json`. Operational exit codes are: `0` accepted,
`1` rejected/inadmissible, `2` clarification required, `3` re-verification
required, and `4` scheduler infrastructure or partial-observability failure.

## Evidence and relation semantics

HPC-Claw keeps scheduler validity, resource feasibility, current availability,
policy compliance, deadline feasibility, preference alignment, admissibility,
approval, and optimality separate. The Core relation is one of `Exact`,
`PermittedRelaxation`, `JustifiedDeviation`, `UnjustifiedDeviation`, `Violation`,
or operational `Undetermined`. Explanations never substitute for structured,
traceable evidence.

Every observed source records command, timestamp, duration, and status. Missing,
permission-denied, timed-out, unparseable, or unsupported sources stay explicit.
Queue and policy facts have independent freshness bounds and stale decisions must
be rebuilt and reverified before execution.

## Research package

`research/verifiable_scheduling/` is independent of production verification. It
contains a separate exhaustive bounded oracle, 100 provisional gold cases queued
for hand review, 1,000 balanced
generated cases, 200 challenge cases, 3,600 evidence-fault cases, A0-A5
ablations, trace replay, three deterministic decisors, and frozen metrics. The
oracle does not import production predicates. Generate the versioned fixtures via:

```bash
python -m research.verifiable_scheduling.scenarios.generate
```

No claim of global schedule optimality or improved cluster utilization is made.

## Basic CLI Usage

Parse a request with a mock LLM response:

```bash
hpc-claw parse examples/requests/mpi_test.txt \
  --mock-response examples/outputs/mpi_test_ir.json \
  --out intent.json
```

Validate the intent:

```bash
hpc-claw validate intent.json \
  --policy hpc_claw/policy/profiles/hcmut_slurm_v1.yaml
```

Generate a Slurm script:

```bash
hpc-claw generate slurm intent.json --out job.sbatch
```

Run the full parse, validate, plan, and generate path:

```bash
hpc-claw run examples/requests/mpi_test.txt \
  --policy hpc_claw/policy/profiles/hcmut_slurm_v1.yaml \
  --mock-response examples/outputs/mpi_test_ir.json \
  --out job.sbatch \
  --trace-out trace.json
```

Expected script:

```bash
#!/bin/bash
#SBATCH --job-name=mpi_test
#SBATCH --account=school_g001
#SBATCH --partition=cpu
#SBATCH --nodes=1
#SBATCH --time=00:30:00
#SBATCH --ntasks=8

# TODO: add command
```

## LLM Configuration

HPC-Claw uses the Agentivium Core `LLMClient` abstraction. The included provider client supports OpenAI-compatible chat completion endpoints.

Local Ollama-style endpoint:

```yaml
llm:
  provider: openai_compatible
  base_url: http://localhost:11434/v1
  api_key_env: null
  model: qwen2.5:7b
  timeout: 60
```

Cloud endpoint:

```yaml
llm:
  provider: openai_compatible
  base_url: https://api.openai.com/v1
  api_key_env: OPENAI_API_KEY
  model: gpt-4.1-mini
  timeout: 60
```

Use it with:

```bash
hpc-claw parse examples/requests/mpi_test.txt \
  --llm-config llm.yaml \
  --out intent.json
```

## Slurm Validation

The initial policy profile is `hpc_claw/policy/profiles/hcmut_slurm_v1.yaml`. It requires:

- `policy.account`
- `policy.partition`
- `time.walltime`

It also validates allowed accounts, partitions, GPU placement, GPU count, node count, and walltime limits.

## Legacy Assistant Runtime

The `hpc_claw.assistant` package retains the existing conversational 0.1 path for compatibility. LLM output is never verification authority and must enter the same structured validation and formal verification path before execution.

One assistant turn:

```text
User request
  -> ConversationState
  -> HPCContextBuilder
  -> HPCIntentIR
  -> HPCPolicyValidator
  -> HPCPlanner
  -> ClarificationRequest or reviewable Slurm artifact
```

This path generates reviewable artifacts only.

## Current Limitations

Slurm is implemented first. PBS and Flux remain behind future scheduler interfaces.
The v0.1 intent profile excludes workflow DAGs, malleable jobs, multi-site
federation, and complex co-scheduling. Predictions use trace-derived or
deterministic evidence rather than claiming a new prediction model. Human-study
extensions remain optional while the formal Core verification path stays stable.

This update delivers the local-shell foundation, global home lifecycle, basic TUI,
static detection, trust, persistence, skill discovery, revision proposal support,
and stricter execution guard. Full conversational Core event integration,
pause/resume verification orchestration, provider health onboarding, and live
scheduler execution through the TUI remain follow-up work. Live mutation stays
fail-closed; the headless guard additionally requires a verified script hash,
authorization confirmation, and durable audit path.

See [the work-package map](notes/verifiable_scheduling_work_packages.md) for the
module mapping, assumptions, and Core integration details.
