Facility at a glance

HPC-DS1 tunes jobs one change at a time. Confident, familiar decisions run automatically; everything else waits here for an engineer. Connected devices each learn their own hardware, and the coordinator sends every task to the device expected to finish it first.

-

devices online

-

tasks and sessions running

-

moves waiting for approval

-

of logged decisions ran automatically

Waiting for an engineer open queue →

Latest decisions audit log →

State

Questions

hpc-ds1-latestauto ≥ 0.85 · escalate < 0.5

Key & instructionsPrimitive type

Edit the job on the left and press Run (Ctrl+Enter).
Without telemetry the job is first run for real on this machine,
then every answer comes from one forward pass of the model.

Devices and task placement

Any machine with HPC-DS1 installed can join over the network. Each device keeps its own measurements and model; the coordinator compares every device's expected finish time (its queue plus its own best measured runtime for the job) and sends the task to the earliest.

Connect a device

1 · On this coordinator, accept other machines

hpc-ds1 serve --host 0.0.0.0

2 · On each device (once: pip install . and hpc-ds1 setup)

hpc-ds1 agent --server http://<coordinator-ip>:8765 --token …

The join token is shown only to this machine. Agents poll the coordinator over HTTP, so devices need no open ports and no direct link to each other.

Devices

Device	Status	Hardware	Measured kernels	Model trained on	Working on

Submit a task

Tasks

Task	Job	Placed on · why	Status	Result

Moves waiting for an engineer

A move lands here when the model is not sure enough (confirm 0.5–0.85, escalate below 0.5) or when the familiarity guard says the job or machine is unlike its training data, however confident it is. The session pauses until you decide; a rejected move ends the session at the last good configuration.

From	Job · current config	Suggested move	Confidence	Why it is here

Tune a job on this machine

Every step runs for real. Faster moves are kept, slower or failing ones are reverted; the session stops when the model says stop or after two misses in a row.

Sessions

Session	Job	Status	Result	Started

Every decision the service made

Kept in memory and appended to ~/.hpc_ds1/audit.jsonl on the coordinator, with who approved what.

#	Time (UTC)	Source	Job · config	Choice	Conf.	Mode	Approved

This machine

HPC-DS1 learns the machine it runs on. The model and its familiarity guard come from measurements of this hardware; on a different machine, run setup once so decisions are based on that machine.

Model on this machine

Set up a new machine

pip install .                 # from the project folder
hpc-ds1 setup --quick         # ~30 min: measure + train
hpc-ds1 setup                 # ~1 h: all 6 sizes per kernel
hpc-ds1 serve                 # this console

Measurements and the model go to ~/.hpc_ds1 (or $HPC_DS1_HOME); hpc_ds1.load() uses them from then on.

What was measured here

Kernel	Sizes measured	Correct runs	Guard: allowed size range

How good it is, measured

All on problem sizes the model never trained on, from real runs on the training machine.

Tuning sessions on held-out sizes

Policy	% of best config reached		Real runs per session

System 1: the whole configuration before anything runs

Approach (held-out test sizes)	% of best config reached		Real runs

Familiarity guard on never-seen kernels

Kernel held out of training	Flagged by guard	By distance alone	Confidently wrong auto-runs: before → after

Plug it into existing tools

The same decision is available as a Python call, a command, or an HTTP request, so it can sit inside a job script, a scheduler hook or a dashboard.

Python

import hpc_ds1

model = hpc_ds1.load()                 # this machine's model
state = model.observe("gemm", 2048,    # runs the job once, for real
                      {"ranks": 2, "threads": 4})
d = model.decide(state)

if d.mode == "auto":
    apply(d.choice)
elif d.mode == "confirm":
    ask_to_confirm(d)                  # one-click yes/no
else:
    escalate(d, d.reasons)             # unsure or unfamiliar

Command line

hpc-ds1 decide --workload gemm --size 2048 \
               --ranks 2 --threads 4
hpc-ds1 tune   --workload stencil --size 160
hpc-ds1 submit --server http://coordinator:8765 \
               --token TOKEN --workload nbody --size 4096
hpc-ds1 agent  --server http://coordinator:8765 \
               --token TOKEN

HTTP

curl -X POST http://127.0.0.1:8765/v1/decide \
  -H "Content-Type: application/json" \
  -d '{"workload": "gemm", "size": 2048,
       "config": {"ranks": 2, "threads": 4}}'

# -> {"choice": "USE_GPU", "confidence": 0.99,
#     "mode": "auto", "reasons": [], ...}

Fleet from Python

from hpc_ds1.fleet import Agent, submit

# on a device
Agent("http://coordinator:8765", TOKEN).run()

# anywhere
t = submit("http://coordinator:8765", TOKEN,
           "montecarlo", 8_000_000)
print(t["placement"]["reason"])