Metadata-Version: 2.4
Name: neurarch-slurm
Version: 0.1.0
Summary: A SLURM cli_filter plugin that sizes a training job's model at submit, before it takes a reservation: refuses or warns when the requested GPU cannot hold the training peak, and warns when it asks for far more.
Author: neurarch-ai
License: MIT
Project-URL: Homepage, https://neurarch.com/memory-calibration.html
Project-URL: Source, https://github.com/neurarch-ai/neurarch
Keywords: slurm,cli_filter,gpu,admission-control,capacity-planning,training,oom
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: neurarch-skypilot>=0.1.1
Dynamic: license-file

# neurarch-slurm

Know what a GPU job needs before it takes a reservation. A SLURM `cli_filter`
plugin that sizes a training job's model when it is submitted, on the submit
host, and warns (or refuses) when the GPU it asks for cannot hold the training
peak, and warns when it asks for far more than the job needs.

It is the SLURM sibling of [`neurarch-skypilot`](../neurarch-skypilot) and
reuses it rather than repeating it: the sizing is that package's `Sizer`, the
verdict is its `judge`, and the card table is its `cards.json`. A job judged
by SkyPilot and the same job judged here get the same numbers and the same
sentence, apart from where the request is written (`--gres` instead of
`accelerators:`). `tests/test_parity.py` holds that over 384 job/request
pairs.

## What a submit looks like

```
$ sbatch examples/finetune-qwen3-8b.sbatch
Neurarch: Qwen/Qwen3-8B needs 297.9 GB to train at batch 8 per device, 2048 tokens (peak 259.1 GB
x 1.15 headroom), and a10g:1 holds 24 GB. No batch fits a10g:1: the weights, gradients and
optimizer state alone need 180.9 GB. No single card holds it; the smallest request that does is
A100-80GB:4, sharded (FSDP or ZeRO-3) at the same global batch of 8. Sized offline by the neurarch
CLI from the model's structure; nothing was trained. Change --gres and resubmit.
```

That message is real output of the helper (`neurarch-slurm`) with the
`neurarch` CLI from this repo sizing `Qwen/Qwen3-8B` on 2026-10-17, as the
plugin prints it to the user's terminal before returning `slurm.ERROR`. It was
**not** captured from a real sbatch, so whatever line sbatch adds of its own
after a refusing cli_filter is not shown (see
[What was and was not run](#what-was-and-was-not-run)). In the default `warn` mode the same message is printed with "Submitted anyway"
appended and the job is submitted.

## How it works

```
sbatch / srun / salloc (user's process, submit host)
  └─ cli_filter.lua  slurm_cli_pre_submit(options)
       ├─ no GPU in --gres/--gpus/--gpus-per-node/--tres-per-node  -> allow, nothing spawned
       ├─ no #NEURARCH in the script and no NEURARCH_MODEL in env  -> allow, nothing spawned
       ├─ SLURM_JOB_ID set (a step inside an allocation)           -> allow, nothing spawned
       └─ neurarch-slurm --script ... --gres ... --nodes ...
            ├─ facts: #NEURARCH directives (validated by neurarch_skypilot.facts)
            ├─ sizing: neurarch_skypilot.Sizer (neurarch CLI offline, then hosted plan)
            └─ verdict: neurarch_skypilot.judge, SLURM wording
          prints decision=allow|warn|reject and message= lines
```

**Why `cli_filter` and not `job_submit.lua`.** `job_submit.lua` runs inside
slurmctld, under its locks: a plugin there that waits seconds on a sizing call
stalls every submit on the cluster. `cli_filter` runs in the user's own
sbatch/srun/salloc process on the submit host, so it can take the time a
sizing call takes and the controller never waits. The cost of that choice,
said plainly: **cli_filter is not an enforcement boundary.** It runs in the
client, so a user with their own build of sbatch, or anything that submits
through slurmrestd, skips it. It is the place to tell people, before they
wait in a queue for a card that cannot hold their job; it is not a control.

**Fail open, always.** No helper on the host, no output, a crash, a sizing
source that is down, a GPU type with no capacity on file, a directive that
cannot be read, an untyped `gpu:2` with no site default: every one of these
lets the submit through, with a warning where there is something to say.
`reject` mode refuses only the two verdicts that mean the job should not take
the reservation as asked: the request cannot hold the training peak, or the
run is over the cost cap. An over-request is a warning in every mode.

## Telling the check about the job

The job's facts go in the batch script as `#NEURARCH` directives, read the
way sbatch reads `#SBATCH`: from the top of the file, stopping at the first
line that is neither blank nor a comment.

```bash
#!/bin/bash
#SBATCH --gres=gpu:a10g:1
#NEURARCH model=Qwen/Qwen3-8B batch=8 seq-len=2048
#NEURARCH lora-rank=16 attention=sdpa hours=6
```

| key | meaning | default |
|---|---|---|
| `model` | Hugging Face id, or a path to a `.neurarch.json` from neurarch-trace | required |
| `batch` | per device | 32 |
| `seq-len` | tokens per sample | 2048, or the model's context if shorter |
| `lora-rank` | omit for a full fine-tune | none |
| `attention` | `sdpa`, `flash` or `eager` | `sdpa` |
| `hours` | expected wall hours, for the cost cap | none |

Why directives, rather than `--comment` or the environment:

- The GPU request and the facts it is judged against sit in one file and
  change in one diff, so whoever reviews the job script sees both.
- `--comment` is free text that sites already use for accounting tags and
  workflow ids; taking it over collides with whatever is there.
- An exported `NEURARCH_MODEL` outlives the job it was meant for, and would
  quietly size every later submit from that shell.

The environment (`NEURARCH_MODEL`, `NEURARCH_BATCH`, ... the same names the
SkyPilot policy reads) is used only when the script names no model, for
`srun`/`salloc` and for `sbatch` reading its script from stdin. The two
sources are never mixed key by key. A value that cannot be read
(`batch=eight`) is reported and the job is submitted unchecked: it is never
guessed.

The training command has to use the same values; nothing passes the
directives on to it. [`examples/finetune-qwen3-8b.sbatch`](examples/finetune-qwen3-8b.sbatch)
shows the pattern.

### The GPU request

`--gres=gpu:<type>:<n>`, `--gpus-per-node` and `--tres-per-node=gres/gpu:...`
are per node and are multiplied by `--nodes` (its minimum, for a range: the
smallest allocation the job accepts is the one that has to hold it).
`--gpus` is the job's total. A multi-card request is sized as one job sharded
across all its cards, the same rule as `npx neurarch rightsize` and the
SkyPilot policy: the peak at the global batch against the summed memory, with
1.15x headroom.

GPU type names are site-specific. Without a map, a type is matched
case-insensitively to a Neurarch card id or SkyPilot name (`a10g`, `l40s`,
`h100`, `h200`, `a100-80gb`; note that a bare `a100` means the **40 GB** card
there). With a map, only the site's own types are judged and every suggestion
names a card the cluster has:

```json
{ "a10": "A10G", "a100": "A100-80", "h100": "H100", "l40s": "L40S" }
```

Card ids: `T4`, `L4`, `A10G`, `A100-40`, `L40S`, `A100-80`, `H100`, `H200`.
A type outside the map (or the table) is not judged.

## Install

On every host users submit from (login nodes), and on compute nodes if jobs
submit jobs:

```bash
# 1. The helper and the package it builds on (neurarch-skypilot comes with it).
pip install neurarch-slurm
which neurarch-slurm          # put this path in cli_filter.lua: `helper`

# 2. The sizing CLI, offline (Node 18+). npx --no never fetches at submit time,
#    so install it, or point `cli` in cli_filter.lua at a checkout:
#      cli = "node /opt/neurarch/packages/neurarch-cli/bin/neurarch.mjs"

# 3. The plugin, next to slurm.conf, and turn it on.
neurarch-slurm --print-lua > /etc/slurm/cli_filter.lua
echo 'CliFilterPlugins=cli_filter/lua' >> /etc/slurm/slurm.conf
```

`CliFilterPlugins` is loaded by the client commands (sbatch, srun, salloc)
from `slurm.conf`; keep `slurm.conf` identical across hosts as usual. A site that already
has a `cli_filter.lua` keeps it: `dofile` this file at its top (it defines a
global `neurarch` table and the three `slurm_cli_*` functions) and then
redefine `slurm_cli_pre_submit` to call `neurarch.pre_submit(options, offset)`
alongside its own logic.

### Settings

At the top of `cli_filter.lua`, in the `neurarch` table. They live in the
admin-owned file on purpose: nothing a user exports can switch the check off
or change the mode.

| field | meaning | default |
|---|---|---|
| `helper` | command that runs the helper; absolute, since sbatch has the user's `PATH` | `/usr/local/bin/neurarch-slurm` |
| `mode` | `warn` or `reject` | `warn` |
| `gpu_map` | path to the JSON map above | none |
| `default_gpu_type` | what an untyped `--gres=gpu:N` means here | none (untyped requests are not judged) |
| `max_cost_usd` | refuse (in `reject` mode) a run whose `hours` x cards x list price is over this | none |
| `prices` | path to a JSON object, card id to USD per GPU-hour | Modal on-demand list prices |
| `annotate` | append `neurarch:<verdict> need=<GB> on <request>` to `--comment`, so `sacct -o comment` keeps what admission said | `false` |
| `sizing` | `auto` (CLI, then hosted), `cli` or `hosted` | `auto` |
| `cli` | the sizing CLI command | `neurarch` on `PATH`, else `npx --no neurarch` |
| `timeout_s` | seconds per sizing source | 30 |
| `debug` | show the helper's stderr | `false` |

Start on `warn`, read what it prints for a week, set `gpu_map`, then switch to
`reject`.

## The repository's `.neurarch.yml`

The helper looks for a `.neurarch.yml` next to the batch script, then in the
submit directory, up to the repository root (or `NEURARCH_POLICY_FILE`), and
applies it through the same CLI that sizes the job (`neurarch policy apply`).
A waiver scoped to this cluster (`cluster: "gpu-dev"`, matched against
`SLURM_CLUSTER_NAME` or `NEURARCH_CLUSTER`) turns a reject into a warning that
names its owner and expiry; `budget.cards_allowed` and the other budget lines
reject in `reject` mode and warn in `warn` mode. Everything that goes wrong on
our side still fails open: a policy file with no CLI to read it is reported and
the submit goes through as asked.

## Where the number comes from

Exactly where `neurarch-skypilot` gets it, through the same code: weights,
gradients and AdamW state plus activations, computed from the model's
structure (a Hugging Face `config.json` or a traced graph), calibrated on
measured runs and published next to `accelerate estimate-memory` at
<https://neurarch.com/memory-calibration.html>. Arithmetic: nothing is
trained, downloaded beyond `config.json`, or run on a GPU. The offline CLI
reads `config.json` from huggingface.co (set `HF_TOKEN` for a gated repo); the
hosted plan takes a `.neurarch.json` graph and does not size LoRA. See the
SkyPilot README for the details of each source.

## What was and was not run

Run, on 2026-10-17:

- `python3 -m unittest discover -s tests` (47 tests): directive and `--gres`
  parsing, every decision and fail-open path, the helper as a subprocess with
  a stand-in sizing CLI, and the SkyPilot parity grid.
- The helper against the **real** `neurarch` CLI from `packages/neurarch-cli`,
  sizing `Qwen/Qwen3-8B` from Hugging Face. The refusal at the top of this file
  and the borderline LoRA case in the example script are its output.
- `tests/test_lua.py` runs `cli_filter.lua` under a real Lua interpreter with
  a stub `slurm` table and the real helper. It is skipped where no `lua` is
  installed (it was skipped on the machine this was written on) and runs in CI,
  which installs `lua5.4`.

Not run:

- **No real slurmctld, sbatch, srun or salloc.** The plugin follows the
  documented `cli_filter/lua` interface (`slurm_cli_setup_defaults`,
  `slurm_cli_pre_submit`, `slurm_cli_post_submit`, option names as the long
  flags), but the option table it reads has only been a stub. Two things in
  particular need checking on a live cluster: that `options["argv"]` carries
  the batch script path in your Slurm version (if it does not, the check falls
  back to the `NEURARCH_*` environment and a script-only job is silently not
  judged), and that assigning `options["comment"]` takes effect.
- No timing at scale: each judged submit spawns Python, and a cold sizing call
  fetches one `config.json`. Jobs without a GPU request or without a model
  named spawn nothing.

## Not done yet

- The time limit (`--time`) is not used for the cost cap; `hours=` is.
- Verdicts are not recorded anywhere but the user's terminal and, with
  `annotate`, the job's comment.
- No cache across submits: a user submitting 50 jobs sizes the model 50 times.
- `--gpus-per-task`, MIG and sharded GPUs (`shard`, `mps`) are not judged.
- Not published to PyPI.

## Tests

```bash
cd packages/neurarch-slurm && python3 -m unittest discover -s tests
```

Standard library only; `neurarch-skypilot` is imported from the sibling
directory, so nothing needs installing.

MIT licensed.
