Metadata-Version: 2.4
Name: argus-forge
Version: 0.1.1
Summary: Training bridge: turn curated dataset exports into ready-to-run LoRA training configs (kohya_ss / OneTrainer / diffusers)
Project-URL: Repository, https://github.com/smk762/argus-forge
Author: smk762
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: pydantic>=2.7
Requires-Dist: structlog>=24.0
Provides-Extra: cli
Requires-Dist: rich>=13.0; extra == 'cli'
Requires-Dist: typer>=0.12; extra == 'cli'
Provides-Extra: dev
Requires-Dist: httpx2>=2.7; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff==0.15.16; extra == 'dev'
Provides-Extra: server
Requires-Dist: argus-cortex[server]>=0.2.0; extra == 'server'
Requires-Dist: fastapi>=0.110; extra == 'server'
Requires-Dist: python-multipart>=0.0.7; extra == 'server'
Requires-Dist: uvicorn>=0.27; extra == 'server'
Description-Content-Type: text/markdown

# argus-forge

Training bridge: turn curated dataset exports into ready-to-run LoRA training configs (kohya_ss / OneTrainer / diffusers).

Part of the [Argus suite](https://github.com/smk762?tab=repositories&q=argus). argus-curator's `/curate` UI already
computes suggested SDXL hyperparameters from the selected set (size, category); forge closes the gap between
"export" and "train" by handing over a **runnable config** instead of numbers on a screen.

```
argus-quarry  ->  argus-curator  ->  argus-lens  ->  argus-forge  ->  your trainer
  acquire          curate/export      caption          configs          LoRA
```

## What it does

Point it at a curator export directory (images + `manifest.jsonl` + `.txt` caption sidecars) and it emits
trainer-native config files, with hyperparameters seeded from the same selection-insight heuristics the
/curate UI shows (dataset size x target category -> repeats/epochs/LR/dim/alpha):

| Trainer | Emits |
| ------- | ----- |
| `kohya` | `dataset.toml` + `config.toml` (+ `train.sh` for `sdxl_train_network.py`) |
| `onetrainer` | `concepts.json` + partial `config.json` to load in the UI |
| `diffusers` | `metadata.jsonl` (HF imagefolder) + `train.sh` for `train_text_to_image_lora_sdxl.py` |

Along the way it:

- validates the manifest (`manifest_version`-aware — refuses an incompatible major),
- **collects captions**: argus-lens writes `.txt` sidecars next to the *source* images, so forge copies them
  next to the exported copies trainers actually read,
- warns when a flattened export collided on basenames (several manifest rows -> one file on disk) and leaves
  the ambiguous file uncaptioned instead of pairing it with the wrong caption,
- emits one kohya subset per image directory (kohya's glob doesn't recurse — a structure-preserving export
  would otherwise train zero images),
- falls back gracefully to a bare folder of images with no manifest,
- resolves the base checkpoint from the manifest's `target_profile.checkpoint` (or the SDXL base).

Everything lands in `<export>/forge/<trainer>/` (plus `metadata.jsonl` at the dataset root for diffusers).

## Install

```bash
uv pip install "argus-forge[cli]"          # CLI
uv pip install "argus-forge[cli,server]"   # + HTTP server for argus-studio
```

Or run the published image (serves `:8103` with `--cors`, and defaults its export root to `/data/out`):

```bash
docker run --rm -p 8103:8103 -v /path/to/out:/data/out ghcr.io/smk762/argus-forge:latest
docker compose up          # local build; see docker-compose.yaml (and .env.example) for the knobs
```

The image contains no trainer, so it defaults to [demo-safe mode](#demo-safe-mode-no-training) — configs
render, training is refused and nothing is written. That default lives in the image itself
(`ARGUS_FORGE_READONLY=1`), so a bare `docker run` is as locked down as `docker compose up`; the port is
published on 0.0.0.0 and a run is real code execution on the host, so the safe direction is the default one.
Set `ARGUS_FORGE_READONLY=0` on a trusted host once you have mounted a trainer.

## CLI

```bash
argus-forge inspect /data/out                          # what's in this export?
argus-forge config /data/out --trainer kohya           # emit configs
argus-forge config /data/out -t diffusers --dry-run    # preview without writing
argus-forge config /data/out --trigger "zxq person" --network-dim 32 --epochs 8
argus-forge trainers                                   # list emitters
argus-forge run /data/out --trainer kohya \
  --env SD_SCRIPTS_DIR=~/kohya-ss/sd-scripts             # run the forged train.sh, streaming progress
```

`run` shells out to the forged `train.sh` (kohya / diffusers; OneTrainer is
driven from its own UI) and streams the trainer's output. It exits with the
trainer's own exit code, so it slots into scripts and CI; `--dry-run` prints the
command without executing, and `--json` streams raw NDJSON `RunEvent`s.

## Server (argus-studio integration)

Start the FastAPI micro-server on **:8103** (peer to lens :8100, curator :8101, quarry :8102):

```bash
argus-forge serve --cors --export-root /data/out
```

`--export-root` is **required** by the API: a request's `export_dir` is untrusted, so it must name a
directory *under* this one, and every `/inspect`, `/config` and `/run` refuses with 400 until it is set.
Without the fence, `POST /config` would forge a tree into any directory the caller names. Also settable as
`ARGUS_FORGE_EXPORT_ROOT` (`FORGE_EXPORT_PATH` is a legacy alias); the published image defaults it to
`/data/out`. The **CLI is unconstrained** by design — it is your own shell, not a request.

Containment is decided on the fully resolved path, so a symlink pointing out of the root is refused rather
than followed, and the root itself is not a valid `export_dir` (a blank field would otherwise mean "treat
the whole shared volume as one dataset"). The path handed to the emitters keeps *your* spelling, though —
`path_map` prefixes are matched against it, so a symlinked root like `/data/out -> /mnt/big/out` still
rewrites correctly. The manifest's `abs_path` caption sources are fenced to the same root: forge will not
copy a sidecar from outside it into your dataset.

`--cors` matters: the studio frontend calls forge **cross-origin** (browser on :3000 → forge on :8103), and
CORS is opt-in — without it the ExportPanel fails with "Failed to fetch" even though `curl` works fine.
(The Docker image passes `--cors` already; this applies to the pip-installed path.)

A bare `--cors` allows the localhost:3000 dev frontend. Name other origins with `--cors-origin`
(repeatable, or `FORGE_CORS_ORIGINS=a,b`); they are **added to** the localhost defaults, not swapped for
them. Entries are compared to the `Origin` header a browser actually sends, so a trailing slash or stray
whitespace is normalised away rather than silently never matching.

`--cors-any` allows *any* origin **credential-less** — anonymous reads from anywhere, for a public demo. A
literal `*` in the allow-list takes that same path rather than becoming credentialed origin reflection.
It never grants a cross-site *write* to an origin you did not name — but origins you **did** name keep
theirs. That pairing is what a public demo needs, because every endpoint that does anything in forge
(`/inspect`, `/config`) is a POST:

```bash
argus-forge serve --cors-any --cors-origin https://demo.example --no-run --export-root /data/out
```

> CORS is not a write boundary. A cross-origin POST with a CORS-safelisted content type is sent with **no
> preflight**, so any page your browser visits can drive an unauthenticated LAN server; the same-origin
> policy only stops it from *reading* the reply. So unsafe methods are additionally gated on `Origin`:
> absent (curl, the CLI, server-to-server) or same host:port or allow-listed → through; anything else → 403.

| Route | Purpose |
| ----- | ------- |
| `GET /health` | liveness + version + export root + whether training is enabled |
| `GET /trainers` | supported trainers + emitted files |
| `POST /inspect` | look at an export dir (counts, manifest, suggested params) |
| `POST /config` | render configs; `dry_run: true` returns contents without writing |
| `POST /run` | start the forged `train.sh` on a background job; returns the run's `RunState` (with `run_id`) — the run outlives the request |
| `GET /runs` | list tracked runs |
| `GET /run/{id}` | a run's status (poll for the terminal `status` + `returncode` — the argus-proof join) |
| `GET /run/{id}/stream` | attach to a run: NDJSON `RunEvent`s, the retained tail then live; reconnect anytime |
| `POST /run/{id}/cancel` | stop a run (SIGTERM→SIGKILL its process group) |

A run is started once (`POST /run`) and watched — or re-watched after a dropped connection — via
`GET /run/{id}/stream`; a client going away never stops the run. `run_id` (also on the stream's
`X-Training-Run-Id` header) is the join key for the argus-proof handoff. The `/curate` page's ExportPanel in
[argus-studio](https://github.com/smk762/argus-studio) uses `/config` to forge a config right after an export
(`docker compose --profile forge up`).

> The local CLI `argus-forge run` streams live in your terminal and is independent of the server registry.

### Demo-safe mode (no training)

`POST /run` executes a script on the host. It runs a forged `train.sh` on the shared dataset volume — the
same single-user LAN assumption as `/config` — but it is **not sandboxed**: treat reaching the port as
equivalent to shell access on the host.

So a host that should render configs but never train — a public demo, a box with no GPU or no `sd-scripts`
— starts forge in demo-safe mode:

```bash
argus-forge serve --cors --no-run       # or: ARGUS_FORGE_READONLY=1 argus-forge serve --cors
```

`/inspect` and `/config` keep working; every `/run` route refuses with **403** and a message saying training
is disabled on this host. The refusal is middleware, so it lands *before* the request body is validated —
on such a host a malformed request deserves the same answer as a well-formed one — and it covers `/run`
routes added later without a per-route guard to forget.

Demo-safe also means **write**-safe: `POST /config` is forced to `dry_run`, so it renders and returns the
files but never touches the volume, and says so in `warnings` — a caller that asked for a real write learns
it did not happen from the body, not by noticing every file's `path` is null. Forge has no authentication,
so on a publicly reachable host an ordinary `curl` would otherwise overwrite the curator's
`metadata.jsonl` and leave an executable `train.sh` behind on shared storage.

`GET /health` reports `"training": "enabled" | "disabled"`, so a frontend can disable its train button up
front rather than discovering the refusal by clicking it. **The published image turns this mode on by
default** (and the compose file with it) — it ships no trainer, so a run there could only ever fail.
`ARGUS_FORGE_READONLY` is a protection flag, so it fails *safe*: a value it cannot parse (`=y`, `=enabled`)
warns and keeps the guard on rather than quietly enabling writes. Only an explicit `0`/`false`/`no`/`off`
turns it off.

### Container ↔ host paths (`path_map`)

When forge runs in the compose stack it sees container paths (`/data/out/...`), but the emitted `train.sh` /
configs are meant to run **on the host**, where those paths don't exist. Tell forge how to translate:

```bash
# per request (CLI: repeatable --path-map; API: "path_map" on POST /config)
argus-forge config /data/out --path-map /data/out=$HOME/argus/out

# or once, via the environment (the compose file can set this from OUTPUT_DIR)
FORGE_PATH_MAP=/data/out=$HOME/argus/out argus-forge serve --cors
```

Every absolute path rendered into configs (`image_dir`, `output_dir`, `--train_data_dir`, OneTrainer concept
paths, ...) gets the longest matching prefix rewritten; the request-level map wins over the env var. The
emitted README notes whether a remap was applied.

## Develop

```bash
make install   # venv + editable install with the "dev,server,cli" extras
make test
make lint
```

## CI / Release

- **CI** runs via the shared [`argus-ci`](https://github.com/smk762/argus-ci) reusable workflow
  (plus `argus-forge schema --check` to keep the committed wire schema honest).
- **Release** publishes to PyPI (OIDC trusted publishing) and GHCR on `v*` tags.
- Versioning is derived from git tags via `hatch-vcs` — tag `vX.Y.Z` to cut a release.

This repo was scaffolded from [`argus-pkg-template`](https://github.com/smk762/argus-pkg-template).
Run `copier update` to pull template changes (CI, release, tooling).

## Roadmap

- argus-proof handoff: post-training validation ([argus-studio#4](https://github.com/smk762/argus-studio/issues/4)).
  `GET /run/{id}` now exposes a run's terminal `status` + `returncode` by `run_id` for the join.
- run registry follow-ups ([#13](https://github.com/smk762/argus-forge/issues/13)): CLI management commands
  (`runs` / `--attach` / `--cancel`), an optional single-flight guard, and durable run metadata across restarts.
