GGUF

Project

required

project

Open a project folder…

A project is one folder holding the downloaded materials, the corpus, the precomputed shards, checkpoints and the exported GGUF. Everything resumes from that folder after a reboot.

Materials

teacher · student · datasets

HF token

Corpus

text prompts

source

Extra local prompt files

one prompt per line, or .jsonl with a "text" field

Train prompts Val prompts Empty fraction Seed

Min chars Max chars

The teacher never sees images, so the corpus is text only. ~1% empty prompts teach the checkpoint's empty CFG prompt. 60k prompts took ~3.7 h of teacher time on an RTX 5090 (4.5 prompts/s).

Precompute

teacher + student▾

Device Shard size Val shard size Student dtype

Teacher batch Token budget Student batch Teacher placement

Teacher GPU memory (GiB, 0 = auto) Teacher CPU memory (GiB, 0 = auto)

The teacher is spread over the CUDA devices and then CPU RAM (accelerate device_map). sequential fills the chosen device up to its memory budget, then the other GPUs, then RAM — on a 5090 + 4050 the whole 30 GB backbone stays on the GPUs. balanced is transformers' default even split: it caps the big card at half the model and offloads the rest to the CPU, where each MoE layer runs its 256 experts eagerly (2.4 vs ~4+ prompts/s). If sequential runs out of VRAM, lower the GPU memory budget by 2–3 GiB instead of switching. Token budget caps batch × (text tokens + 256 query rows).

Dev smoke test: mock teacher (synthetic targets, no download — the adapter is meaningless)

Training

adapter▾

Width Depth Steps Batch size

Learning rate Warmup steps Weight decay Grad clip

Cosine weight Sigma floor Grad checkpoint Device

Log every Validate every Checkpoint every Seed

trainer8 recipe: width 1024, depth 6, 20k steps, batch 32, lr 2e-4, warmup 1k, cosine to 10%, whitened MSE + 0.5·(1−cos) on standardized targets. Judge by val centred cosine (0.965 on the 5090 run, ~55 min); below ~0.9 widen (1536) or deepen before touching the recipe.

Output

f16 GGUF

Adapter name

output dir

Default: <project>/output

also copy to

Optional: your ggk model folder

Also export the paired vision encoder GGUF (SigVQ, for --llm_vision) when downloaded Evaluate the exported GGUF on the val shards Resume automatically when gguf-trainer --auto-resume launches after a reboot

Project

Materials

Corpus

Extra local prompt files

Precompute

Training

Output

Pipeline

Training metrics

Live hardware

Hardware

Environment

Pipeline log

Exported files

Evaluation

Use it in ggk