RFC-005: A research program for a 250M-class English base model

• Status: draft proposal
• Version: 0.4
• Audience: core team, data-science reviewers, training and infrastructure contributors
• Date: September 11, 2026
• Related: RFC-001 — project work, RFC-002 — dataset lifecycle, RFC-003 — corpus pipeline, proposed, RFC-004 — contribution and release, proposed

1. Goal and decision

Train an English general-purpose base model from scratch, optimizing capability at a fixed parameter budget. The research opportunity is data selection and optimization, measured through controlled proxy experiments before committing to the main run. Success is a credible attempt at a leading small base model, not a promise that a particular training budget will achieve SOTA.

The recommended proxy program is 10–20 controlled runs at approximately 125M parameters, each reaching 2–5B tokens, using the available no-rental-cost compute. Test data mixture, tokenizer, Muon versus AdamW, depth/width, GQA, ReLU² versus SwiGLU, LR, and curriculum/annealing. Reserve slots for a combined recipe and independent seeds. A 50M rung is optional debugging work, not the decisive judge of data quality.

Recommended main-run target: 100B tokens, with annealed 25B and 50B checkpoints and an optional 200B extension, as confirmed by the proposer. Preserve a stable-LR trunk and separately annealed endpoints. The target is a research allocation, not a minimum needed for a 250M model or a guarantee of SOTA; execution remains subject to the stage gates below.

Token budget and the Chinchilla comparison

The familiar Chinchilla-style planning heuristic is roughly 20 training tokens per parameter. For a nominal 250M model that is 5B tokens. “20 tokens per parameter” and “20 times that budget” are different quantities:

The 100B target is therefore approximately 20× Chinchilla-style exposure. It is deliberately longer training at a fixed small model size, rather than optimizing the allocation of a fixed pretraining compute budget between model size and data. It is not a statement that every 250M model needs 100B. Use actual implemented parameter counts for reported ratios, and keep the denominator distinct from a source's “scaling parameter” convention.

Free compute changes the financial constraint on proxies, not the need to record GPU-hours, failures and data exposure. Each stage has an owner, a work/allocation budget and an evidence review. Rented stages additionally require a numeric spend cap. Instruction tuning, an assistant deployment and public model publication are separate decisions.

Research questions

1. Which recipe choices improve held-out English quality at matched size and compute?
2. Which data/tokenizer choices survive longer proxy training and another seed?
3. What marginal BPB/CORE gains do longer, comparably annealed main-run endpoints provide?
4. Does the selected result reproduce across fresh initialization and data-order seeds?

DataDecide reports approximately 80% correct pairwise data-ranking decisions from 150M to its 1B target and useful continuous likelihood proxies at small scales. This motivates intermediate-scale data experiments; it is not an 80% guarantee for our campaign. Its nominal 150M primary runs reach approximately 15B tokens, whereas our proxies reach 2–5B. Validate ranking stability rather than assuming that the result transfers to longer-trained English models.

2. Starting point and boundaries

The training audit is based on Fabryka Track commit bc90c207, inspected September 11. It establishes source behavior, not current deployed configuration or measured GPU performance.

RFC-001 governs task ownership and review. RFC-002 supplies the current documented manual path for data acceptance. RFC-003 and RFC-004 were open proposals when this draft was prepared; this RFC neither adopts nor supersedes them. If adopted, their versioned outputs and decisions become inputs here. A manual accepted data snapshot can support the first experiment while the automated pipeline is being built.

Corpus build and release policy remains in the data RFCs. This RFC adds experiment-specific splitting, tokenization, sampling, checkpointing, and evaluation. In particular, RFC-003's proposed cl100k counts describe the corpus; they are not this model's training-token budget. The experiment records both units separately.

3. Target architecture and optimizer hypotheses

The preferred research starting point is deep and narrow, with ReLU² and Muon. They remain hypotheses with controls. MobileLLM motivates testing depth, embedding sharing and GQA below 1B; it does not establish this exact configuration.

Under these definitions the analytic count is 251,714,304 trainable parameters. With FFN width 3,040 it is 249,944,832, a candidate for a strict ≤250M category. Choose the category before the final recipe is frozen: a 251.7M model must never be labeled a strict ≤250M entry. Count all unique trainable weights, including tied embeddings once, and audit the implemented model. Report non-embedding/scaling counts separately.

For tied embeddings, learned pre-attention/pre-MLP/final RMSNorm and parameter-free QK norm:

N = V*d + L*(2*d*d + 2*d*k + m*d*f + 2*d) + d

Here k = KV_heads * head_dim; m=2 for ReLU² and m=3 for SwiGLU. An untied head adds V*d; learned QK gains would add parameters and require another audit.

The two deep rows have different head dimensions as well as activation. They are candidate bundles, not a clean activation ablation. The controlled proxy activation test keeps attention unchanged and switches FFN width from 4d ReLU² to 8d/3 SwiGLU. Test head layout separately if promoting the 12Q:4KV candidate. Define the shallow/deep width mappings at the proxy scale before execution, including any 24-versus-36-block comparison; no unspecified “depth equivalence” is accepted.

Keep total parameters within a proposed ±2% band around the proxy target. Tying and GQA change parameter allocation: record compensating width changes and describe size-matched recipe comparisons honestly. Preserve a fixed-shape control if isolating the mechanism is important. No value embeddings or recurrent hybrids in this first matrix.

Optimizer and numerical controls

Use a pinned Muon implementation and explicit matrix partitioning; do not route embeddings to Muon merely because they are 2-D. The tied embedding appears in AdamW exactly once. Maintain an all-AdamW control, and serialize both optimizers' state for resume.

PyTorch's Muon documentation distinguishes LR adjustment conventions, including original and match_rms_adamw. Select and record the convention. With matched-RMS adjustment, proposed initial LR candidates are 6e-4, 1e-3 and 1.6e-3, subject to a short stability check. These are not directly interchangeable with a raw Muon LR copied from another implementation. Pin a supported library release rather than relying on moving main documentation.

Starting settings: Muon momentum 0.95; AdamW betas 0.9/0.95; hidden-matrix decay 0.1, zero decay on embeddings/norms; gradient norm clipping 1.0 after accumulation. Treat each optimizer's calibrated group LRs as part of the recipe. Give both optimizer families comparable bounded tuning opportunities. Recalibrate at 250M, then freeze the scale-specific LR before comparing data/horizons.

BF16 is the comparison reference. FP8 is a later engineering candidate only on compatible hardware after loss/gradient and recovery checks against BF16; do not assume the reported 4×3090 setup supports an FP8 path. Any accepted numerical change becomes a new recorded recipe rather than an unreported speed optimization.

4. Data candidates and research-use boundary

Establish pure ClimbMix as the research data baseline before mixture changes. NVIDIA's dataset card describes approximately 400B GPT-2-tokenized tokens and specifies CC BY-NC 4.0 and research/development-only use. This RFC treats ClimbMix runs as research artifacts; commercial deployment is not cleared by this proposal. Non-Climb candidates also need source/use review and are not automatically cleared alternatives.

These are interpretable starting experiments, not final optimal weights. B versus A estimates the combined math/code intervention, not the separate contribution of each. If that distinction matters, use an interaction slot to separate them. Scientific/encyclopedic text or ClimbLab semantic-cluster weights are follow-ups competing for the same finite slots; do not build a new clustering/search service before testing broad mixtures.

Sources: FineWeb-Edu, DCLM-Baseline, FineMath, and Stack-Edu. Freeze the FineMath subset, programming languages, quality filters, source-license filters and internal weights. Stack-Edu supplies file identifiers rather than all code contents; actual retrieval, checksums and eligible-source metadata are readiness work.

For NVIDIA's tokenized release, decode GPT-2 sequences with the pinned procedure, preserve document boundaries/special tokens and retokenize. For an already decoded mirror, validate conversion/provenance first and tokenize its text directly. The advertised upstream token count is not the count under our tokenizer.

Proposed cluster-addressable input: gvlassis/ClimbMix

Use the user-supplied gvlassis/ClimbMix snapshot as the preferred input candidate, subject to provenance and coverage verification. Its card/schema expose decoded text in Parquet and 20 cluster_id=N configurations. GPT-2 decoding is therefore not needed on this ingestion path.

The card's ratios are document-count shares, not model-token shares. Topic names come from a model labeling 100 samples per cluster; they are exploratory labels, not validated semantic classes. Do not equate these 20 partitions with NVIDIA's original 1,000 clustering groups.

For baseline A, preserve the verified release distribution: inventory every partition, tokenize and compute each cluster's token mass, then sample token blocks using those measured masses. Uniform cluster weighting or reusing document-share percentages as token weights changes the mixture. Validate sampled text, boundary mapping and counts against upstream before claiming an unchanged ClimbMix baseline. The mirror's missing license metadata is not permission to disregard upstream restrictions.

A cluster-reweighting contrast can occupy P20 instead of a generic interaction. Freeze its weights before execution, recount after deduplication and compare against the same-tokenizer A control. Until equivalence is verified, identify A as the pinned mirror baseline rather than an exact NVIDIA reproduction.

Deduplicate within and across sources, record provenance/tie-breaks, and split by document/related duplicate group before tokenizer fitting and packing. Training data must exclude the union of development and final evaluation material under a recorded decontamination method. Pin source revisions and checksums, preprocessing, split IDs and token counts. Pre-stage local shards; do not make the main loop depend on a live remote stream.

Record actual sampling weights, unique tokens, repeated passes and residual duplicate/contamination limitations. Check whether each component can support the contemplated 200B extension without excessive unplanned reuse; any later weight/reuse change is a new recipe decision. A proposed general-text → capability-rich curriculum is tested with matched total source exposure before adoption. Horizon experiments otherwise keep the selected data policy fixed.

5. Proxy program: 10–20 controlled 125M runs

Use 125M as the principal decision scale and 2B as the first comparable exposure, extending selected runs to 5B. The recommended ceiling is twenty trajectories, including independent seed repeats. Ten runs is a smaller program with explicit deferred axes, not full coverage of every variable. The optional 50M smoke work does not replace these runs.

The default twenty-slot allocation below provides sixteen exploration/control trajectories and four combination/confirmation trajectories. Each row specifies its parent control. A single-factor label permits only the stated intervention, subject to documented parameter matching. The exact execution order is adaptive but each next contrast is registered before its outcome is observed.

P01–P16 normally reach 2B; P17–P20 may reach 5B. Any original control compared against a 5B trajectory must also reach 5B, and that extension consumes additional tokens within its trajectory allowance. Save stable checkpoints for valid continuation. Track actual new work when resuming or forking; never charge an already-trained prefix twice as executed compute, and never omit a new tail.

Hold data order, tokenizer, context, batch, token exposure and schedule constant within non-tokenizer comparisons. Use the same initialization/data-order seed labels across candidate/control pairs, while noting that different shapes cannot have identical weight tensors. A curriculum contrast uses the same total source counts and, where practical, the same document multiset in a different order; this isolates order from extra specialist exposure.

For tokenizer P09, use the same tokenizer-training text sample, total vocabulary budget and special-token inventory where possible. Different tokenization changes bytes seen per training token. Report matched-raw-byte/document exposure and matched-compute readouts as well as token-budget curves; compare BPB on identical text rather than cross-tokenizer perplexity. Maintain 2,048-token context but disclose its different text span. Freeze the selected tokenizer before confirmatory data comparisons. A tokenizer change invalidates reuse of old packed shards and weights.

QK norm is enabled in the reference. A disable-QK-norm ablation can replace P14 or P20 if prioritized; it must not be reported as tested unless actually run. Scientific/cluster mixtures and FP8 similarly consume slots instead of silently expanding the matrix. This is a controlled screening program with selected interactions, not a complete factorial experiment.

Selection and stopping

Use a fixed English development panel. Optimize the BPB / DCLM CORE plane, with continuous answer-likelihood diagnostics as the sensitive proxy selector. Freeze practical BPB/CORE regression tolerances, metric weights and a tie rule before the screen. At 2B, reject instability and clearly dominated candidates; do not reject solely on noisy near-chance accuracy. Extend close, consequential contrasts to a matched 5B exposure within the allowance.

A recipe advances when its continuous improvement is consistent with held-out BPB and CORE guardrails and is not a single-task/seed artifact. Report the frontier when metrics conflict instead of inventing a total ordering after seeing results. Prefer a simpler, cheaper and usable recipe under a predeclared tie rule when evidence is indistinguishable. Publish all attempted cells and reasons for stopping internally with the experiment report.

The first main-model research baseline is the selected architecture on pure A. If another data recipe wins, keep its same-architecture A control identifiable at the proxy scale and budget a matched 250M control when claiming a target-scale data gain. A pure-A 250M baseline is a real extra run if it is not the main trajectory; it is not free because a public ClimbMix checkpoint exists.

6. Stable main run, endpoints and independent confirmation

Train the locked recipe with warmup in absolute tokens (initial candidate 100–200M, calibrated before launch), then stable LR. This avoids committing the entire trajectory to a cosine endpoint. WSD research supports this flexible scheduling pattern; it does not fix our optimal cooldown length.

The source notes proposed both a short 1–3B linear cooldown and a 10%-of-horizon cooldown. Test the scheduling contrast at proxy scale; default for budget illustration is a fixed 2B linear tail to 5% of peak at each main endpoint. A 10%-tail option remains viable but has different cost. Freeze the selected curve, floor and duration before endpoint comparisons.

Producing 25B/50B/100B costs 98B stable tokens plus three 2B tails, 104B executed tokens. A 200B extension with all four endpoints costs 206B. With 10%-of-horizon tails the equivalent totals are 107.5B and 217.5B. These calculations stop the trunk at the final fork; running stable training through the endpoint adds further work.

Save unannealed monitoring checkpoints at 10B and 20B and then at least every 5B, plus all exact fork boundaries. They are diagnostics, not directly comparable finished endpoints. Extra annealed 30B/40B or 10B/20B results are optional branch work, recorded and budgeted before execution. At each continuation gate measure marginal BPB/CORE improvement and actual cost. Two successive comparable endpoints with little useful improvement trigger a stop/recipe review, not an automatic extension.

Fork complete model, optimizer, RNG and data cursor from each stable parent. Restore the stable parent to continue the trunk; never continue it from a decayed child. Branches get distinct Track IDs, parent hashes, schedules, actual fork positions and realized exposures. Only the declared annealing policy changes during a horizon contrast. They share ancestry and are not independent seeds.

If curriculum wins, freeze the stable-trunk curriculum and endpoint policy before main training. A horizon-relative late-mixture switch belongs in an explicit child branch; it must not contaminate a trunk intended for later horizons. Compare curriculum and constant-mixture policies on matched exposures rather than attributing a combined data/schedule intervention to extra tokens alone.

Confirmation and claims

Choose the horizon using development results and a predeclared practical-improvement threshold. Then run another one or two independent seeds from initialization to that selected horizon H, with unchanged data, tokenizer, optimizer and schedule. Confirmation costs H or 2H additional tokens, not another tail. Report discovery and confirmatory seeds separately and retain every seed's result. A changed recipe restarts exploration.

Use a final audit panel only after recipe/horizon selection. Two or three total seeds reduce the risk of a lucky winner but do not guarantee statistical power. Parameter-limited quality and training efficiency are separate claims. To claim SOTA, specify the parameter category, accessible comparison set, evaluation protocol, contamination limitations and uncertainty; reporting the best seed or mixing model-card scores is insufficient.

7. Trainer, tracking, and recovery

The first implementation is an on-demand trainer integrated through Track's SDK. Its code and recipes should live in the Track repository unless the team selects another implementation home at kickoff. Reuse an established model library and the existing logging client; implement only the data, training, resume, and evaluation integration needed here.

Record the code commit, image digest, dependency lock, model/tokenizer configuration, corpus and split hashes, evaluation protocol ID, seeds, hardware, precision, batch settings, schedule, and launch command. Log training loss, subword and byte-normalized validation metrics, tokens seen, learning rate, gradient norm, throughput, memory, and cost where available. Tokenizer and metric units must be visible in the run metadata; subword perplexity must not be labeled byte perplexity.

Save full resume state every 30 minutes, at evaluation milestones, and on a graceful stop: model, optimizer, scheduler, RNG states, sampler/shard cursor, counters, and manifest hashes. Save at optimizer boundaries. Write a temporary checkpoint, verify it, then atomically mark it complete; a partial checkpoint must never replace the previous usable one. Keep latest and previous recovery checkpoints plus the selected milestone exports.

Before the main run, compare uninterrupted and interrupted/resumed execution on a bounded fixture. Token order, counters and schedule must match exactly; compare losses/weights within a declared numerical tolerance for the selected kernels. Bitwise equality across different GPU/software stacks is not promised. Also load the exported model and tokenizer in a clean inference process.

Tracking outages should spool events locally and leave a recoverable run. Check storage headroom before continuing; pause if checkpoints or local logs cannot be written. Verify copied artifact hashes before releasing the training device. The studio controller's 24-hour limit is not bypassed silently: use a separately supervised external job for this first version, with its own deadline, spend cap and tested recovery.

Record experiment lineage in a versioned run manifest: stage, cell, seed, architecture/data/optimizer IDs, parent checkpoint and branch schedule. Retain full stable fork checkpoints until all planned endpoints and verification are complete. Retention of only the best development checkpoint would destroy the planned branching experiment.

8. Evaluation and comparison baselines

Pin the DCLM evaluation implementation, its full CORE task configuration, normalization, prompts and revisions. Full CORE is a defined aggregate; a cheap subset or continuous proxy is labeled separately and never called the official full score. Maintain a fixed multi-domain English held-out BPB panel independently of mixture weights. Target at least 5M text tokens each for development LM and final LM panels, with document-level split isolation.

For cheap comparisons retain per-answer conditional log-likelihoods for HellaSwag, ARC-Easy/Challenge, PIQA, BoolQ, WinoGrande and MMLU. Add reference-solution likelihood for MBPP/HumanEval as code diagnostics; it is not execution-based pass@k. A DataDecide-style score is mean(exp(log P(correct continuation | prompt) / answer character count)). It is character-normalized likelihood, not probability normalized over choices. Freeze formatting, Unicode character definition and special-token handling. Report raw likelihood, normalized-choice probability/margins and ordinary accuracy separately.

For text, BPB = total text NLL / (original UTF-8 byte count × ln(2)). Score every text token once with a fixed context/window policy, reset document context, exclude artificial special-token loss from text BPB and report EOS separately. Across tokenizers use the same raw documents, not per-token perplexity or equal-token validation samples with different text.

Use a small fixed loss panel every 0.5B training tokens, cheap task panels at 2B increments, and full development CORE/BPB at completed proxy readouts and annealed main endpoints. Final checkpoints additionally get ordinary task scores, BLiMP/LAMBADA where included in the pinned audit, fixed generation probes and a contamination report. These are proposed frequencies; measure their overhead before freezing them.

Keep context at 2,048 during core comparisons. Record per-task prompt-length distributions, lost demonstrations, truncated/dropped examples and exact truncation policy. Even 2,048 does not guarantee a prompt fits. A shortened-shot/context variant is a separate protocol and must be rerun for every baseline.

Little-LM reports a ReLU² throughput benefit, Muon/AdamW engineering and substantial CORE changes caused by prompt truncation. Its later recipe still used a long cooldown, so the write-up does not validate a universal 1–3B anneal. Its single-project results motivate ablations rather than establish the best ≤250M model. We do not transfer its GPU throughput, cost or score to this campaign.

Use paired bootstrap over shared evaluation examples/documents, with between-seed variation reported separately. Item bootstrap cannot estimate training-seed uncertainty. Missing required components mean an incomplete result; no silent weight redistribution. Screening is adaptive, and only frozen final comparisons are confirmatory.

CLIMB used development-task signals in mixture optimization; record the overlap between its selection tasks and ours and require evidence outside that set before claiming broad generalization. Reserve a disjoint final panel from all local tuning and record known public-benchmark exposure limitations.

Baseline register

The inspected plain ladder card lists 268,435,514 total and 234,881,024 scaling parameters. Nominal 200 scaling-TPP is about 46.98B tokens, with realized exposure from the saved-step record. The optimized 286M card lists 286,261,730 total and 110,100,912 scaling parameters; 200 scaling-TPP is about 22.02B tokens. This is why TPP must always carry its denominator. Cards were inspected, not independently benchmarked here; weight-count and checkpoint audits remain execution work.

Ask two distinct questions: can the candidate improve parameter-limited quality, and can an early endpoint improve quality at comparable measured exposure/cost? A 100B winner against the ~47B baseline does not establish better training efficiency. A 30–40B win would be more informative for that claim, but still needs tokenizer/compute and protocol accounting.

9. Compute envelope and gates

The user identifies no-rental-cost proxy compute and a 4×3090 setup. Verify current access, GPU inventory, competing jobs, usable memory and disk before scheduling; this draft has not measured that host. Independent proxy runs may use separate idle GPUs when fit/throughput permits. Do not reserve devices merely because they appear in an earlier configuration. Rental H100/B200 or other hardware is selected only after measuring the locked 250M recipe.

A nominal parameter-times-token proxy makes the default 52B-at-125M program about 26% of a 250M × 100B trunk, with the 20–100B envelope spanning roughly 10–50%. Thus the expanded free-compute program is not automatically within the earlier 15–20% exploration allowance. Report the expanded allocation honestly. Parameter-times-token is a rough accounting aid, not measured FLOPs or runtime across architectures/tokenizers.

Benchmark each materially different recipe at final batch/context/optimizer settings, including at least 30 minutes and a checkpoint/evaluation cycle. Record steady and end-to-end aggregate tokens/s, memory, utilization, storage stalls and optimizer time. Use whole-cluster rate with aggregate throughput, never per-device price with cluster-wide speed.

These scenarios exclude branch/evaluation/setup overhead and are not hardware predictions. Cost is the sum over executed runs of tokens / measured aggregate tokens_per_second / 3600 × whole-cluster hourly rate, with overhead and storage/transfer charges added once. Confirmatory full runs are outside a small generic contingency. No provider quote, booked device or completed performance measurement is claimed.

A materialized 100B-token uint16 stream is about 200 GB; 200B is 400 GB, before raw/decoded data, alternative tokenizations, indexes, checkpoints and backups. Size storage for unique inputs and planned repetitions, not just the final model file. Muon/AdamW state depends on matrix partitioning, so measure actual checkpoint size and total VRAM rather than treating an all-AdamW rule of thumb as a fit guarantee.

Stop for nonfinite training, artifact corruption, inability to save state or projected allocation/spend breach. Checkpoint early enough to transfer before a deadline. Negative or inconclusive results can satisfy RFC-001 when their evidence and conclusion are reviewed.

10. Ownership, delivery and remaining decisions

Name an experiment owner, independent technical reviewer, data owner with the required data reviewers, evaluation owner and infrastructure/budget owner. Roles may overlap but nobody accepts their own result. Agree a readiness work budget and review date; no calendar promise for the whole program is implied.

Deliver a versioned matrix and exact configs; accepted source/split/tokenizer manifests; runnable training and evaluation; recovery/fork tests; complete comparison ledger; stable parents and annealed exports; independent confirmation; and a model card with scope, failures, costs and limitations. A second contributor must be able to load the model and reproduce the evaluation from the record.

Use the existing Track SDK first. Studio support is separate work across model schemas, manifests, optimizer state, tokenizer-aware generation/evaluation, artifact handling, runtime and labels. Preserve explicit byte-model checkpoint compatibility. No new always-on service, clustering platform or synthetic-data generation pipeline is required for this campaign.

Public release requires the agreed destination, access scope, applicable data/use decisions and hash verification. ClimbMix-derived research artifacts are not silently promoted into a commercial release. Any later data issue must be traceable to the affected runs and branches through their manifests.

Before execution, settle: strict ≤250M versus 250M-class; concrete proxy depth mappings; tokenizer alternative; optimizer conventions/LRs; specialist-source inventory; practical evaluation tolerances; cooldown policy; named owners; hardware/storage; and each stage's allocation. These are explicit decisions at the relevant gates, not fabricated measured outcomes.

The alternatives are a direct fixed-recipe run (less research overhead), a reduced ten-run proxy matrix (deferred axes), a full factorial design (much larger), or continued pretraining of an existing model (different research question). The selected program earns its cost only if its controlled comparisons help choose the main recipe. Free compute is a reason to investigate more carefully, not to treat every added experiment as informative.