# Hyperparameter Guide

This guide provides practical tuning advice beyond what the API docs cover. For parameter definitions and types, consult the live docs at https://ai-innovation.team/training_hub/#/ and the function reference pages for `sft()`, `osft()`, and `lora_sft()`.

## Universal hyperparameters

### Learning rate

The most impactful hyperparameter. Starting points by algorithm:

| Algorithm | Starting LR | Notes |
|-----------|------------|-------|
| SFT | 5e-6 to 1e-6 | Use lower end for larger datasets or larger models |
| OSFT | 5e-6 to 1e-6 | Same guidance as SFT |
| LoRA | 1e-5 | Adjust up toward 1e-4 only if underfitting; 1e-4 is the aggressive end |

### Effective batch size

Dataset-size-dependent recommendations for SFT/OSFT:

| Dataset size | Recommended batch size |
|-------------|----------------------|
| < 1,000 samples | 32-64 |
| 1,000 - 10,000 samples | 128 |
| > 10,000 samples | 256+ |

`effective_batch_size` is the exact minibatch size used on any backend. Training Hub internally translates it into the correct gradient accumulation steps based on the number of GPUs and per-GPU capacity. Users do not need to compute accumulation themselves.

### Number of epochs

Start with 2-3 epochs for SFT and OSFT. More epochs risk overfitting, especially on small datasets. Monitor validation loss (where available) to determine the optimal stopping point.

### Max sequence length

`max_seq_len` controls the maximum token length per sample. Samples exceeding this are dropped. This directly affects:
- **Memory**: Longer sequences require more GPU memory per forward/backward pass
- **Training speed**: Longer sequences are slower to process
- **Data coverage**: Setting it too low drops valuable long-context samples

Set it based on your data distribution. Check the 95th or 99th percentile of your sample lengths and set `max_seq_len` accordingly.

## SFT-specific parameters

### Pretraining mode

Enable with `is_pretraining=True` when training on raw documents rather than chat-formatted data. Requires `block_size` to specify how documents are packed into fixed-length sample blocks.

- `block_size=2048`: Good default starting point
- `block_size=512`: Better for short or numerous documents
- `document_column_name`: Key in your JSONL that contains the document text (default: `"document"`)

### Unmasking for knowledge data

When using knowledge data generated by sdg_hub where the knowledge documents are embedded inside user messages, set `unmask=true` at the sample level in the JSONL data. This unmasks all messages except the system message for loss computation (not just assistant responses), which significantly boosts knowledge ingestion.

### Full-state checkpointing

`accelerate_full_state_at_epoch=True` saves FP32 full-state checkpoints (model + optimizer + scheduler) at every epoch. This enables training resumption but is very expensive: an 8B parameter model produces ~108GB per checkpoint. Only enable when you need resumable training.

## OSFT-specific parameters

### unfreeze_rank_ratio

The most important OSFT hyperparameter. Controls what fraction of each weight matrix's singular value spectrum is unfrozen for training.

| Value | Effect |
|-------|--------|
| 0.1-0.3 | Conservative: minimal changes, strong preservation of existing capabilities |
| 0.5 | Recommended default: good balance of learning and preservation |
| > 0.5 | Rarely needed for models at 8B+ scale; approaches standard SFT behavior |

**Model size matters.** Larger models generally need a lower `unfreeze_rank_ratio` because their weight matrices have higher rank, meaning the same ratio captures more unique dimensions. For example, a Qwen 1.5B model with a hidden size of ~1.4k has min(m,n) around 1,400 for its weight matrices — so `unfreeze_rank_ratio=0.5` unfreezes ~700 dimensions. A 7B model with hidden size ~3.8k unfreezes ~1,900 dimensions at the same ratio. The practical effect is that smaller models may need a higher ratio to get sufficient learning capacity, while larger models can achieve the same or better learning at lower ratios. This is rarely something you need to think about, but it explains why the same ratio can behave differently across model scales.

**Memory implications**: Higher `unfreeze_rank_ratio` increases memory usage because the full SVD must be computed and more subspaces are unfrozen as trainable parameters. Reducing this is one of the available knobs for OOM situations.

### target_patterns

Advanced users can restrict which modules undergo OSFT using regex patterns. Examples:
- Target only MLPs
- Target only attention matrices (q_proj, k_proj, v_proj, o_proj)
- Any combination that suits the use case

### Unmasking for knowledge data

`unmask_messages=True` unmasks all conversational content except system messages. Same use case as SFT unmasking: knowledge data where documents are embedded in user messages.

### Pretraining mode

Same as SFT: `is_pretraining=True` with `block_size` and optional `document_column_name`. OSFT converts document data into pretraining-style samples internally.

## LoRA-specific parameters

### Rank and alpha

- `lora_r`: Start at 16. Increase to 32 or 64 if the model is underfitting. Higher rank = more trainable parameters and more memory.
- `lora_alpha`: Typically set to 2x the rank (e.g., alpha=32 for rank=16). Controls the scaling magnitude of LoRA updates.
- `lora_dropout`: Default 0.0, optimized for Unsloth.

### Target modules

`target_modules` controls which model layers get LoRA adapters. Auto-detected if not specified. Common choices: attention layers only, or all linear layers for maximum capacity.

### Quantization (QLoRA)

`load_in_4bit=True` enables 4-bit quantization for significantly reduced memory. Useful for training large models (20B+) on consumer GPUs. Use `load_in_8bit=True` as a less aggressive alternative. Do not combine both.

### Multi-GPU

Two strategies (mutually exclusive):
- **Data-parallel (DDP)**: Launch with `torchrun --nproc-per-node=N`. Each GPU holds a full model copy.
- **Model splitting**: Set `enable_model_splitting=True`. Launch with plain `python`. For models too large for a single GPU.

## General advice

- Always establish an evaluation baseline before training. Evaluate the base model, train, then re-evaluate to measure improvement.
- The choice of initial model matters significantly. A strong instruction-tuned model will respond differently to the same hyperparameters than a base model.
- Not all users train on 8xH100 nodes. Scale batch sizes and learning rates to the available hardware. Smaller setups may need smaller batch sizes and careful memory management.
- Refer to the example scripts in `examples/scripts/` and notebooks in `examples/notebooks/` for model-specific configurations that have been tested.
