Training · practice

The model reads training text and adjusts its weights to get better at predicting the next token. Training loss measures how surprised it is by the correct next tokens. Lower is better on that text, but a model can memorize it.

Validation · a progress check

Validation text is held out from weight updates. Track checks it during training to decide which checkpoint to save and when to stop. Validation loss asks: does the model also improve on text it is not learning from directly?

If training loss keeps falling while validation loss rises, the model may be overfitting: learning the practice material without improving on unseen text.

Test · the final exam

A test set is separate data kept out of training and model selection. Use it after choosing your model and settings to estimate performance on new examples. If you repeatedly change the model based on a test score, that set becomes part of your validation process.

Track’s training validation score is not a final test score. Benchmarks evaluate a saved checkpoint on separate tasks; their value as a final test depends on avoiding training overlap and tuning on their results.

Perplexity · prediction uncertainty

Perplexity turns average prediction loss into an easier-to-read scale: perplexity = exp(loss). Lower means the model assigns more probability to the correct next tokens. For example, a loss of about 1.39 gives a perplexity of 4.

As an intuition, perplexity 4 is like being equally uncertain among four choices at each prediction. It does not mean 25% accuracy or four mistakes. Track’s studio models predict UTF-8 bytes, so this is uncertainty per byte, not per word.

Compare perplexity only with the same evaluation text, tokenizer and settings. A lower score on easier text does not establish a better model.