GPT-2 Residual Initialization Scaling

Weight decay (decay weights not biases and layer norm)

cosine LR function

Batch acummalator normalization