| Name | Dataset | Tokenizer | Training Library | Embeddings | Normalization | Parallel Layers | Biases | Activation Function | D Attn / D FF | Optimizer | Optimizer Hyperparameters | LR Warmup | LR Decay | Precision | Clipping | Dropout | Weight Decay | Date | Source |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT1 | Unreleased | GPT-1 | Unreleased | Learned | LayerNorm | No | Yes | GeLu | 4 | Adam | Not Disclosed | Cosine to 0 | Not Disclosed | Not Disclosed | 0.1 | Not disclosed | 6/11/2018 | https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf | |
| Pythia | the Pile | GPT-NeoX-20B | GPT-NeoX | Rotary | LayerNorm | Yes | Yes | GeLu | 4 | Adam | 0.9, 0.95 | Cosine to 10% | fp32 / fp16 | 1.0 | 0 | 0.1 | 12/10/2022 | https://arxiv.org/abs/2304.01373 |
Many papers say that they follow another paper except for some changes. When such a paper doesn't mention a config, we assume it's the same as the previous paper unless there's a specific reason to not do so. Assumed values, whether becasue of the above principle or because they've been inferred from materials but not explicitly stated, are in italics
My primary goal is to document best practices and how they evolve over time. Therefore if something was supposed to be done but failed to due to a bug, I list what was intended. All such examples have a note disclaiming them when they occur
There are close to 100 LMs that have been trained in the past three years. For ease of use I've been focusing on what I consider to be central examples that people look to to inform their decision-making. If there are models you'd like added, feel free to leave a comment! Some of this info has been divined from staring at model inference code for a long time. I'm reasonably confidant it's correct, but please comment with any corrections (w/ a source!)