Pretrain Practice (Continuously Updated)
Small scale LLM pretraining experiments based on Marin.
Experiments
Sampled Softmax with Over-Tokenized Transformer
Experiment Template
The layout and the rules every experiment page here follows: a schematic cover and its animation, the setup with a commented core diff, the experiments, and the closest related work.