ImageNet Classification with Deep Convolutional Neural Networks
A deep convolutional network demonstrated how data, GPU computation and learned representations could transform visual recognition.
THE RESEARCH ATLAS
Training Compute-Optimal Large Language Models
The study examines how to allocate a training-compute budget between model size and training data.
Institution and location sources
Location refers to credited institutions or the listed organization, not author citizenship or an exhaustive collaboration list.
Why read this
A foundation for reasoning about training budgets rather than parameter counts alone.
Citation
@misc{hoffmann2022training,
title = {Training Compute-Optimal Large Language Models},
author = {Jordan Hoffmann and Sebastian Borgeaud and Arthur Mensch and Elena Buchatskaya and Trevor Cai and Eliza Rutherford and Diego de Las Casas and Lisa Anne Hendricks and Johannes Welbl and Aidan Clark and Tom Hennigan and Eric Noland and Katie Millican and George van den Driessche and Bogdan Damoc and Aurelia Guy and Simon Osindero and Karen Simonyan and Erich Elsen and Jack W. Rae and Oriol Vinyals and Laurent Sifre},
year = {2022},
eprint = {2203.15556},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2203.15556}
}Rights remain with the original authors. Refer to the original source for citations and licensing.
Related research
Attention Is All You Need
The Transformer replaces recurrent sequence processing with attention, creating a foundation for many later language and multimodal models.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Masked-language pretraining learns bidirectional representations that can be adapted to language-understanding tasks.