ImageNet Classification with Deep Convolutional Neural Networks
A deep convolutional network demonstrated how data, GPU computation and learned representations could transform visual recognition.
THE RESEARCH ATLAS
Mixtral of Experts
Introduces a sparse mixture-of-experts model exploring model capacity and activated computation.
Institution and location sources
Location refers to credited institutions or the listed organization, not author citizenship or an exhaustive collaboration list.
Why read this
Understand sparse activation as a route to efficient model scaling.
Citation
@misc{jiang2024mixtral,
title = {Mixtral of Experts},
author = {Albert Q. Jiang and Alexandre Sablayrolles and Antoine Roux and Arthur Mensch and Blanche Savary and Chris Bamford and Devendra Singh Chaplot and Diego de las Casas and Emma Bou Hanna and Florian Bressand and Gianna Lengyel and Guillaume Bour and Guillaume Lample and Lélio Renard Lavaud and Lucile Saulnier and Marie-Anne Lachaux and Pierre Stock and Sandeep Subramanian and Sophia Yang and Szymon Antoniak and Teven Le Scao and Théophile Gervet and Thibaut Lavril and Thomas Wang and Timothée Lacroix and William El Sayed},
year = {2024},
eprint = {2401.04088},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2401.04088}
}Rights remain with the original authors. Refer to the original source for citations and licensing.
Related research
Attention Is All You Need
The Transformer replaces recurrent sequence processing with attention, creating a foundation for many later language and multimodal models.
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Masked-language pretraining learns bidirectional representations that can be adapted to language-understanding tasks.