Papers

3,000 expert-written questions across 100 subjects, built to outlast saturated benchmarks.

Reasoning emerges from reinforcement learning with verifiable rewards, no supervised chain-of-thought data. Open weights, MIT licence.

A 671B mixture-of-experts model trained for about 2.8 million H800 GPU-hours. Multi-head latent attention and FP8 training.

Llama 3.1 up to 405B, trained on 15 trillion tokens. The most detailed frontier training report published.

Mixtral 8x7B: an open mixture-of-experts model that matched GPT-3.5.

Google DeepMind's first Gemini report. Multimodal from the start.

A state space model with input-dependent gating that matches Transformers at linear cost.

448 questions written by PhDs that experts get 65% on. The hard science benchmark.

2,294 real GitHub issues. The benchmark for coding agents.

A 7B model that beat Llama 2 13B. Grouped-query and sliding-window attention. Released as a torrent link.

Meta's second generation, released with a licence for commercial use. The open-weights default of 2023.

Phi-1: a 1.3B code model trained on synthetic textbook-quality data. Data quality over size.

Use a strong model to grade others, and crowdsource pairwise votes. Chatbot Arena became the leaderboard that mattered.

Preference tuning without a separate reward model or RL loop. Replaced RLHF in most open models.

Fine-tune a 65B model on one 48GB GPU with 4-bit weights and LoRA adapters.

A promptable segmentation model trained on a billion masks.

Results without architecture, data or compute. The report that marked the end of open frontier research.

Meta's 7B to 65B models trained on public data. The weights leaked within a week and the open model ecosystem began.

A model learns when to call a calculator, search or calendar API.

Anthropic replaces human harmlessness labels with a written constitution and AI feedback. How Claude is trained.

Whisper: speech recognition trained on 680,000 hours of web audio. Open weights.

A thousand researchers train an open GPT-3-scale model on a French supercomputer.

5.85 billion image-text pairs scraped from the web. Stable Diffusion's training data.

Interleave thoughts and tool calls. The loop that most agent frameworks run.

BIG-bench: 204 tasks from 450 authors.

Compute attention in GPU SRAM tiles. Exact, and several times faster. In every training stack now.

Imagen: a big frozen text encoder matters more than a big image model.

Bolts a vision encoder onto a frozen language model. The template for multimodal LLMs.

DALL-E 2: diffusion in CLIP space. Photorealistic text-to-image reaches the public.

Google's 540B model. Showed discontinuous jumps in ability with scale.

Chinchilla: for a fixed compute budget, train a smaller model on more tokens. Reset the scaling recipe.

InstructGPT: supervised fine-tuning then RLHF on GPT-3. A 1.3B model people prefer to the 175B base. ChatGPT's method.

Ask the model to show its steps and it gets maths right. The seed of reasoning models.

Run diffusion in a compressed latent space. Stable Diffusion is this paper's model.

DeepMind's 280B model and a careful study of what scale does and does not fix.

GSM8K: 8,500 grade-school maths problems. The standard reasoning benchmark until it saturated.

AlphaFold 2 predicts protein structures to near experimental accuracy. Nobel Prize in Chemistry 2024.

Codex, the model behind GitHub Copilot, and HumanEval, the benchmark for it.

Fine-tune by training small low-rank matrices. The reason fine-tuning fits on one GPU.

Costs, bias and the illusion of understanding in large language models. The critique everyone cites.

CLIP: match images and captions with contrastive learning. The text encoder inside Stable Diffusion.

DALL-E: a Transformer that generates images from text, token by token.

Mixture of experts with one expert per token. The design behind Mixtral, DeepSeek and most frontier models.

EleutherAI's open training corpus. The base for most open models of 2021 to 2023.

The Vision Transformer. Patches as tokens. Transformers take over vision too.

57 subjects of multiple-choice questions. The headline benchmark for language models from 2021 to 2024.

Train a network to remove noise step by step. Diffusion models become competitive with GANs.

GPT-3, 175B parameters. Shows that a large enough model learns new tasks from a few examples in the prompt.

Fetch documents, then generate. Named the pattern that most enterprise AI apps use.

A network that stores a 3D scene and renders new views of it.

Loss falls as a power law in parameters, data and compute. The paper that justified GPT-3.

Chollet defines intelligence as skill-acquisition efficiency and introduces the ARC benchmark.

T5 and the C4 dataset. Every task as text in, text out.

Nvidia's tensor-parallel training. How you spread one model across many GPUs.

GPT-2: a 1.5B-parameter model that does tasks it was never trained on. OpenAI held back the weights at first.

Masked language modelling on a Transformer encoder. Set new records on eleven tasks and became the default NLP model.

GPT-1: pretrain a Transformer decoder on text, then fine-tune. Generative pretraining works.

Nine language understanding tasks in one score. Saturated within two years.

ELMo: word vectors that depend on context, from a bidirectional LSTM language model. The step before BERT.

Train a reward model from human comparisons, then optimise against it. The recipe behind ChatGPT.

Drops recurrence for self-attention. Every large language model since is a Transformer.

Policy and value networks plus Monte Carlo tree search beat the European Go champion 5-0. Two months later, Lee Sedol.

Skip connections let networks go 152 layers deep. The most cited paper of the decade.

LeCun, Bengio and Hinton's Nature review. The field's statement of what it had achieved by 2015.

Normalise activations per mini-batch. Made deep networks train faster and more reliably.

The default optimizer for deep learning. Adaptive learning rates from moment estimates.

An LSTM encoder and decoder translate whole sentences. The encoder-decoder pattern for everything after.

Adds attention to the encoder-decoder so the decoder looks back at the source. The idea the Transformer built on.

A generator and a discriminator trained against each other. Dominated image generation until diffusion.

Randomly drop units during training. One of the tricks that made AlexNet work, written up in full.

Stanford's word vectors from global co-occurrence counts. The other standard embedding of the 2010s.

The variational autoencoder and the reparameterisation trick.

A convolutional network learns seven Atari games from pixels with Q-learning and experience replay. DeepMind's first hit.

word2vec: skip-gram and CBOW embeddings trained fast on billions of words. Vectors with arithmetic.

AlexNet wins ImageNet 2012 by ten points using two GPUs, ReLU and dropout. The start of the deep learning era.

Fourteen million labelled images organised by WordNet. The dataset that made deep vision possible.

Layer-wise pretraining makes deep networks trainable again. The paper that gave deep learning its name.

Breiman's ensemble of decision trees on random feature subsets. Still a default for tabular data.

LeNet-5 and the MNIST benchmark. The reference CNN for a decade.

Gated memory cells that keep gradients alive over long sequences. The workhorse of sequence modelling until the Transformer.

Cortes and Vapnik's support vector machine. The dominant classifier of the late 1990s and 2000s.

Watkins and Dayan prove Q-learning converges. The algorithm that DQN scaled up twenty years later.

Convolutional network trained with backpropagation reads zip codes from US Mail. The first practical CNN.

Hornik, Stinchcombe and White prove a network with one hidden layer can approximate any continuous function.

Sutton's TD learning: update predictions from later predictions, not only from final outcomes.

The paper that made backpropagation standard. Hidden units learn useful internal representations.

Stochastic networks with hidden units that learn a probability distribution. Ackley, Hinton and Sejnowski.

A recurrent network that stores patterns as energy minima and recalls them from partial input. Brought physicists into neural networks.

A hierarchy of local feature detectors and pooling layers. The ancestor of the convolutional network.

Harvard PhD thesis that describes backpropagation for multilayer networks.

Minsky and Papert prove single-layer perceptrons cannot compute XOR. Funding for neural networks dried up for over a decade.

A checkers program that improves by playing itself. Uses the phrase machine learning and an early form of temporal-difference learning.

Introduces the perceptron and its learning rule. The first trainable classifier.

Asks whether machines can think and replaces the question with the imitation game. Also predicts learning machines.

Defines information, entropy and channel capacity. Cross-entropy loss, the objective of every language model, comes from here.

Models the neuron as a threshold logic unit and shows networks of them can compute any logical function. The first artificial neuron.