Contributors

• Andrej Karpathy
• Fei-Fei Li
• +many other researchers

CS231n: Convolutional Neural Networks for Visual Recognition

The course notes that teach convolutional networks from first principles, from linear classifiers up to deep architectures for images.

CS231n: Convolutional Neural Networks for Visual Recognition

The course notes that teach convolutional networks from first principles, from linear classifiers up to deep architectures for images.

Andrej KarpathyFei-Fei Li Plus many more

Contributors

• Andrej Karpathy

The Unreasonable Effectiveness of Recurrent Neural Networks

A hands on blog post that trains character level RNNs to generate text and shows, with vivid examples, how much structure they capture.

The Unreasonable Effectiveness of Recurrent Neural Networks

A hands on blog post that trains character level RNNs to generate text and shows, with vivid examples, how much structure they capture.

Andrej Karpathy

Contributors

• Christopher Olah

Understanding LSTM Networks

The clearest visual explanation of how LSTM gates carry information across long sequences, widely used as a first introduction.

Understanding LSTM Networks

The clearest visual explanation of how LSTM gates carry information across long sequences, widely used as a first introduction.

Christopher Olah

Contributors

• Alex Krizhevsky
• Ilya Sutskever
• Geoffrey Hinton

ImageNet Classification with Deep Convolutional Neural Networks

AlexNet. The convolutional network that won ImageNet by a wide margin and set off the modern deep learning era.

ImageNet Classification with Deep Convolutional Neural Networks

AlexNet. The convolutional network that won ImageNet by a wide margin and set off the modern deep learning era.

Alex KrizhevskyIlya Sutskever

Contributors

• Kaiming He
• Xiangyu Zhang
• Shaoqing Ren
• JS Jian Sun

Deep Residual Learning for Image Recognition

ResNet. Introduces residual connections that let networks grow to hundreds of layers by learning changes to the input rather than full transformations.

Deep Residual Learning for Image Recognition

ResNet. Introduces residual connections that let networks grow to hundreds of layers by learning changes to the input rather than full transformations.

Kaiming HeXiangyu Zhang

Contributors

• FY Fisher Yu
• Vladlen Koltun

Multi-Scale Context Aggregation by Dilated Convolutions

Shows how dilated convolutions expand the receptive field without losing resolution, which sharpened dense prediction tasks like segmentation.

Multi-Scale Context Aggregation by Dilated Convolutions

Shows how dilated convolutions expand the receptive field without losing resolution, which sharpened dense prediction tasks like segmentation.

FY Fisher YuVladlen Koltun

Contributors

• Kaiming He
• Xiangyu Zhang
• Shaoqing Ren
• JS Jian Sun

Identity Mappings in Deep Residual Networks

A follow up to ResNet that studies why identity shortcuts work so well and proposes a cleaner pre-activation residual block.

Identity Mappings in Deep Residual Networks

A follow up to ResNet that studies why identity shortcuts work so well and proposes a cleaner pre-activation residual block.

Kaiming HeXiangyu Zhang

Contributors

• Wojciech Zaremba
• Ilya Sutskever
• Oriol Vinyals

Recurrent Neural Network Regularization

Shows how to apply dropout to LSTMs correctly, on the non-recurrent connections, so large recurrent models stop overfitting.

Recurrent Neural Network Regularization

Shows how to apply dropout to LSTMs correctly, on the non-recurrent connections, so large recurrent models stop overfitting.

Wojciech ZarembaIlya Sutskever

Contributors

• Dario Amodei
• RA Rishita Anubhai
• +many other researchers

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

An end to end speech recognition system trained with connectionist temporal classification that worked across two very different languages.

Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

An end to end speech recognition system trained with connectionist temporal classification that worked across two very different languages.

Dario Amodei RA Rishita Anubhai Plus many more

Contributors

• Oriol Vinyals
• Samy Bengio
• MK Manjunath Kudlur

Order Matters: Sequence to Sequence for Sets

Examines how the order of inputs and outputs affects sequence to sequence models, and how to handle data that is really a set.

Order Matters: Sequence to Sequence for Sets

Examines how the order of inputs and outputs affects sequence to sequence models, and how to handle data that is really a set.

Oriol VinyalsSamy Bengio

Contributors

• Dzmitry Bahdanau
• Kyunghyun Cho
• Yoshua Bengio

Neural Machine Translation by Jointly Learning to Align and Translate

Introduces the attention mechanism, letting a translation model look back at the relevant source words instead of a single fixed summary.

Neural Machine Translation by Jointly Learning to Align and Translate

Introduces the attention mechanism, letting a translation model look back at the relevant source words instead of a single fixed summary.

Dzmitry BahdanauKyunghyun Cho

Contributors

• Oriol Vinyals
• Meire Fortunato
• Navdeep Jaitly

Pointer Networks

A sequence model whose outputs point back at positions in the input, which suits problems whose answer is a selection or ordering of the inputs.

Pointer Networks

A sequence model whose outputs point back at positions in the input, which suits problems whose answer is a selection or ordering of the inputs.

Oriol VinyalsMeire Fortunato

Contributors

• Ashish Vaswani
• Noam Shazeer
• +many other researchers

Attention Is All You Need

The Transformer. Replaces recurrence entirely with self attention, the architecture that underpins almost every modern large language model.

Attention Is All You Need

The Transformer. Replaces recurrence entirely with self attention, the architecture that underpins almost every modern large language model.

Ashish VaswaniNoam Shazeer Plus many more

Contributors

• Sasha Rush
• +many other researchers

The Annotated Transformer

A line by line, runnable reimplementation of the Transformer that turns the original paper into working, readable code.

The Annotated Transformer

A line by line, runnable reimplementation of the Transformer that turns the original paper into working, readable code.

Sasha Rush Plus many more

Contributors

• Alex Graves
• GW Greg Wayne
• ID Ivo Danihelka

Neural Turing Machines

Couples a neural network to an external memory it can read and write with differentiable attention, learning simple algorithms from examples.

Neural Turing Machines

Couples a neural network to an external memory it can read and write with differentiable attention, learning simple algorithms from examples.

Alex Graves GW Greg Wayne

Contributors

• Adam Santoro
• DR David Raposo
• +many other researchers

A Simple Neural Network Module for Relational Reasoning

Introduces the relation network, a small plug in module that lets a network reason about how pairs of objects relate to each other.

A Simple Neural Network Module for Relational Reasoning

Introduces the relation network, a small plug in module that lets a network reason about how pairs of objects relate to each other.

Adam Santoro DR David Raposo Plus many more

Contributors

• Adam Santoro
• RF Ryan Faulkner
• +many other researchers

Relational Recurrent Neural Networks

Adds a self attention based memory to recurrent networks so that stored memories can interact, improving tasks that need relational reasoning over time.

Relational Recurrent Neural Networks

Adds a self attention based memory to recurrent networks so that stored memories can interact, improving tasks that need relational reasoning over time.

Adam Santoro RF Ryan Faulkner Plus many more

Contributors

• Justin Gilmer
• SS Samuel S. Schoenholz
• PF Patrick F. Riley
• Oriol Vinyals
• D Dahl

Neural Message Passing for Quantum Chemistry

Unifies many graph neural networks under a message passing framework and applies it to predicting molecular properties.

Neural Message Passing for Quantum Chemistry

Unifies many graph neural networks under a message passing framework and applies it to predicting molecular properties.

Justin Gilmer SS Samuel S. Schoenholz

Contributors

• Jared Kaplan
• SM Sam McCandlish
• +many other researchers

Scaling Laws for Neural Language Models

Measures how language model loss falls as a smooth power law in model size, data, and compute, the empirical basis for building ever larger models.

Scaling Laws for Neural Language Models

Measures how language model loss falls as a smooth power law in model size, data, and compute, the empirical basis for building ever larger models.

Jared Kaplan SM Sam McCandlish Plus many more

Contributors

• YH Yanping Huang
• YC Youlong Cheng
• +many other researchers

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

A pipeline parallelism library that splits a giant model across devices and keeps them busy, making it practical to train very large networks.

GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

A pipeline parallelism library that splits a giant model across devices and keeps them busy, making it practical to train very large networks.

YH Yanping Huang YC Youlong Cheng Plus many more

Contributors

• Geoffrey Hinton
• DV Drew van Camp

Keeping Neural Networks Simple by Minimizing the Description Length of the Weights

An early information-theoretic argument that good networks are ones whose weights can be described with few bits, linking generalization to compression.

Keeping Neural Networks Simple by Minimizing the Description Length of the Weights

An early information-theoretic argument that good networks are ones whose weights can be described with few bits, linking generalization to compression.

Geoffrey Hinton DV Drew van Camp

Contributors

• Peter Grunwald

A Tutorial Introduction to the Minimum Description Length Principle

A readable introduction to choosing models by how well they compress the data, treating learning as finding the shortest description.

A Tutorial Introduction to the Minimum Description Length Principle

A readable introduction to choosing models by how well they compress the data, treating learning as finding the shortest description.

Peter Grunwald

Contributors

• Scott Aaronson

The First Law of Complexodynamics

A blog essay asking for a formal law that explains why the complexity of a closed system rises, peaks, and falls, rather than simply tracking entropy.

The First Law of Complexodynamics

A blog essay asking for a formal law that explains why the complexity of a closed system rises, peaks, and falls, rather than simply tracking entropy.

Scott Aaronson

Contributors

• Scott Aaronson
• Sean M. Carroll
• LO Lauren Ouellette

Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton

Uses a simple cellular automaton model of coffee mixing with cream to ask why complexity rises and then falls as a system moves toward equilibrium.

Quantifying the Rise and Fall of Complexity in Closed Systems: The Coffee Automaton

Uses a simple cellular automaton model of coffee mixing with cream to ask why complexity rises and then falls as a system moves toward equilibrium.

Scott AaronsonSean M. Carroll

Contributors

• Thomas M. Cover
• JA Joy A. Thomas

Kolmogorov Complexity

The textbook treatment of the shortest program that produces a string, the formal backbone behind description length and algorithmic randomness.

Kolmogorov Complexity

The textbook treatment of the shortest program that produces a string, the formal backbone behind description length and algorithmic randomness.

Thomas M. Cover JA Joy A. Thomas

Contributors

• XC Xi Chen
• Diederik P. Kingma
• +6 6 more researchers

Variational Lossy Autoencoder

Combines variational autoencoders with autoregressive decoders, and shows how to control which information the latent code is forced to keep.

Variational Lossy Autoencoder

Combines variational autoencoders with autoregressive decoders, and shows how to control which information the latent code is forced to keep.

XC Xi ChenDiederik P. Kingma Plus 6 more

Contributors

• Shane Legg

Machine Super Intelligence

A doctoral thesis that proposes a formal, universal measure of machine intelligence and explores its consequences for very capable agents.

Machine Super Intelligence

A doctoral thesis that proposes a formal, universal measure of machine intelligence and explores its consequences for very capable agents.

Shane Legg

GitHub