Stanford / Spring 2026 ( previous offerings) Spring 2025 | Spring 2024
Course Staff
Logistics
Lectures: Monday/Wednesday 3:00-4:20pm in Skilling Auditorium
Recordings: YouTube playlist
Office hours: Percy Liang: Fridays 11am-12pm in Gates 366 Tatsu Hashimoto: Tuesdays 11-12am in Gates 364 Marcel Rød: Tuesdays 4:30-5:30pm in Gates 498, Wednesdays 4:30-5:30pm in Gates 415 Herman Brunborg: Wednesdays 1:30-2:30pm, Fridays 1:30-2:30pm, location Gates 392 Steven Cao: Mondays 4:30-5:30pm, Thursdays 9:30-10:30am, Gates 200
Percy Liang: Fridays 11am-12pm in Gates 366
Tatsu Hashimoto: Tuesdays 11-12am in Gates 364
Marcel Rød: Tuesdays 4:30-5:30pm in Gates 498, Wednesdays 4:30-5:30pm in Gates 415
Herman Brunborg: Wednesdays 1:30-2:30pm, Fridays 1:30-2:30pm, location Gates 392
Steven Cao: Mondays 4:30-5:30pm, Thursdays 9:30-10:30am, Gates 200
Contact: Students should ask all course-related questions in public Slack channels. All announcements will also be made in Slack. For personal matters, email cs336-spr2526-staff@lists.stanford.edu.
Content
What is this course about?
Language models serve as the cornerstone of modern natural language processing (NLP) applications and open up a new paradigm of having a single general purpose system address a range of downstream tasks. As the field of artificial intelligence (AI), machine learning (ML), and NLP continues to grow, possessing a deep understanding of language models becomes essential for scientists and engineers alike. This course is designed to provide students with a comprehensive understanding of language models by walking them through the entire process of developing their own. Drawing inspiration from operating systems courses that create an entire operating system from scratch, we will lead students through every aspect of language model creation, including data collection and cleaning for pre-training, transformer model construction, model training, and evaluation before deployment.
Prerequisites
Proficiency in Python The majority of class assignments will be in Python. Unlike most other AI classes, students will be given minimal scaffolding. The amount of code you will write will be at least an order of magnitude greater than for other classes. Therefore, being proficient in Python and software engineering is paramount.
The majority of class assignments will be in Python. Unlike most other AI classes, students will be given minimal scaffolding. The amount of code you will write will be at least an order of magnitude greater than for other classes. Therefore, being proficient in Python and software engineering is paramount.
Stanford / Spring 2026 ( previous offerings) Spring 2025 | Spring 2024
Course Staff
Logistics
Lectures: Monday/Wednesday 3:00-4:20pm in Skilling Auditorium
Recordings: YouTube playlist
Office hours: Percy Liang: Fridays 11am-12pm in Gates 366 Tatsu Hashimoto: Tuesdays 11-12am in Gates 364 Marcel Rød: Tuesdays 4:30-5:30pm in Gates 498, Wednesdays 4:30-5:30pm in Gates 415 Herman Brunborg: Wednesdays 1:30-2:30pm, Fridays 1:30-2:30pm, location Gates 392 Steven Cao: Mondays 4:30-5:30pm, Thursdays 9:30-10:30am, Gates 200
Percy Liang: Fridays 11am-12pm in Gates 366
Tatsu Hashimoto: Tuesdays 11-12am in Gates 364
Marcel Rød: Tuesdays 4:30-5:30pm in Gates 498, Wednesdays 4:30-5:30pm in Gates 415
Experience with deep learning and systems optimization A significant part of the course will involve making neural language models run quickly and efficiently on GPUs across multiple machines. We expect students to be able to have a strong familiarity with PyTorch and know basic systems concepts like the memory hierarchy.
A significant part of the course will involve making neural language models run quickly and efficiently on GPUs across multiple machines. We expect students to be able to have a strong familiarity with PyTorch and know basic systems concepts like the memory hierarchy.
College Calculus, Linear Algebra (e.g. MATH 51, CME 100) You should be comfortable understanding matrix/vector notation and operations.
You should be comfortable understanding matrix/vector notation and operations.
Basic Probability and Statistics (e.g. CS 109 or equivalent) You should know the basics of probabilities, Gaussian distributions, mean, standard deviation, etc.
You should know the basics of probabilities, Gaussian distributions, mean, standard deviation, etc.
Machine Learning (e.g. CS221, CS229, CS230, CS124, CS224N) You should be comfortable with the basics of machine learning and deep learning.
You should be comfortable with the basics of machine learning and deep learning.
Note that this is a 5-unit class. This is a very implementation-heavy class, so please allocate enough time for it.
Coursework
Assignments
Assignment 1: Basics Implement all of the components (tokenizer, model architecture, optimizer) necessary to train a standard Transformer language model. Train a minimal language model.
Implement all of the components (tokenizer, model architecture, optimizer) necessary to train a standard Transformer language model.
Train a minimal language model.
Assignment 2: Systems Profile and benchmark the model and layers from Assignment 1 using advanced tools, optimize Attention with your own Triton implementation of FlashAttention2. Build a memory-efficient, distributed version of the Assignment 1 model training code.
Profile and benchmark the model and layers from Assignment 1 using advanced tools, optimize Attention with your own Triton implementation of FlashAttention2.
Build a memory-efficient, distributed version of the Assignment 1 model training code.
Assignment 3: Scaling Understand the function of each component of the Transformer. Query a training API to fit a scaling law to project model scaling.
Understand the function of each component of the Transformer.
Query a training API to fit a scaling law to project model scaling.
Assignment 4: Data Convert raw Common Crawl dumps into usable pretraining data. Perform filtering and deduplication to improve model performance.
Convert raw Common Crawl dumps into usable pretraining data.
Perform filtering and deduplication to improve model performance.
Assignment 5: Alignment and Reasoning RL Apply supervised finetuning and reinforcement learning to train LMs to reason when solving math problems. Optional Part 2: implement and apply safety alignment methods such as DPO.
Apply supervised finetuning and reinforcement learning to train LMs to reason when solving math problems.
Optional Part 2: implement and apply safety alignment methods such as DPO.
GPU compute for self-study
If you are following along at home, you can access GPU compute from a cloud provider to complete the assignments.
Here are a few options (public pricing for a single B200 GPU on March 28, 2026):
Modal (sponsor): $6.25/hour. Offers $30 of free monthly compute. You are only charged for actual compute (no idle resources) and their UX makes switching between local dev and large-scale gpu experiments simple. (Modal Pricing)
For convenience and to save money, we recommend debugging correctness of your implementation on CPU first and then using GPU(s) (with the count recommended in the assignments) for completing training runs (A1, A4, A5) or benchmarking GPU operations (A2).
Honor code
Collaboration: Study groups are allowed, but students must understand and complete their own assignments, and hand in one assignment per student. If you worked in a group, please put the names of the members of your study group at the top of your assignment. Please ask if you have any questions about the collaboration policy.
AI tools: Prompting LLMs such as ChatGPT is permitted for low-level programming questions or high-level conceptual questions about language models, but using it directly to solve the problem is prohibited. We strongly encourage you to disable AI autocomplete (e.g., Cursor Tab, GitHub CoPilot) in your IDE when completing assignments (though non-AI autocomplete, e.g., autocompleting function names is totally fine). We have found that AI autocomplete makes it much harder to engage deeply with the content. See the AI policy (inspired by this).
Existing code: Implementations for many of the things you will implement exist online. The handouts we'll give will be self-contained, so that you will not need to consult third-party code for producing your own implementation. Thus, you should not look at any existing code unless when otherwise specified in the handouts.
Submitting coursework
All coursework are submitted via Gradescope by the deadline. Do not submit your coursework via email.
If anything goes wrong, please ask a question in Slack or contact a course assistant.
You can submit as many times as you'd like until the deadline: we will only grade the last submission.
Partial work is better than not submitting any work.
Late days
Each student has 6 late days to use. A late day extends the deadline by 24 hours.
You can use up to 3 late days per assignment.
Regrade requests
If you believe that the course staff made an objective error in grading, you may submit a regrade request on Gradescope within 3 days after the grades are released.
Sponsor
We would like to thank Modal for sponsoring compute for this class.
Schedule (YouTube playlist)
Date Description Course Materials Deadlines 1 Mon March 30 Overview, tokenization [Percy] lecture_01.py Assignment 1 out [code] [preview] 2 Wed April 1 PyTorch (einops), resource accounting (FLOPs, memory, arithmetic intensity) [Percy] lecture_02.py (recording version) 3 Mon April 6 Architectures, hyperparameters [Tatsu] lecture 3.pdf 4 Wed April 8 Attention alternatives and mixture of experts [Tatsu] lecture 4.pdf 5 Mon April 13 GPUs, TPUs [Tatsu] lecture 5.pdf 6 Wed April 15 Kernels, Triton [Percy] lecture_06.py Assignment 1 due Assignment 2 out [code] [preview] 7 Mon April 20 Parallelism [Percy] lecture_07.py 8 Wed April 22 Parallelism [Tatsu] lecture_08.pdf 9 Mon April 27 Scaling laws [Tatsu] lecture_09.pdf 10 Wed April 29 Inference [Percy] lecture_10.py Assignment 2 due Assignment 3 out [code] [preview] 11 Mon May 4 Scaling laws [Tatsu] lecture_11.pdf 12 Wed May 6 Evaluation [Percy] lecture_12.py Assignment 3 due Assignment 4 out [code] [preview] 13 Mon May 11 Data (sources, datasets) [Percy] lecture_13.py 14 Wed May 13 Data (filtering, deduplication, mixing, synthetic data) [Percy] lecture_14.py 15 Mon May 18 Mid/post-training (SFT/RLHF) [Tatsu] lecture_15.pdf 16 Wed May 20 Post-training - RLVR [Tatsu] lecture_16.pdf Assignment 4 due Assignment 5 out [code] [preview] [Optional Part 2] Mon May 25 No class (Memorial Day) 17 Wed May 27 Alignment - multimodality [Percy] lecture_17.py 18 Mon June 1 Guest lecture: Daniel Selsam 19 Wed June 3 Guest lecture: Dan Fu Assignment 5 due
Herman Brunborg: Wednesdays 1:30-2:30pm, Fridays 1:30-2:30pm, location Gates 392
Steven Cao: Mondays 4:30-5:30pm, Thursdays 9:30-10:30am, Gates 200
Contact: Students should ask all course-related questions in public Slack channels. All announcements will also be made in Slack. For personal matters, email cs336-spr2526-staff@lists.stanford.edu.
Content
What is this course about?
Language models serve as the cornerstone of modern natural language processing (NLP) applications and open up a new paradigm of having a single general purpose system address a range of downstream tasks. As the field of artificial intelligence (AI), machine learning (ML), and NLP continues to grow, possessing a deep understanding of language models becomes essential for scientists and engineers alike. This course is designed to provide students with a comprehensive understanding of language models by walking them through the entire process of developing their own. Drawing inspiration from operating systems courses that create an entire operating system from scratch, we will lead students through every aspect of language model creation, including data collection and cleaning for pre-training, transformer model construction, model training, and evaluation before deployment.
Prerequisites
Proficiency in Python The majority of class assignments will be in Python. Unlike most other AI classes, students will be given minimal scaffolding. The amount of code you will write will be at least an order of magnitude greater than for other classes. Therefore, being proficient in Python and software engineering is paramount.
The majority of class assignments will be in Python. Unlike most other AI classes, students will be given minimal scaffolding. The amount of code you will write will be at least an order of magnitude greater than for other classes. Therefore, being proficient in Python and software engineering is paramount.
Experience with deep learning and systems optimization A significant part of the course will involve making neural language models run quickly and efficiently on GPUs across multiple machines. We expect students to be able to have a strong familiarity with PyTorch and know basic systems concepts like the memory hierarchy.
A significant part of the course will involve making neural language models run quickly and efficiently on GPUs across multiple machines. We expect students to be able to have a strong familiarity with PyTorch and know basic systems concepts like the memory hierarchy.
College Calculus, Linear Algebra (e.g. MATH 51, CME 100) You should be comfortable understanding matrix/vector notation and operations.
You should be comfortable understanding matrix/vector notation and operations.
Basic Probability and Statistics (e.g. CS 109 or equivalent) You should know the basics of probabilities, Gaussian distributions, mean, standard deviation, etc.
You should know the basics of probabilities, Gaussian distributions, mean, standard deviation, etc.
Machine Learning (e.g. CS221, CS229, CS230, CS124, CS224N) You should be comfortable with the basics of machine learning and deep learning.
You should be comfortable with the basics of machine learning and deep learning.
Note that this is a 5-unit class. This is a very implementation-heavy class, so please allocate enough time for it.
Coursework
Assignments
Assignment 1: Basics Implement all of the components (tokenizer, model architecture, optimizer) necessary to train a standard Transformer language model. Train a minimal language model.
Implement all of the components (tokenizer, model architecture, optimizer) necessary to train a standard Transformer language model.
Train a minimal language model.
Assignment 2: Systems Profile and benchmark the model and layers from Assignment 1 using advanced tools, optimize Attention with your own Triton implementation of FlashAttention2. Build a memory-efficient, distributed version of the Assignment 1 model training code.
Profile and benchmark the model and layers from Assignment 1 using advanced tools, optimize Attention with your own Triton implementation of FlashAttention2.
Build a memory-efficient, distributed version of the Assignment 1 model training code.
Assignment 3: Scaling Understand the function of each component of the Transformer. Query a training API to fit a scaling law to project model scaling.
Understand the function of each component of the Transformer.
Query a training API to fit a scaling law to project model scaling.
Assignment 4: Data Convert raw Common Crawl dumps into usable pretraining data. Perform filtering and deduplication to improve model performance.
Convert raw Common Crawl dumps into usable pretraining data.
Perform filtering and deduplication to improve model performance.
Assignment 5: Alignment and Reasoning RL Apply supervised finetuning and reinforcement learning to train LMs to reason when solving math problems. Optional Part 2: implement and apply safety alignment methods such as DPO.
Apply supervised finetuning and reinforcement learning to train LMs to reason when solving math problems.
Optional Part 2: implement and apply safety alignment methods such as DPO.
GPU compute for self-study
If you are following along at home, you can access GPU compute from a cloud provider to complete the assignments.
Here are a few options (public pricing for a single B200 GPU on March 28, 2026):
Modal (sponsor): $6.25/hour. Offers $30 of free monthly compute. You are only charged for actual compute (no idle resources) and their UX makes switching between local dev and large-scale gpu experiments simple. (Modal Pricing)
For convenience and to save money, we recommend debugging correctness of your implementation on CPU first and then using GPU(s) (with the count recommended in the assignments) for completing training runs (A1, A4, A5) or benchmarking GPU operations (A2).
Honor code
Collaboration: Study groups are allowed, but students must understand and complete their own assignments, and hand in one assignment per student. If you worked in a group, please put the names of the members of your study group at the top of your assignment. Please ask if you have any questions about the collaboration policy.
AI tools: Prompting LLMs such as ChatGPT is permitted for low-level programming questions or high-level conceptual questions about language models, but using it directly to solve the problem is prohibited. We strongly encourage you to disable AI autocomplete (e.g., Cursor Tab, GitHub CoPilot) in your IDE when completing assignments (though non-AI autocomplete, e.g., autocompleting function names is totally fine). We have found that AI autocomplete makes it much harder to engage deeply with the content. See the AI policy (inspired by this).
Existing code: Implementations for many of the things you will implement exist online. The handouts we'll give will be self-contained, so that you will not need to consult third-party code for producing your own implementation. Thus, you should not look at any existing code unless when otherwise specified in the handouts.
Submitting coursework
All coursework are submitted via Gradescope by the deadline. Do not submit your coursework via email.
If anything goes wrong, please ask a question in Slack or contact a course assistant.
You can submit as many times as you'd like until the deadline: we will only grade the last submission.
Partial work is better than not submitting any work.
Late days
Each student has 6 late days to use. A late day extends the deadline by 24 hours.
You can use up to 3 late days per assignment.
Regrade requests
If you believe that the course staff made an objective error in grading, you may submit a regrade request on Gradescope within 3 days after the grades are released.
Sponsor
We would like to thank Modal for sponsoring compute for this class.
Schedule (YouTube playlist)
Date Description Course Materials Deadlines 1 Mon March 30 Overview, tokenization [Percy] lecture_01.py Assignment 1 out [code] [preview] 2 Wed April 1 PyTorch (einops), resource accounting (FLOPs, memory, arithmetic intensity) [Percy] lecture_02.py (recording version) 3 Mon April 6 Architectures, hyperparameters [Tatsu] lecture 3.pdf 4 Wed April 8 Attention alternatives and mixture of experts [Tatsu] lecture 4.pdf 5 Mon April 13 GPUs, TPUs [Tatsu] lecture 5.pdf 6 Wed April 15 Kernels, Triton [Percy] lecture_06.py Assignment 1 due Assignment 2 out [code] [preview] 7 Mon April 20 Parallelism [Percy] lecture_07.py 8 Wed April 22 Parallelism [Tatsu] lecture_08.pdf 9 Mon April 27 Scaling laws [Tatsu] lecture_09.pdf 10 Wed April 29 Inference [Percy] lecture_10.py Assignment 2 due Assignment 3 out [code] [preview] 11 Mon May 4 Scaling laws [Tatsu] lecture_11.pdf 12 Wed May 6 Evaluation [Percy] lecture_12.py Assignment 3 due Assignment 4 out [code] [preview] 13 Mon May 11 Data (sources, datasets) [Percy] lecture_13.py 14 Wed May 13 Data (filtering, deduplication, mixing, synthetic data) [Percy] lecture_14.py 15 Mon May 18 Mid/post-training (SFT/RLHF) [Tatsu] lecture_15.pdf 16 Wed May 20 Post-training - RLVR [Tatsu] lecture_16.pdf Assignment 4 due Assignment 5 out [code] [preview] [Optional Part 2] Mon May 25 No class (Memorial Day) 17 Wed May 27 Alignment - multimodality [Percy] lecture_17.py 18 Mon June 1 Guest lecture: Daniel Selsam 19 Wed June 3 Guest lecture: Dan Fu Assignment 5 due