在单张 L40S 上，用 LoRA 微调 Qwen2.5-1.5B 在 GSM8K 上达到 ≥ 57% 的准确率，能有多快？
这是用于微调的 modded-nanogpt：固定任务、固定硬件，以及一个公开的挂钟时间排行榜。每条记录都必须在相同硬件上使用全新随机种子独立重复运行 3 次，方可计入榜单。
尝试和验证都是免费的：官方计时运行在 Modal L40S 沙盒上，而 Modal 的免费月度计算额度足以覆盖完整运行——因此任何人都可以参与竞争，任何人都可以用一条命令重新验证任何一条记录。
排行榜
赛道 1 — GSM8K · Qwen2.5-1.5B · 目标 ≥ 57.0% · 1× L40S
当前记录：6 分 05 秒，由 @Saivineeth147 创造——序列打包 + 仅补全损失掩码，2 个 epoch。LoRA 配置与 #0 相同；在更高精度下速度提升约 2 倍。
| # | 日期 | 作者 | 训练时间 | GSM8K/EM | Δ | 技术 |
|---|---|---|---|---|---|---|
| 0 | 2026-07-18 | @Saivineeth147 | 11 分 57 秒 | 59.4% | — | 基线：在所有线性层上使用普通 LoRA r=16，3 个 epoch，余弦学习率。无技巧。（报告） |
| 1 | 2026-07-18 | @Saivineeth147 | 6 分 05 秒 | 61.1% | −49% | 序列打包 + 仅补全损失掩码，2 个 epoch。LoRA 配置与 #0 相同；在更高精度下速度提升约 2 倍。（报告） |
赛道 2 — SQuAD v1.1 · SmolLM2-1.7B · 目标 ≥ 75.5% · 1× L40S
当前记录：11 分 08 秒，由 @Saivineeth147 创造——赛道 2 基线：在 SQuAD 上使用普通 LoRA r=16，前 20k 个样本，1 个 epoch，全序列损失。无技巧。
| # | 日期 | 作者 | 训练时间 | GSM8K/EM | Δ | 技术 |
|---|---|---|---|---|---|---|
| 0 | 2026-07-20 | @Saivineeth147 | 11 分 08 秒 | 77.5% | — | 赛道 2 基线：在 SQuAD 上使用普通 LoRA r=16，前 20k 个样本，1 个 epoch，全序列损失。无技巧。（报告） |
完整历史记录及验证报告：records/RECORDS.md
赛道说明
两条固定赛道，特意选择了不同的模型系列和任务类型——因此，一项技术只有在两条赛道上都获胜，才能证明其通用性。所有赛道使用相同的硬件、上限和验证方式。
| 赛道 1 | 赛道 2 | |
|---|---|---|
| 基础模型 | Qwen/Qwen2.5-1.5B | HuggingFaceTB/SmolLM2-1.7B |
| 任务 | GSM8K 数学 → ≥ 57.0% 精确匹配 | SQuAD v1.1 问答 → ≥ 75.5% 精确匹配 |
| 训练数据 | 仅 GSM8K 训练集 | 仅 SQuAD 训练集 |
| 评估指标 | 训练挂钟时间。时间越短越好。 | 相同 |
| 硬件 | 1× L40S（48 GB），Modal 沙盒 | 相同 |
| 约束条件 | 仅适配器，可训练参数 ≤ 30M | 相同 |
机器可读规格：spec.yaml · spec-t2.yaml。完整规则：TASK.md。
你控制其他一切：LoRA 的秩与放置位置、量化方式、学习率调度、序列打包、数据子集的选择与排序、自定义内核，以及何时停止训练。如果你能明确达标标准，就可以在精心挑选的 1000 个样本上训练 90 秒。
为什么会有这个项目
LoRA/QLoRA 是大多数实际微调的真实方式，而相关技术空间已经爆炸式发展——DoRA、rsLoRA、PiSSA、LoRA+、NEFTune、Unsloth 内核、秩自适应方法——但缺乏一个对抗性的、公平对等的竞技场，让这些想法在公开场合相互比拼。论文在不同模型、数据和硬件上报告结果；没有任何东西是可比较的。
nanoGPT 速度挑战赛解决了预训练领域的问题，并产出了真正的科学成果（Muon 优化器就源于此）。这个代码库为参数高效微调做了同样的事情：一个固定的任务，一块 GPU，按挂钟时间计，且需要提供凭证。
快速开始
官方硬件运行（免费）。注册一个 Modal 账户，然后：
git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun
pip install modal pyyaml && modal setup # one-time browser auth
python harness/modal_verify.py --prefetch # one-time: cache model + data in a volume
# one timed, evaluated attempt of the baseline on the exact spec hardware:
python harness/modal_verify.py --submission submissions/000-baseline --runs 1
# full record-style verification (3 fresh seeds, all must pass):
python harness/modal_verify.py --submission submissions/000-baseline --runs 3
本地迭代（可选）。任何 24 GB 以上的显卡都能运行基线版本，便于快速实验——运行 `bash scripts/setup_gpu.sh`，然后执行 `python harness/run_submission.py submissions/000-baseline --runs 1`。本地时间不计入官方成绩；排行榜的计时标准是 Modal L40S。
然后复制 `submissions/TEMPLATE/`，让它跑得更快，并提交一个 PR。详见 CONTRIBUTING.md。
记录如何被验证
一次真实的验证过程，从日志中回放（种子 463953844，时间压缩）：训练 → 完整性及适配器审计 → 评估 → 三个种子的裁决。
- 你提交一个 PR，包含你的训练脚本、配置、说明以及自报的成绩。
- CI 进行静态验证，然后一个自动化的 Claude 安全审查会检查代码差异（数据外泄尝试、网络使用、对测试框架的篡改、接触测试集），并将其发现公开发布。
- 一位维护者审查代码，然后评论 `/verify`——这会在一个网络隔离的 Modal 沙箱中，在指定的 L40S 上，用全新的种子将你的提交重新运行 3 次。所有 3 次运行都必须达到目标；官方时间取平均值。
- 测试框架会审计适配器的参数数量，并重新验证模型/数据内容的哈希值（防篡改），验证报告会发布在 PR 上，并连同接受/拒绝的理由一起提交到 `records/verifications/` 目录下。
完整协议、评分标准及威胁模型：JUDGING.md · SECURITY.md。
尚未有人认领的想法
目前已认领：序列打包 + 仅补全掩码（记录 #1）。
1轮次激进学习率调度 · 数据剪枝（仅训练最难的2000个样本？） · 块对角/变长序列打包注意力 · QLoRA NF4与bf16的权衡 · rsLoRA / DoRA / PiSSA初始化 · LoRA+（A/B矩阵非对称学习率） · NEFTune噪声 · 课程学习顺序 · 秩/放置位置搜索（仅MLP vs 仅注意力） · torch.compile · Unsloth内核 · Liger内核 · 融合交叉熵 · 短训练轮次的更优预热策略
认领一个想法，击败6分05秒的记录，你的名字就能上榜。
常见问题解答
这些技巧难道不会过拟合到某一个模型和某一个任务上吗？这正是为什么设置两个赛道，使用不同的模型族和任务类型——后续还会有更多遵循相同冻结与校准协议的赛道加入。一个只在单一赛道有效的技巧可以算作一项记录，但真正值得信赖的技术是那些能够迁移的。赛道系统将这个问题变成了一个实证问题，而非一场争论。
为什么选择Qwen2.5-1.5B？它不是在数学数据上预训练的吗？很可能，就像所有现代基础模型一样。但这并不重要：目标只是一个锚点，而非关于数学发现的声明。比赛过程本身才是关键——这与nanoGPT速通比赛将任意验证损失作为目标的原因相同。（赛道2使用了不同的模型族SmolLM2，部分原因就在于此。）
为什么使用挂钟时间而不是FLOPs或训练步数？因为挂钟时间才是你实际需要付费的，它迫使内核、数据加载和算法在同一个衡量标准下竞争。这与modded-nanogpt的规则相同。
为什么在Modal上使用L40S，而不是4090或H100？有三个原因。首先，它是一个统一的数据中心SKU，因此时间具有实际可比性（租用的消费级显卡性能因主机而异）。其次，通过Modal的月度免费额度可以免费使用，因此参赛和重新验证无需任何成本。最后，提交的是陌生人的代码——Modal沙箱会在网络阻断且无密钥的环境下运行它们。（L40S与4090采用相同的AD102芯片，因此消费级GPU的技巧可以迁移。）
我可以在其他数据上训练，或者从更大的模型中蒸馏吗？不行。仅限GSM8K训练集，不允许使用教师模型，不允许使用合成数据。完整禁止列表请参见TASK.md。
可以使用多个GPU吗？不行。仅限一块L40S。这就是比赛的意义所在。
许可证
麻省理工学院。记录、报告和书面材料与代码一样公开。
How fast can you LoRA-fine-tune Qwen2.5-1.5B to ≥ 57% on GSM8K — on a single L40S?
This is modded-nanogpt for fine-tuning: a frozen task, frozen hardware, and a public leaderboard of wall-clock records. Every record is independently re-run 3× with fresh seeds on identical hardware before it counts.
Attempting and verifying are free: official timing runs on a Modal L40S sandbox, and Modal's free monthly compute credits cover full runs — so anyone can compete, and anyone can re-verify any record with one command.
Leaderboard
Track 1 — GSM8K · Qwen2.5-1.5B · target ≥ 57.0% · 1× L40S
Current record: 6m 05s by @Saivineeth147 — Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy.
| # | Date | Author | Train time | GSM8K/EM | Δ | Technique |
|---|---|---|---|---|---|---|
| 0 | 2026-07-18 | @Saivineeth147 | 11m 57s | 59.4% | — | Baseline: plain LoRA r=16 on all linear layers, 3 epochs, cosine LR. No tricks. (report) |
| 1 | 2026-07-18 | @Saivineeth147 | 6m 05s | 61.1% | −49% | Sequence packing + completion-only loss masking, 2 epochs. Same LoRA config as #0; ~2x faster at higher accuracy. (report) |
Track 2 — SQuAD v1.1 · SmolLM2-1.7B · target ≥ 75.5% · 1× L40S
Current record: 11m 08s by @Saivineeth147 — Track 2 baseline: plain LoRA r=16 on SQuAD, first 20k examples, 1 epoch, full-sequence loss. No tricks.
| # | Date | Author | Train time | GSM8K/EM | Δ | Technique |
|---|---|---|---|---|---|---|
| 0 | 2026-07-20 | @Saivineeth147 | 11m 08s | 77.5% | — | Track 2 baseline: plain LoRA r=16 on SQuAD, first 20k examples, 1 epoch, full-sequence loss. No tricks. (report) |
Full history with verification reports: records/RECORDS.md
The tracks
Two frozen tracks, deliberately different model families and task types — so a technique only proves general by winning on both. Same hardware, caps, and verification everywhere.
| Track 1 | Track 2 | |
|---|---|---|
| Base model | Qwen/Qwen2.5-1.5B | HuggingFaceTB/SmolLM2-1.7B |
| Task | GSM8K math → ≥ 57.0% exact-match | SQuAD v1.1 QA → ≥ 75.5% EM |
| Training data | GSM8K train split only | SQuAD train split only |
| Metric | Training wall-clock. Lower wins. | same |
| Hardware | 1× L40S (48 GB), Modal sandbox | same |
| Constraint | adapter-only, ≤ 30M trainable params | same |
Machine-readable specs: spec.yaml · spec-t2.yaml. Full rules: TASK.md.
You control everything else: LoRA rank and placement, quantization, learning-rate schedules, sequence packing, data subset selection and ordering, custom kernels, when to stop. Train on 1,000 well-chosen examples for 90 seconds if you can make it clear the bar.
Why this exists
LoRA/QLoRA is how most real-world fine-tuning actually happens, and the technique space exploded — DoRA, rsLoRA, PiSSA, LoRA+, NEFTune, Unsloth kernels, rank-adaptive methods — but there's no adversarial, apples-to-apples arena where these ideas race each other in public. Papers report numbers on different models, data, and hardware; nothing is comparable.
The nanoGPT speedrun fixed this for pretraining and produced real science (Muon came out of it). This repo does the same for parameter-efficient fine-tuning: one frozen task, one GPU, wall-clock time, receipts required.
Quickstart
Official-hardware run (free). Make a Modal account, then:
git clone https://github.com/Saivineeth147/lora-speedrun && cd lora-speedrun
pip install modal pyyaml && modal setup # one-time browser auth
python harness/modal_verify.py --prefetch # one-time: cache model + data in a volume
# one timed, evaluated attempt of the baseline on the exact spec hardware:
python harness/modal_verify.py --submission submissions/000-baseline --runs 1
# full record-style verification (3 fresh seeds, all must pass):
python harness/modal_verify.py --submission submissions/000-baseline --runs 3
Local iteration (optional). Any 24 GB+ card runs the baseline for fast experimenting — bash scripts/setup_gpu.sh, then python harness/run_submission.py submissions/000-baseline --runs 1. Local times aren't official; the leaderboard clock is the Modal L40S.
Then copy submissions/TEMPLATE/, make it faster, and open a PR. See CONTRIBUTING.md.
How records get verified
A real verification, replayed from the logs (seed 463953844, time-compressed): train → integrity + adapter audit → eval → 3-seed verdict.
- You open a PR with your training script, config, notes, and self-reported numbers.
- CI statically validates it, and an automated Claude security screen reviews the diff (exfiltration attempts, network use, harness tampering, test-set contact) and posts its findings publicly.
- A maintainer reviews the code, then comments
/verify— which re-runs your submission 3× with fresh seeds in a network-blocked Modal sandbox on the spec L40S. All 3 runs must clear the target; official time is the mean. - The harness audits the adapter param count and re-verifies model/data content hashes (anti-tampering), and the verification report is posted on the PR and committed to records/verifications/ with the accept/reject reasoning.
Full protocol, rubric, and threat model: JUDGING.md · SECURITY.md.
Ideas nobody has claimed yet
Taken so far: sequence packing + completion-only masking (record #1).
1-epoch aggressive-LR schedules · data pruning (train on the hardest 2k examples?) · block-diagonal/varlen packing attention · QLoRA NF4 vs bf16 tradeoff · rsLoRA / DoRA / PiSSA init · LoRA+ (asymmetric LR for A/B) · NEFTune noise · curriculum ordering · rank/placement search (MLP-only vs attention-only) · torch.compile · Unsloth kernels · Liger kernels · fused cross-entropy · smarter warmup for short runs
Claim one, beat 6m 05s, get your name on the board.
FAQ
Won't techniques overfit to one model + one task? That's exactly why there are two tracks with different model families and task types — and more will follow the same freeze-and-calibrate protocol. A trick that only wins on one track is a record, but the techniques worth trusting are the ones that transfer. The track system makes that an empirical question instead of an argument.
Why Qwen2.5-1.5B? Isn't it pretrained on math? Probably, like every modern base model. It doesn't matter: the target is an anchor, not a claim about mathematical discovery. The race is the interesting part — same reason nanoGPT speedrunning targets an arbitrary val loss. (Track 2 uses a different family, SmolLM2, partly for this reason.)
Why wall-clock instead of FLOPs or steps? Because wall-clock is what you pay for, and it forces kernels, data loading, and algorithms to compete in the same currency. Same rule as modded-nanogpt.
Why an L40S on Modal instead of a 4090 or H100? Three reasons. It's one consistent datacenter SKU, so times are actually comparable (rented consumer cards vary host-to-host). It's free to use via Modal's monthly credits, so competing and re-verifying costs nothing. And submissions are strangers' code — Modal sandboxes run them network-blocked and secretless. (The L40S is the same AD102 silicon as the 4090, so consumer-GPU tricks transfer.)
Can I train on other data / distill from a bigger model? No. GSM8K train split only, no teacher models, no synthetic data. See TASK.md for the full banned list.
Multiple GPUs? No. One L40S. That's the point.
License
MIT. Records, reports, and write-ups are as public as the code.