# Google MaxText 弹性训练：训练中途终止 TPU，数秒内恢复

- 来源：Google Developers Blog（RSS）
- 发布时间：2026-07-07 00:12
- AIHOT 分数：40
- AIHOT 链接：https://aihot.virxact.com/items/cmr9f5qtk03o5slsm4qnq1x4i
- 原文链接：https://developers.googleblog.com/we-terminated-a-tpu-mid-training-and-it-recovered-in-seconds-introduction-to-elastic-training-with-maxtext

## AI 摘要

分布式 AI 训练常因单台机器故障导致整个多节点作业崩溃，需耗时重启。Google 的 JAX 生态通过 Pathways 实现弹性训练，将硬件故障转化为可捕获的 Python 异常，使运行进程存活。测试中，训练中途终止一个 TPU 后，系统自动替换损坏的节点、从 Cloud Storage 恢复最新检查点并原地恢复训练，总停机时间低于 2 分钟，主控制器进程全程无需重启。

## 正文

Distributed AI training is notoriously fragile because losing a single machine typically crashes the entire multi-node job, forcing a time-consuming, full-workload infrastructure restart. To address this, Google’s JAX ecosystem utilizes elastic training via Pathways, which converts a hardware failure into a catchable Python exception so the running process can survive. When an unplanned failure occurs, the system automatically replaces only the broken worker, restores the last viable checkpoint from Cloud Storage, and resumes training in place—minimizing total downtime to under two minutes without ever restarting the main controller process.