这是一份关于三个间歇性导致 Claude 回复质量下降的 bug 的技术报告。下文将解释发生了什么、为何修复耗时较长，以及我们正在做出哪些改变。
在八月至九月初期间，三个基础设施 bug 间歇性地降低了 Claude 的回复质量。目前这些问题已得到解决，我们在此说明事件经过。
八月初，部分用户开始反馈 Claude 回复质量下降。这些初期报告与正常的用户反馈波动难以区分。到八月下旬，此类报告频率和持续性显著增加，促使我们展开调查，最终发现了三个独立的基础设施 bug。
明确说明：我们绝不会因需求、时段或服务器负载而降低模型质量。用户反馈的问题完全由基础设施 bug 导致。
我们深知用户期望 Claude 保持稳定的质量，并始终以极高标准确保基础设施变更不影响模型输出。在近期事件中，我们未能达到这一标准。以下事后分析将说明问题所在、为何检测和解决耗时超出预期，以及我们正在采取哪些措施防止类似事件再次发生。
我们通常不会公开如此详细的基础设施技术细节，但鉴于这些问题的规模和复杂性，有必要进行更全面的说明。
我们如何大规模提供 Claude 服务
我们通过自有 API、Amazon Bedrock 和 Google Cloud 的 Vertex AI 为数百万用户提供 Claude 服务。Claude 部署在多种硬件平台上，包括 AWS Trainium、NVIDIA GPU 和 Google TPU。这种方案提供了服务全球用户所需的容量和地理分布。
每个硬件平台具有不同特性，需要特定的优化。尽管存在这些差异，我们对模型实现有严格的等效标准。目标是无论用户请求由哪个平台处理，都能获得相同质量的回复。这种复杂性意味着任何基础设施变更都必须在所有平台和配置上进行仔细验证。
事件时间线
这些 bug 相互叠加的特性使得诊断工作尤为困难。第一个 bug 于 8 月 5 日引入，影响了约 0.8% 发往 Sonnet 4 的请求。另外两个 bug 则源于 8 月 25 日和 26 日的部署。
尽管最初影响有限，但 8 月 29 日的一次负载均衡变更开始导致受影响流量增加。这使得更多用户遇到问题，而其他用户则继续看到正常表现，从而产生了令人困惑且相互矛盾的反馈报告。
三个叠加的问题
下面我们描述导致此次故障的三个 bug、它们发生的时间以及我们如何解决它们：
1. 上下文窗口路由错误
8 月 5 日，部分 Sonnet 4 请求被错误地路由到为即将推出的 100 万 token 上下文窗口配置的服务器。此 bug 最初影响了 0.8% 的请求。8 月 29 日，一次常规的负载均衡变更无意中增加了被路由到 100 万上下文服务器的短上下文请求数量。在 8 月 31 日受影响最严重的一小时内，16% 的 Sonnet 4 请求受到了影响。
在此期间发出请求的 Claude Code 用户中，约有 30% 至少有一条消息被路由到了错误的服务器类型，从而导致响应质量下降。在 Amazon Bedrock 上，从 8 月 12 日起，错误路由的流量峰值占所有 Sonnet 4 请求的 0.18%。在 8 月 27 日至 9 月 16 日期间，Google Cloud 的 Vertex AI 上受错误路由影响的请求不到 0.0004%。
然而，部分用户受到的影响更为严重，因为我们的路由是“粘性的”。这意味着一旦某个请求由错误的服务器处理，后续的跟进请求很可能也会由同一台错误的服务器处理。
解决方案：我们修复了路由逻辑，以确保短上下文和长上下文请求被导向正确的服务器池。我们于 9 月 4 日部署了该修复。到 9 月 16 日，该修复已推广至我们的第一方平台和 Google Cloud 的 Vertex AI，并于 9 月 18 日推广至 AWS Bedrock。
2. 输出损坏
8月25日，我们在Claude API的TPU服务器上部署了一项配置错误，导致token生成过程中出现错误。一项由运行时性能优化引发的问题，偶尔会为某些token分配高概率，而这些token在给定上下文中本应极少出现，例如在回应英文提示词时生成泰语或中文字符，或在代码中产生明显的语法错误。举例来说，一小部分用英文提问的用户可能会在回复中间看到"สวัสดี"这样的内容。
此错误影响了8月25日至28日期间对Opus 4.1和Opus 4的请求，以及8月25日至9月2日期间对Sonnet 4的请求。第三方平台未受此问题影响。
解决方案：我们定位了该问题，并于9月2日回滚了相关变更。我们已在部署流程中增加了针对意外字符输出的检测测试。
3. 近似top-k XLA:TPU编译错误
8月25日，我们部署了用于改进Claude在文本生成过程中选择token方式的代码。此项变更无意中触发了XLA:TPU[1]编译器中的一个潜在缺陷，该缺陷已被确认影响了针对Claude Haiku 3.5的请求。
我们还认为，这可能也影响了Claude API上一部分Sonnet 4和Opus 3的请求。第三方平台未受此问题影响。
解决方案：我们最初观察到影响Haiku 3.5的缺陷，并于9月4日将其回滚。随后，我们注意到用户报告了与Opus 3相关的、与该缺陷表现一致的问题，并于9月12日将其回滚。经过大量调查，我们无法在Sonnet 4上复现此缺陷，但出于谨慎考虑，仍决定将其一并回滚。
与此同时，我们（a）正在与XLA:TPU团队合作修复编译器缺陷，并且（b）已推出了一项修复方案，采用增强精度的精确top-k方法。详情请参阅下方的深度解析。
XLA编译器缺陷深度剖析
为说明这些问题的复杂性，以下将介绍XLA编译器缺陷是如何显现的，以及为何它被证明尤其难以诊断。
当 Claude 生成文本时，它会计算每个可能的下一个词的概率，然后从该概率分布中随机选择一个样本。我们使用“top-p 采样”来避免无意义的输出——只考虑累积概率达到某个阈值（通常为 0.99 或 0.999）的词。在 TPU 上，我们的模型跨多个芯片运行，概率计算发生在不同位置。为了对这些概率进行排序，我们需要在芯片之间协调数据，这很复杂。[2]
2024 年 12 月，我们发现当温度为 0 时，我们的 TPU 实现偶尔会丢弃概率最高的 token。我们部署了一个临时解决方案来修复这个问题。
根本原因涉及混合精度运算。我们的模型以 bf16（16 位浮点数）计算下一个 token 的概率。然而，向量处理器原生支持 fp32，因此 TPU 编译器（XLA）可以通过将某些运算转换为 fp32（32 位）来优化运行时。此优化过程受 `xla_allow_excess_precision` 标志控制，该标志默认为 true。
这导致了一个不匹配：本应对最高概率 token 达成一致的计算，却以不同的精度级别运行。精度不匹配意味着它们无法就哪个 token 具有最高概率达成一致。这导致最高概率的 token 有时会完全从考虑范围中消失。
8 月 26 日，我们部署了对采样代码的重写，以修复精度问题，并改进我们在达到 top-p 阈值极限时处理概率的方式。但在修复这些问题的过程中，我们暴露了一个更棘手的问题。
我们的修复移除了十二月的临时方案，因为我们相信已经解决了根本原因。这导致近似 top-k 操作中出现了一个更深层的错误——这是一种用于快速找出最高概率模型 token 的性能优化方法。[3] 这种近似计算有时会返回完全错误的结果，但仅针对特定的批次大小和模型配置。十二月的临时方案此前一直在无意中掩盖这个问题。
该错误的行为表现极不稳定，令人沮丧。它会根据无关因素而变化，例如在其前后运行了哪些操作，以及调试工具是否启用。同一个提示词可能在一次请求中完美运行，而在下一次请求中失败。
在调查过程中，我们还发现精确 top-k 操作不再像过去那样存在高昂的性能代价。我们从近似 top-k 切换到了精确 top-k，并将一些额外操作标准化为 fp32 精度。[4] 模型质量没有妥协余地，因此我们接受了微小的效率影响。
为何检测困难
我们的验证流程通常依赖基准测试以及安全评估和性能指标。工程团队会进行抽查，并首先部署到小规模"金丝雀"测试组。
这些问题暴露了我们本应更早发现的关键漏洞。我们运行的评估根本没有捕捉到用户报告的性能下降，部分原因在于 Claude 通常能很好地从个别错误中恢复。我们自身的隐私实践也给调查用户报告带来了挑战。我们的内部隐私和安全控制限制了工程师访问用户与 Claude 交互的方式和时机，特别是当这些交互未被作为反馈报告给我们时。这保护了用户隐私，但也阻止了工程师检查识别或复现错误所需的问题交互。
每个 Bug 在不同平台上以不同频率表现出不同症状。这造成了一系列令人困惑的报告，无法指向任何单一原因。看起来像是随机、不一致的性能退化。
更根本的是，我们过度依赖了有噪声的评估。尽管我们注意到线上报告有所增加，但缺乏明确的方法将这些报告与我们最近的每次变更联系起来。当负面报告在 8 月 29 日激增时，我们并未立即将其与一项原本常规的负载均衡变更联系起来。
我们正在改变什么
随着我们持续改进基础设施，我们也在改进评估和预防上述类似 Bug 的方式，覆盖我们提供 Claude 服务的所有平台。以下是我们的改进方向：
- 更灵敏的评估：为了帮助发现任何给定问题的根本原因，我们开发了能够更可靠地区分正常实现与故障实现的评估方法。我们将持续改进这些评估，以更密切地监控模型质量。
- 在更多环节进行质量评估：虽然我们定期对系统运行评估，但我们将在真实的生产系统上持续运行评估，以捕获诸如上下文窗口负载均衡错误之类的问题。
- 更快的调试工具：我们将开发基础设施和工具，以便在不牺牲用户隐私的前提下，更好地调试来自社区的用户反馈。此外，这里开发的一些定制化工具将用于缩短未来类似事件（如果发生）的修复时间。
评估和监控很重要。但这些事件表明，当 Claude 的回复未达到通常标准时，我们还需要来自用户的持续信号。用户报告的具体变化观察、遇到的意外行为示例，以及跨不同用例的模式，都帮助我们隔离了问题。
用户持续直接向我们发送反馈仍然特别有帮助。您可以在 Claude Code 中使用 `/bug` 命令，或在 Claude 应用中使用“踩”按钮来提交反馈。开发者和研究人员经常创造出新颖有趣的模型质量评估方法，这些方法对我们内部测试形成了补充。如果您愿意分享您的成果，请通过 feedback@anthropic.com 联系我们。
我们始终感谢社区做出的这些贡献。
致谢
本文由 Sam McAllister 撰写，感谢 Stuart Ritchie、Jonathan Gray、Kashyap Murali、Brennan Saeta、Oliver Rausch、Alex Palcuie 以及众多其他同仁。
[1] XLA:TPU 是一种优化编译器，它将 XLA 高级优化语言（通常使用 JAX 编写）转换为 TPU 机器指令。
[2] 我们的模型规模过大，无法在单个芯片上运行，因此被划分到数十个或更多芯片上，这使得我们的排序操作成为一种分布式排序。TPU（与 GPU 和 Trainium 一样）也具有与 CPU 不同的性能特性，需要使用基于向量化操作而非串行算法的不同实现技术。
[3] 我们之前一直使用这种近似操作，因为它能带来显著的性能提升。该近似方法通过接受最低概率 token 中可能存在的误差来工作，这通常不会影响质量——除非该 bug 导致它反而丢弃了最高概率的 token。
[4] 请注意，现在已修正的 top-k 实现可能会导致 top-p 阈值附近 token 的包含情况出现细微差异，在极少数情况下，用户可能需要重新调整其 top-p 参数的选择。
This is a technical report on three bugs that intermittently degraded responses from Claude. Below we explain what happened, why it took time to fix, and what we're changing.
Between August and early September, three infrastructure bugs intermittently degraded Claude's response quality. We've now resolved these issues and want to explain what happened.
In early August, a number of users began reporting degraded responses from Claude. These initial reports were difficult to distinguish from normal variation in user feedback. By late August, the increasing frequency and persistence of these reports prompted us to open an investigation that led us to uncover three separate infrastructure bugs.
To state it plainly: We never reduce model quality due to demand, time of day, or server load. The problems our users reported were due to infrastructure bugs alone.
We recognize users expect consistent quality from Claude, and we maintain an extremely high bar for ensuring infrastructure changes don't affect model outputs. In these recent incidents, we didn't meet that bar. The following postmortem explains what went wrong, why detection and resolution took longer than we would have wanted, and what we're changing to prevent similar future incidents.
We don't typically share this level of technical detail about our infrastructure, but the scope and complexity of these issues justified a more comprehensive explanation.
How we serve Claude at scale
We serve Claude to millions of users via our first-party API, Amazon Bedrock, and Google Cloud's Vertex AI. We deploy Claude across multiple hardware platforms, namely AWS Trainium, NVIDIA GPUs, and Google TPUs. This approach provides the capacity and geographic distribution necessary to serve users worldwide.
Each hardware platform has different characteristics and requires specific optimizations. Despite these variations, we have strict equivalence standards for model implementations. Our aim is that users should get the same quality responses regardless of which platform serves their request. This complexity means that any infrastructure change requires careful validation across all platforms and configurations.
Timeline of events
The overlapping nature of these bugs made diagnosis particularly challenging. The first bug was introduced on August 5, affecting approximately 0.8% of requests made to Sonnet 4. Two more bugs arose from deployments on August 25 and 26.
Although initial impacts were limited, a load balancing change on August 29 started to increase affected traffic. This caused many more users to experience issues while others continued to see normal performance, creating confusing and contradictory reports.
Three overlapping issues
Below we describe the three bugs that caused the degradation, when they occurred, and how we resolved them:
1. Context window routing error
On August 5, some Sonnet 4 requests were misrouted to servers configured for the upcoming 1M token context window. This bug initially affected 0.8% of requests. On August 29, a routine load balancing change unintentionally increased the number of short-context requests routed to the 1M context servers. At the worst impacted hour on August 31, 16% of Sonnet 4 requests were affected.
Approximately 30% of Claude Code users who made requests during this period had at least one message routed to the wrong server type, resulting in degraded responses. On Amazon Bedrock, misrouted traffic peaked at 0.18% of all Sonnet 4 requests from August 12. Incorrect routing affected less than 0.0004% of requests on Google Cloud's Vertex AI between August 27 and September 16.
However, some users were affected more severely, as our routing is "sticky". This meant that once a request was served by the incorrect server, subsequent follow-ups were likely to be served by the same incorrect server.
Resolution: We fixed the routing logic to ensure short- and long-context requests were directed to the correct server pools. We deployed the fix on September 4. Rollout to our first-party platform and Google Cloud's Vertex AI was completed by September 16, and to AWS Bedrock by September 18.
2. Output corruption
On August 25, we deployed a misconfiguration to the Claude API TPU servers that caused an error during token generation. An issue caused by a runtime performance optimization occasionally assigned a high probability to tokens that should rarely be produced given the context, for example producing Thai or Chinese characters in response to English prompts, or producing obvious syntax errors in code. A small subset of users that asked a question in English might have seen "สวัสดี" in the middle of the response, for example.
This corruption affected requests made to Opus 4.1 and Opus 4 on August 25-28, and requests to Sonnet 4 August 25–September 2. Third-party platforms were not affected by this issue.
Resolution: We identified the issue and rolled back the change on September 2. We've added detection tests for unexpected character outputs to our deployment process.
3. Approximate top-k XLA:TPU miscompilation
On August 25, we deployed code to improve how Claude selects tokens during text generation. This change inadvertently triggered a latent bug in the XLA:TPU[1] compiler, which has been confirmed to affect requests to Claude Haiku 3.5.
We also believe this could have impacted a subset of Sonnet 4 and Opus 3 on the Claude API. Third-party platforms were not affected by this issue.
Resolution: We first observed the bug affecting Haiku 3.5 and rolled it back on September 4. We later noticed user reports of problems with Opus 3 that were compatible with this bug, and rolled it back on September 12. After extensive investigation we were unable to reproduce this bug on Sonnet 4 but decided to also roll it back out of an abundance of caution.
Simultaneously, we have (a) been working with the XLA:TPU team on a fix for the compiler bug and (b) rolled out a fix to use exact top-k with enhanced precision. For details, see the deep dive below.
A closer look at the XLA compiler bug
To illustrate the complexity of these issues, here's how the XLA compiler bug manifested and why it proved particularly challenging to diagnose.
When Claude generates text, it calculates probabilities for each possible next word, then randomly chooses a sample from this probability distribution. We use "top-p sampling" to avoid nonsensical outputs—only considering words whose cumulative probability reaches a threshold (typically 0.99 or 0.999). On TPUs, our models run across multiple chips, with probability calculations happening in different locations. To sort these probabilities, we need to coordinate data between chips, which is complex.[2]
In December 2024, we discovered our TPU implementation would occasionally drop the most probable token when temperature was zero. We deployed a workaround to fix this case.
The root cause involved mixed precision arithmetic. Our models compute next-token probabilities in bf16 (16-bit floating point). However, the vector processor is fp32-native, so the TPU compiler (XLA) can optimize runtime by converting some operations to fp32 (32-bit). This optimization pass is guarded by the xla_allow_excess_precision flag which defaults to true.
This caused a mismatch: operations that should have agreed on the highest probability token were running at different precision levels. The precision mismatch meant they didn't agree on which token had the highest probability. This caused the highest probability token to sometimes disappear from consideration entirely.
On August 26, we deployed a rewrite of our sampling code to fix the precision issues and improve how we handled probabilities at the limit that reach the top-p threshold. But in fixing these problems, we exposed a trickier one.
xla_allow_excess_precision flag.Our fix removed the December workaround because we believed we'd solved the root cause. This led to a deeper bug in the approximate top-k operation—a performance optimization that quickly finds the highest probability tokens.[3] This approximation sometimes returned completely wrong results, but only for certain batch sizes and model configurations. The December workaround had been inadvertently masking this problem.
The bug's behavior was frustratingly inconsistent. It changed depending on unrelated factors such as what operations ran before or after it, and whether debugging tools were enabled. The same prompt might work perfectly on one request and fail on the next.
While investigating, we also discovered that the exact top-k operation no longer had the prohibitive performance penalty it once did. We switched from approximate to exact top-k and standardized some additional operations on fp32 precision.[4] Model quality is non-negotiable, so we accepted the minor efficiency impact.
Why detection was difficult
Our validation process ordinarily relies on benchmarks alongside safety evaluations and performance metrics. Engineering teams perform spot checks and deploy to small "canary" groups first.
These issues exposed critical gaps that we should have identified earlier. The evaluations we ran simply didn't capture the degradation users were reporting, in part because Claude often recovers well from isolated mistakes. Our own privacy practices also created challenges in investigating reports. Our internal privacy and security controls limit how and when engineers can access user interactions with Claude, in particular when those interactions are not reported to us as feedback. This protects user privacy but prevents engineers from examining the problematic interactions needed to identify or reproduce bugs.
Each bug produced different symptoms on different platforms at different rates. This created a confusing mix of reports that didn't point to any single cause. It looked like random, inconsistent degradation.
More fundamentally, we relied too heavily on noisy evaluations. Although we were aware of an increase in reports online, we lacked a clear way to connect these to each of our recent changes. When negative reports spiked on August 29, we didn't immediately make the connection to an otherwise standard load balancing change.
What we're changing
As we continue to improve our infrastructure, we're also improving the way we evaluate and prevent bugs like those discussed above across all platforms where we serve Claude. Here's what we're changing:
- More sensitive evaluations: To help discover the root cause of any given issue, we’ve developed evaluations that can more reliably differentiate between working and broken implementations. We’ll keep improving these evaluations to keep a closer eye on model quality.
- Quality evaluations in more places: Although we run regular evaluations on our systems, we will run them continuously on true production systems to catch issues such as the context window load balancing error.
- Faster debugging tooling: We'll develop infrastructure and tooling to better debug community-sourced feedback without sacrificing user privacy. Additionally, some bespoke tools developed here will be used to reduce the remediation time in future similar incidents, if those should occur.
Evals and monitoring are important. But these incidents have shown that we also need continuous signal from users when responses from Claude aren't up to the usual standard. Reports of specific changes observed, examples of unexpected behavior encountered, and patterns across different use cases all helped us isolate the issues.
It remains particularly helpful for users to continue to send us their feedback directly. You can use the /bug command in Claude Code or you can use the "thumbs down" button in the Claude apps to do so. Developers and researchers often create new and interesting ways to evaluate model quality that complement our internal testing. If you'd like to share yours, reach out to feedback@anthropic.com.
We remain grateful to our community for these contributions.
Acknowledgments
Written by Sam McAllister, with thanks to Stuart Ritchie, Jonathan Gray, Kashyap Murali, Brennan Saeta, Oliver Rausch, Alex Palcuie, and many others.
[1] XLA:TPU is the optimizing compiler that translates XLA High Level Optimizing language—often written using JAX—to TPU machine instructions.
[2] Our models are too large for single chips and are partitioned across tens of chips or more, making our sorting operation a distributed sort. TPUs (just like GPUs and Trainium) also have different performance characteristics than CPUs, requiring different implementation techniques using vectorized operations instead of serial algorithms.
[3] We had been using this approximate operation because it yielded substantial performance improvements. The approximation works by accepting potential inaccuracies in the lowest probability tokens, which shouldn't affect quality—except when the bug caused it to drop the highest probability token instead.
[4] Note that the now-correct top-k implementation may result in slight differences in the inclusion of tokens near the top-p threshold, and in rare cases users may benefit from re-tuning their choice of top-p.