本月初,我在 AI 工程师世界博览会上与 Anthropic 的 Claude Code 团队成员 Cat Wu 和 Thariq Shihipar 进行了一场炉边对谈。我们聊了 Claude Code、Claude Tag、Fable、编程智能体安全、评测、工具设计,以及 Anthropic 内部如何使用这些工具。
本次对谈的完整视频现已在 YouTube 上线。以下是经过编辑的实录版本,附有额外链接和我加粗的重点标注。
如果你不想看视频或通读整篇实录,这里先列几条要点:
- Claude Tag(Claude 新的协作式 Slack 集成)目前已经承担了 Claude Code 团队 65% 的产品工程 PR。
- Claude Code 的功能会先向 Anthropic 员工开放,而且只发布那些在该群体中展现出用户留存效果的功能。
- 对 Claude Code 的关键改动仍会人工审查,但团队越来越依赖自动化代码审查来处理产品的“外层”部分。
- 对于 Fable 5 甚至 Opus 4.8 这类模型,在系统提示词中添加示例已不再是最佳实践。Claude Code 的系统提示词最近缩减了 80% 的体量。
- 同样,“不要做 X、不要做 Y”这类禁令列表也会降低最新模型的输出质量。
- Anthropic 内部的自产自用(dogfooding)被称为“蚂蚁试食”(ant fooding)。
- Anthropic 非常看好他们的自动模式,并将其视为 Claude Tag 的使能技术。
- Thariq 建议,要抵消编程智能体带来的“深蓝效应”,可以在自己承接的工作上“更有野心”。
- Fable 在视频剪辑方面能力不俗,Thariq 就用它来剪辑 Fable 自己的发布视频。
- Anthropic 这种(内部)公开工作的文化是他们成功的关键,这一点从他们在公开 Slack 频道中使用 Claude Tag 的方式就可见一斑。
过去一年里,你的日常工作发生了哪些变化?
Simon:Claude Code 是去年二月发布的——到现在还不到一年半,最初它只是 Claude Sonnet 3.7 发布稿里的一个要点。如今我们有了这些真正能为我们干活的编程智能体,过去一年里你的日常工作发生了哪些变化?
Cat:我记得我们刚推出 Claude Code 和 Sonnet 3.7 的时候,你给它一个任务,就得密切盯着它尝试做的每一件小事。我会极其仔细地阅读每一个权限提示。我经常说“不”——不不不,你检查过这个文件吗?你检查过那个文件吗?而现在,随着每一代模型的迭代,变化简直令人难以置信。我觉得我们都有机会退后一步,把更多琐碎的实现工作委托给 Claude。这释放了我们大量时间,可以去思考更有创造性的工作,比如:既然我们知道 Claude Code 能实现其中大部分功能,那我们应该为用户提供什么样的正确体验?而现在有了 Fable,这又是一个完全不同的阶跃式提升。我们看到,在很多用例中,你现在真的可以用 Fable 一次性搞定大量功能。
Thariq:我记得我收到第一条关于 Claude Code 的消息。我一个最好的朋友说:“你得去试试 Claude Code。”那大概是在 Opus 4 发布的时候,我试了一下,然后心想:“天哪,我现在必须去 Anthropic 工作。”那还是 Opus 4——很棒的模型,但你还在读权限提示。想想我们有多健忘,真是有点疯狂——我现在会觉得,自动模式不是一直都在吗?我甚至都不记得自己点过“是”和“允许”。对我来说,我一直在逼自己做的事情是:我们必须做出比以往任何时候都更高质量的工作。输出质量高得惊人。我一直在用它剪辑视频,然后我会想,好吧,它必须在几个小时内达到我们品牌团队极其严苛的要求,否则我们就做不了。这就是我想借助 Fable 实现的转变:用比以往更快的速度,做出我们做过的最好的作品。
传统软件工程中,哪一条已经不再成立了?
Simon:有没有一条传统软件工程的原则,一年前还成立,但你觉得在这个新世界里已经不再适用了?
Cat:我们在工程技能方面看到的最大变化之一是:两年前,产品经理去跟一群客户聊,花六个月时间与跨职能团队对齐PRD,然后在写第一行代码之前,写出一份详尽的技术规格说明,这是很典型的做法。现在情况完全反过来了。对很多工程师来说,我会给在场的人的建议是,多培养自己的商业嗅觉和产品判断力,搞清楚我们到底应该做什么,因为从产生想法到把它做出来的时间线大大缩短了——从六到十二个月缩短到可能只要一周。这意味着我们所有人都需要更好的品味,判断什么值得做,什么真正能推动我们所做的业务。所以产品品味和商业判断力的价值上升了,而在大多数产品领域,执行力的价值略有下降。当然,对于基础设施来说,确保所有细节都正确仍然是非常重要的事情。
Thariq:对我来说,最大的变化是重写现在变成了一件好事。
Simon:以前最糟糕的事情,现在反而变成好事了!
Thariq:没错。所有《人月神话》里说的——永远不要重写——我现在是支持重写的。如果你有一套好的测试套件——而且我认为重写实际上会迫使你确保自己有一套好的测试套件——但我觉得人们低估的一点是,代码库本身就是一份规格说明,而且可能你手上只有这一份规格说明,因为没有人知道代码库的每一个分支部分。你可以把它当作一个产物,从中提炼或者创建它的其他版本。我们用Rust重写了Bun,效果非常好——我现在就在用它。
Simon:你们还没有把Claude Code跑在Rust版Bun上对外发布,对吧?
Thariq:内部我们已经这么做了。
(实际上,看起来Anthropic从6月17日开始已经向所有人发布了基于Rust版Bun的Claude Code。)
非工程师用Claude Tag都在做什么?
Simon:最近另一个重大发布是Claude Tag——到现在大概一周了吧,至少对我们这些外部人来说是这样。我了解到它在Anthropic内部被非工程师大量使用。非工程师用Claude Tag都在做什么?
Cat:Claude Tag 是一个住在你们团队协作工具里的 Claude。我们上周把它发布到了 Slack 里。Claude Tag 与众不同的地方在于,它默认就是多人协作的。一旦你把 Claude Tag 加进一个 Slack 频道,你可以随时插话,你的队友也可以随时插话,你们可以一起在 PR 上协作。另一个重大区别是,它是主动式的,而不是被动响应的。你可以告诉 Claude Tag:“嘿,监控这个频道里的每一个 bug 报告,提一个 PR 去修复它,然后 @ 一下最近改动过这部分代码库的工程师。”它会在该频道的整个生命周期内持续这样做,你不需要手动去 @ 它。第三个重大转变是,我们把团队记忆加了进来。如果你在频道里告诉 Claude Tag 你的偏好,它会在以后的每一条帖子里都记住这些偏好。如果你总是希望它去调试宕机问题,但不想让它去调试警告信息,只要在频道里用自然语言告诉它就行,它会为你和你团队里的其他所有人记住这一点。
在内部,我们把 Claude Tag 看作是 Claude Code 的演进。我们认为这是我们内部工作方式的一次重大转变。Claude Tag 目前负责了我们产品工程团队 65% 的 PR。
Simon:是整个 Anthropic 的,还是只针对 Claude Code 的?
Cat:这只是针对我们的产品工程团队——我们的内部版 Claude Tag 目前负责了我们产品 PR 的 65%。这是一个巨大的转变;这超过了我们 PR 总量的一半。我们看到人们是这样在 Claude Code 和 Claude Tag 之间分配工作的:Claude Code 仍然是你处理最复杂任务的最佳选择,也就是当你需要与智能体进行交互式迭代的时候。但 Claude Tag 非常适合让它主动地为你工作,这样你就不需要再为你正在开发的功能所出现的所有 bug 报告手动启动 Claude Code 了。
Thariq:再举一个非编程场景的例子:比如在这次演讲之前,我们问 Claude Tag:“嘿,Fable 什么时候发布?”我们想确保它能和公告时间对上。Claude Tag 会搜索我们的 Slack,看看大家都在聊什么。作为公司的搜索引擎,它真的非常有价值。它掌握了你产品的全部上下文,所以你可以问它各种指标相关的问题——通常做决策时,你希望决策有数据支撑,所以你可以把它接到事件存储上。我见过我们的市场团队做类似“嘿,给我讲讲这个功能”的事情。他们不是程序员,但 Claude 是程序员——它可以克隆代码库,然后说:“这就是这个功能,长这样,这是我使用该功能的录屏。”它解锁了各种各样的可能性,我觉得我们还在摸索的早期阶段。
Claude Tag 作为团队协作层
Simon:我在使用编程智能体时遇到的一个问题是,我知道怎么作为个人去用,但不太清楚在团队环境里该怎么用。听起来 Claude Tag 就是你们目前针对这类事情的团队协作层解决方案。
Cat:完全正确。而且现在我们很大比例的会话其实都是多人协作的。比如我可能会说:“嘿,我觉得我们应该在 Cowork 里实现这个新功能,”然后我会把 Claude Tag 拉进来做第一版。接着我会告诉 Claude Tag:“分享一段你最终实现的录屏,”再把设计团队拉进来看。他们会稍微调整一下,然后传给工程团队收尾,最后发布到生产环境。整个过程非常流畅。我们还在摸索如何协调同一会话中的社交互动规则,但我们发现大家会观察别人怎么用,然后遵循这些社交规范——把 Claude Tag 融入我们的团队,对我们来说相当自然。
Thariq:它也很适合带新人,同时还能减少“垃圾产出”,因为大家都能看到你如何使用 Claude,这本身也会提升你使用 Claude 的水平。
这让我想起 Midjourney 是如何通过在 Discord 频道中强制公开提示词来解决教用户高级图像提示词这一挑战的。
当构建成本大幅降低时,你如何决定哪些功能值得构建?
我自己觉得非常难的一点是,在功能构建成本大幅下降的当下,如何判断一个功能是否值得发布。
Simon:你如何应对工程领域最棘手的问题——优先级排序?当构建一个功能的成本现在低得多时,你如何决定哪些功能值得构建和发布?
Cat:这确实是个难题。我们有几种处理方式。一是我们每天都在内部使用自己的产品。每当我们在产品中想做某件事却做不到时,我们不会去找替代方案,而是修复产品本身,让它能支持这个场景。我们内部有非常浓厚的“吃自家狗粮”文化。在我们把产品分享给全世界之前,我们会先分享给 Anthropic 内部的每个人,以及一些早期客户,他们会给出非常坦诚的反馈——越犀利越好——然后我们不断迭代,直到人们真正喜欢为止。我们内部对功能在分享给世界之前需要达到的活跃用户数和留存率有一个明确标准。因为这个标准非常清晰,每个工程师都知道自己要达到什么目标。我认为这也提升了我们的打磨程度,因为如果功能不够精致,用户就会流失——那这个功能就不应该发布。
用内部用户留存率来决定一个功能是否应该发布,对我来说非常合理。
你有没有一个让你感到意外的功能例子?
Simon:你有没有一个让你感到意外的功能例子?你推出后参与度爆表——一个原本不太可能发布的东西最终变成了真正的产品功能。
Cat:我确实有一个。我们团队里很多人喜欢远程控制功能。远程控制让你可以用移动设备或浏览器里的 Claude,连接到你在命令行中运行的本地 Claude Code 会话。我从来没有这个需求,因为我直接在手机上启动任务,任务在云端会话中运行,不需要用到我的本地环境——我想是因为我做的都是很简单的编码任务。这是我一开始不太理解的东西;我当时想,嘿,大家应该直接搭远程开发环境才对。但实际上,自从我们推出远程控制后,我聊过的很多人告诉我,他们每天晚上做的事就是把笔记本电脑插上电源,打开一堆远程控制会话,锁上屏幕,然后坐在沙发上用手机控制 Claude Code。所以这已经成了我们现在正在积极推进的一个工作流,我一开始没搞懂——但现在我懂了。
Claude Code 里的每一行生产代码都会有人工审查吗?
这次大会的一个核心主题是审查:人们花了多少注意力去审查编码智能体为他们编写的代码。我非常想听听 Claude Code 团队对这个问题的看法!
Simon:代码审查是怎么运作的?进入 Claude Code 的每一行生产代码都会有人工审查吗?如果没有,你们在做什么——怎么保证质量?
Thariq:这很大程度上取决于任务。对于重要领域,我们有代码负责人。系统提示词就是一个有代码负责人的例子——你确实需要得到他们的批准。
Simon:所以代码负责人直接对那个代码领域的质量负责。
Thariq:没错。
Cat:而且任何涉及该领域的 PR 都需要他们批准。
Thariq:我们有代码审查 GitHub 机器人审查所有内容——它作用于每一个 PR,而且很多时候它承担了大部分审查工作。我在团队里观察到的一点是,对于更复杂的 PR,你可能会创建一个 artifact 来解释这个 PR,这样其他人就可以接着审查。我们在验证、CI/CD 等方面投入了大量精力,确保任何时候出现问题我们都有测试兜底。我们有一个非常健壮的环境,Claude 可以控制 Claude Code 并对其进行测试。所以代码审查是多管齐下的。
Cat:总的来说,我们正在努力走向一个人类无需介入循环的世界。对于 Claude Code 核心以及其他产品核心最关键的变更,始终有代码负责人,他们会手动审查所有变更。但越来越普遍的是,对于外层变更,我们实际上让 Claude Code 全面审查这些内容。这听起来挺吓人的,但我们花了六个多月的时间才走到这一步,而且是通过一步步的小步快跑来建立对代码审查的信任。一开始我们对所有内容都做人工审查,然后逐渐地我们会说,好吧,对于触及这些文件的代码变更,代码审查已经能 100% 捕获那里的问题——所以我们实际上不需要人工手动审查那些了。当我们做事故复盘时,我们会查看导致事故的 PR,然后说,好吧,我们怎么更新代码审查来捕获这类问题?——我们会把这些 PR 加入评估集,确保我们未来对代码审查的改动永远不会让那个指标回退。把人类从代码审查循环中移除是向前迈出的一大步。这听起来可能吓人,也不是一夜之间就能做到的事,但通过数月对基础设施的投入,让你有信心代码审查能捕获所有你在意的问题,这是可以做到的。
所以关键似乎在于持续迭代自动化审查系统本身,以便随着时间推移建立对它们的信任。
一个新模型会如何影响你对它能做什么、不能做什么的直觉判断?
我们深入研究了评估——这也是整个大会期间的另一个热门话题。
Simon:我知道,如果让 Opus 4.8 帮我构建一个运行 SQL 查询并输出 JSON 的 JSON 端点,它一定能做对——这种事我不需要仔细审查。但如果有新模型出来,我不知道该如何快速建立对 Fable 的信任,确保它不会把 Opus 没搞砸的事情搞砸。新模型会如何影响你对它“能做什么、不能做什么”的判断直觉?
Cat:我们之所以长期搭建这套评测基础,就是为了让新模型能成为“即插即用”的替代品。当我们拿到新模型时,会跑完整套评测集,确保比如 Fable 严格优于 Opus 4.8——这样我们才有信心把它直接替换上去。
Simon:这些模型评测是针对整个 Anthropic 的,还是 Claude Code 团队专属的?
Cat:两种都有。我们团队有自己的评测,同时我们也会对 Anthropic 内部每个代码仓库做代码审查,所以这方面也有评测。至于像自动模式这类功能,我们不仅对 Anthropic 内部每个用户做评测,还委托了多个外部测试人员进行红队测试,搭建包含提示词注入和恶意输入的测试环境,确保自动模式不会放过任何这类攻击。
你们是如何建立信心,确认系统提示词的调整确实能带来更好的输出?
Simon:我想知道我做的系统提示词改进是否真的提升了产品效果——这是最基础的产品级评测形式,但我仍然不太清楚该怎么做。你们是否在做这件事,并且对“系统提示词的某个调整能带来更好输出”有十足把握?
Cat:我们并没有百分之百的信心,但我们做了大量工作来确保性能不会回退。起点是一套我们信任的外部评测,再辅以一套规模更大的、我们同样信任的内部评测。最初,我们主要针对能力进行优化:给定任务的完整定义和完整代码库,Claude 是否能做出正确的决策、彻底修复所有 bug、并通过全部测试?这是我们的出发点和优化目标,因为它最直接地反映了用户的需求。但还有很多行为会影响用户在使用 Claude Code 时的感受。例如,人们非常不喜欢 Claude Code 说“该睡觉了”。或者人们非常不喜欢它说:“嘿,我完成了五部分中的两部分——你想让我继续吗?”是的,请继续。因此,我们正在构建一套行为评测来捕捉这些问题。随着我们收到用户反馈——请大声地向我们反馈——我们会按优先级对问题排序,然后逐一处理,为每个问题构建评测。这还达不到 100% 的覆盖率,但提高覆盖率是我们的优先事项。
Claude Code 团队与模型训练团队之间有多少互动?
Simon:Claude Code 团队与 Anthropic 内部最初训练模型的团队之间有多少互动?这是否是一种非常紧密的合作?
Cat:在整个 Anthropic,我们的合作都非常紧密。我们经常开会讨论我们对下一代模型能力的预期。我们的研究团队在公开分享这方面也做得非常出色——我们经常在博客文章中谈到,我们正致力于支持越来越长周期的任务,以及我们如何训练 Claude 本身做到诚实、无害、乐于助人。我们还投入了大量精力,确保它能与你的意图对齐,即使你的意图表达得比较模糊。当然,你最好尽量具体地说明你的需求,这样 Claude 就能掌握所有上下文——但即使你不够具体,我们也会教会 Claude 做出合理的假设。这是一段富有成效的合作关系。
系统提示词缩减了80%——你们都删掉了哪些内容?
这一节里包含了好多实用的提示词技巧!
Simon:Thariq,你今天早上提到,由于Claude Fable,Claude Code的系统提示词缩减了80%。能再详细讲讲吗?你们都删掉了哪些类型的内容?
Thariq:不只是Fable——Opus 4.8也一样,而且未来新模型也会如此。我们现在针对不同模型使用不同的系统提示词。我们发现的一个模式是,我们之前对Claude约束过多了。最初那批,大概是Opus 4那一代的模型,需要大量示例,而删掉示例效果极其显著,因为它比我们给出的示例更有创造力。
Simon:这真的很有意思,因为我给人们讲的最重要的提示词技巧之一就是:给它示例。如果这条不再成立,那多少有点颠覆我的提示词方法论了。
Thariq:我也是——听到这个我也很惊讶。我觉得现在更关键的是你给它的“形态”——你给Claude的工具、你的系统提示词,诸如此类。我们做的另一件事是尽量给它更多上下文、更少“不要这样做”的指令,因为那种指令对Claude来说是一种很强的冲动,尤其是如果它后来与用户指令冲突,会让Claude极度困惑——“我这个技能里这么写,系统提示词里又那么说。”所以我们尽量少用硬性约束,多给上下文,整体指令也更精简。这绝对是一门科学——我们跑了一堆评测才打磨出来。
Cat:一般来说,当你给这些模型写提示词时,你始终应该思考:我给出的指令是否存在边界情况?当我们回头审查 Claude Code 系统提示词中的所有指令时,我们发现了一些情况——是的,这句话有 90% 是对的,但确实有 10% 的真实场景下它并不成立。我们不想约束模型,也不想把它搞糊涂,让它以为它应该总是这样做。一个很好的例子是验证。这里的每个人都希望 Claude 验证它的工作成果,我们在提示词里有一些指令说:如果你做了前端改动,一定要验证。但这是有限度的。如果它只是把文案从一个字符串改成另一个字符串,而用户说“做个快速修复然后更新测试”,也许你就不需要验证。所以我们已经把措辞从“一定要验证、验证、再验证”调整成了类似这样的说法:大多数时候,当你在做前端工作时,你无法通过调用后端接口来完全理解用户体验,所以当你对用户体验做较大改动时,请在本地运行应用。事实上,那条指令可能也不够好,因为什么叫“较大的改动”?也许小改动也应该测试。总的来说,每当你给模型写提示词时,你都应该思考一个善意的普通人可能会在哪些方面误解它,从而更好地理解模型可能会如何解读它——然后把提示词软化,让它真正做到 100% 准确,因为你每次都会把这个提示词交给模型。
Simon:这件事最吸引人的地方在于,你在依赖模型的判断力——这必须是 Opus/Fable 级别的能力。一年前的模型还不具备那种判断力,无法决定自己要不要测试某个改动。但如果你是在为一大类模型做开发,并且试图用更便宜的模型来处理更便宜的任务,那这套逻辑就行不通了。
Cat:正是因为这个原因,我们现在实际上为每个模型准备了不同的系统提示词。只有我们最前沿的模型才有这 80% 的 token 缩减——较老的模型仍然使用完整的系统提示词。
Simon:你觉得 Fable 和 Opus 够不够聪明,能带着更多细节去提示 Haiku 吗?因为它们明白 Haiku 的判断力和品味都比较弱。
Cat:我们还没法对它做评估——我们没有任何硬数据来证明这一点。
Thariq:小模型有时候会碰到一个棘手的问题,因为大模型在难题上有时比小模型更省 token。所以这里需要建立一点直觉——有时候你几乎就是全程都想要前沿智能。帕累托曲线会移动,而且很难找到那个平衡点。
Simon:一年前,我不相信模型能写出提示词。如今,好的模型非常擅长写提示词——我的很多提示词都是模型写的,这听起来很荒谬,但效果确实很好。让我接受这一点的一个关键思路,是去想子智能体——那完全就是一个 Claude 模型为另一个 Claude 模型搭建提示词。
Thariq:工作流其实就是一个很好的例子,因为 Claude 不只是给单个子智能体写提示词,而是给多个子智能体的编排写提示词,而且每一个子智能体都会得到一份非常详细的提示词。这几乎比单纯生成一个子智能体又高了一个层级。我自己也在个人机器上用它,给它接上 Gemini API,然后说:来,生成图片。它比我给图像模型写提示词时勤快多了。这就是 Claude 一路提示 Claude,层层到底。
Cat:我觉得工作流工具的提示词也是 Claude 写的。
Simon:我读过那份提示词——写得很好。这其实是我对 Anthropic 整体上感到不满的一点:你们发布了 Claude Chat 的提示词,但没有把工具提示词和 Claude Code 的提示词一起放出来。我还是得跑一个代理去拦截它们。我非常希望 Claude Code 的提示词能被有意地公开发布——它们就是文档。它们是了解这个工具能做什么、怎么运作的途径。
Cat:我会把这个功能请求记下来。我会让 Claude Tag 去办。
值得注意的是,OpenAI 针对 GPT-5.6 的提示词最佳实践,也给他们最新的模型提供了类似的建议:
倾向于更精简的提示词
删除重复的指令和示例、简化工具描述,可以提升任务表现和 token 效率。在一组内部编码智能体评测运行样本中,采用更精简系统提示词的配置使评测分数提升了约 10–15%,同时总 token 量减少了 41–66%,成本降低了 33–67%。
你引入一个新工具的标准是什么?
Simon:Claude Code 基本上就是一个装满工具的大袋子。你引入一个新工具的标准是什么?你如何判断在那一层面投入额外的工程开发是值得的?
Cat:你想回答这个问题吗?你可是引入了我们最好用的工具之一。
Thariq:我的职业生涯巅峰就是引入了“询问用户问题”这个工具。这真的很难。尤其是某些工具——ask user question 是 Claude 用来向你提问的工具——所以它很难做评测,而且有时候这更多是用户偏好问题。当时我们的评测还比较少,所以很大程度上是基于“吃自己的狗粮”(dogfooding)——或者说是我们蚂蚁版的“ant fooding”。但总的来说,我们一直在努力朝着更少工具的方向发展。我们最近引入的一组工具是 task 工具,我记得是这样——而且我们尽量给 Claude 提供更通用的版本来完成各种任务。
你的文件编辑工具最新进化到了什么程度?
我一直对文件编辑工具抱有长期的兴趣——它们是当年 Aider 代码编辑排行榜的主题,我也饶有兴趣地关注着它们在不同编码智能体中从基于搜索替换、到基于行号、再到更复杂模式的演变过程。
Claude API 文档描述了一个文本编辑工具,推荐在基于 API 构建时使用,但 Claude Code 在这里似乎采用了略有不同的方法。
Simon:最有趣的工具之一是文件编辑工具——你可以把文件编辑作为一个工具,也可以让它用 sed 和 grep 来做这些事情。你的文件编辑工具最新进化到了什么程度?
Thariq:我们确实还保留了一个,但比如说,我们已经移除了 grep 和其他搜索工具——glob 工具——转而使用原生 bash。就像我之前在演讲里说的,模型更像生物学而不是物理学,而工具设计尤其困难。我不确定 Cat 是否不同意,觉得评估这件事有科学方法可循,但我觉得工具设计更像一门艺术,也许——或者说更像生物学。
Cat:我大体同意,但总的来说,随着我们引入更多工具,我们会尽量把工具数量控制在较低水平,并确保我们添加的每个工具都与其他工具功能明确区分,这样 Claude 就能很容易判断什么时候该调用哪个。至于文件编辑,我们之所以保留它,其实是因为我们可以把它渲染出来。当 Claude 修改文件时,我们会向用户展示,并且有一个专门的漂亮 UI 界面,上面写着:你是否批准对这份文件的这次编辑?我们之所以做一个专门的编辑文件工具,是为了能确定性地知道 Claude 正在修改文件,从而可以向用户展示这个漂亮的 UI。很多刚上手的新用户仍然很喜欢这种体验,所以我们一直保留着它。但对于我们很多现在处于自动模式的人来说——希望你们不是处于 YOLO 模式——我觉得这其实无所谓,我们可能直接把文件编辑功能删掉也完全没问题。
Anthropic 内部关于安全运行 Claude Code 的建议是什么?
这就是提示词注入的问题!还有谁比 Anthropic 的员工更适合来解释 Anthropic 如何看待提示词注入攻击导致其 Claude Code 实例失控的风险呢?
事实证明,他们非常信任自己的自动模式——并认为正是这个功能让 Claude Tag 成为可能。
Simon:我们来谈谈安全与安保问题。我非常清楚提示词注入的风险,如果别人告诉我的 Claude Code 去做什么,可能会有很多糟糕的事情发生。我目前大部分时间还是在 YOLO 模式下运行 Claude Code,并为此感到非常内疚。Anthropic 内部关于安全运行 Claude Code 的建议是什么?
Cat:为什么不用自动模式呢?
Simon:我开始用自动模式了,但我对它了解得还不够,不知道它到底有多安全。大概三周前开始,我默认使用自动模式了。
Cat:在 Anthropic 内部,几乎所有员工都在使用自动模式。这是在 Claude Code 中安全地执行长期任务的最佳方式。我们做了大量的压力测试。我们有数千个评测集。我们还委托了许多红队成员创建对抗性环境,试图诱骗 Claude Code 做出恶意行为,而我们已经修复了他们发现的每一个问题。我们将在未来几周发布一些评测结果,但我们已经基本修复了所有攻击手段。
Simon:这是一个很大的声明。
Cat:我们会分享相关评测,让大家可以自行评估,但我们在识别 Claude 可能出错的所有方式并更新自动模式以应对这些问题上,确实非常严谨。它并不能 100% 覆盖所有情况——那样说就太过头了。但就我们关心的主要风险类别,比如提示词注入和数据外泄而言,风险已经远低于普通人类审查员。
我非常期待进一步了解他们的评测方法以及验证自动模式的方式。
Thariq:简单说一下自动模式的工作原理——建立一个这样的心智模型很有用。每当 Claude 执行一轮操作或一次 bash 调用时,都会有一个 Sonnet 分类器在判断这个工具调用以及对话的上下文——也就是你的指令。有些权限是取决于你的请求的:你不想一直给 git push 权限,但如果你说“把这个推送到 GitHub”,你就希望它去做——而如果你说“不要推送”,你就希望它拒绝。自动模式就能做到这一点。这种情况在我身上经常发生,Claude 因为非常乐于助人且积极主动而试图去做某件事,自动模式看到“不要做这个”就会把它拦截下来。所以它很擅长处理你在提示词里自己给出的动态权限,我认为这一点非常重要。它和我们的沙箱基础设施也配合得很好,因为沙箱就是那种有太多不同边界情况、我们很难确定性地逐一处理的东西。我们有一个沙箱,当某些操作需要逃出沙箱时——比如一个网络请求——自动模式可以查看这个请求并判断:这合理吗?——然后放行。
Simon:我之前没意识到自动模式也在和网络沙箱交互。
Cat:它会与用户原本会看到的任何权限提示进行交互。
Simon:自动模式有多久了?作为一个我能访问的功能,它好像才几个月大,对吧?
(它于 3 月 24 日首次向公众开放。)
Cat:我们从 1 月份开始就在 Anthropic 内部使用它了,所以我们已经对它进行了相当长时间的加固。Anthropic 极其重视安全与保障,我们一直在与对齐和安全团队广泛合作,以便在向全世界发布之前,实现内部推广、构建这些评估,并让自动模式变得更加稳健。
Thariq:这也是 Claude Tag 如此出色的原因——Claude Tag 使用了自动模式。我听到过很多关于 Slackbot 是自建还是外购的讨论,我的想法是:拜托,你们大概不应该自己构建 AI Slackbot。攻击向量太多了。你有一个用户可以发布反馈的反馈渠道,现在你的机器人正在读取它。我们在自动模式上投入的工作——而且我们有通用的瑞士奶酪式安全防御;我们还针对这类问题做了强化学习——我认为这才是 Claude Tag 真正奏效的原因。它与你的权限体系无缝协作,而且你肯定不希望自己的 Slack 里被提示词注入。
除了自动模式之外,还有更多安全方面的规划吗?
Simon:除了自动模式之外,还有更多安全方面的规划吗?
Thariq:我认为我们已经非常安全了。使用 Claude Tag,你可以为 Claude 配置自己的凭据,这样它就不需要以你的名义行事——你可以将 Claude 作为一个独立身份,这也让审计和检查 Claude 的行为变得更加容易。
Simon:因为 Claude Tag 会受到任何能与它对话的人的影响——它面对的是更庞大的人群在告诉它该做什么。
Thariq:没错。当然,我们还有 Fable 的探针,这是我们安全与研究工作的下游成果。我认为现在正是看到 Anthropic 作为一家 AI 安全公司真正产生回报的时刻:我们确实希望 Claude 能够以对齐的方式长时间运行,而自动模式必须做到几乎完美无缺才能实现这一点——这一切都源于我们是一家 AI 安全公司。
Cat:我们还为那些希望更安全的远程控制用户推出了可信设备。而且对于我们所有的远程环境,我们都支持凭据注入。如果你希望 Claude Code 能够访问 Datadog,但又不希望 Claude Code 本身持有 Datadog 凭据,你可以设置我们的身份与凭据管理系统,这样 Datadog 凭据只能被智能体使用,但智能体本身无法访问——当智能体尝试发起 Datadog 请求时,我们会即时注入这些凭据。
我非常喜欢这种凭据注入模式:Claude Code 可以通过代理访问 API,而该代理既能审计请求,又能注入相应的 API 密钥——这样一来,Claude 就能访问需要认证的端点,而无需直接接触 API 凭据本身。
过去一年半的经历如何改变了你对自己这门手艺的看法?
Thariq 在上午的主题演讲中谈到了 Fable 级模型带来的一种失落感,我们在对话中进一步深入探讨了这一点。我一直把这种现象称为“深蓝时刻”。
Simon:我们来聊聊人的因素。现在很多人感到一种失落,因为他们原本视为自己在构建软件中的角色,正被模型大量取代。你怎么看这件事?过去一年半如何改变了你对自己这门手艺以及你所创造价值的看法?
Thariq:Cat 和 Boris 就是很好的提醒,让你必须更有野心。他们总说:我们发展得太快了,我们必须站在前沿,我们必须做到最好。这对我来说是一个持续的提醒——每当我做事慢了,我就会想,好,我能做得更快吗?我能在这里更有野心吗?而很多时候答案就是 Claude,因为 Claude 会随着你的使用而变得更好——上次我试这个的时候,用的还是之前的模型。关于你说的失落感:我认为这是真实存在的。如果你只是试图做 LLM 出现之前就在做的同样的工作,而现在它变成了一条提示词,那我觉得确实是一种悲哀的感觉。而抵消这种失落感的方式,就是变得更有野心。我觉得 Jared 就是一个很好的例子——他在奥克兰的公寓里,用大约一年时间手写了全部 Zig 代码,几乎没出过门,而且乐在其中。现在我看到他把整个 Bun 用 Rust 重写,他也乐在其中——这件事野心大得多,而这就是他抵消失落感的方式。总的来说,就是问自己如何去做更大的事、做更多的事——我觉得成功是有趣的。这改变的是你的野心。
“抵消这种失落感的方式,就是变得更有野心”——这句话也精准地概括了我自己在这个问题上的立场。
Simon:那么 Cat,从产品管理的角度来看,这又是什么样的呢?
Cat:我觉得产品这个角色真的是每个月都在变。我们团队里所有的产品经理都是工程师、设计师、产品经理的混合体——他们大多数人以前其实都是全职工程师。对我们来说,这真的意味着哪里有缺口就往哪里补。如果我们有了一个想法,但没有激发任何工程师去把它做出来,那我们就应该自己动手做,把它放进 notebook 里,然后激发别人把它带到生产环境。如果设计看起来有点不对劲,那就找一个类似的页面,先做一版初稿设计,再拉一个非常注重细节的人来填补空缺。或者如果我们注意到我们团队和产品在公司内部的采用率更大了,更多的人需要知道 Claude Code、Claude Tag 和 Cowork 接下来会推出什么——那就让我们自动化整个发布日历的规划,自动化获取那些异步状态更新,这样就不用去打扰别人,同时确保我们在内部公告频道里的更新内容详尽且切中要点。对我们来说,这很大程度上就是理解一个好想法和把它送到客户手中之间目前存在的差距是什么,以及我们如何尽可能多地自动化这个过程。
这反映了我注意到的一点:当你能以快得多的速度产出代码时,花在等待别人做决定上的时间就变成了一个更明显的瓶颈。能够做产品决策的工程师可以推进得飞快,而做出一个错误决策的代价也不再那么令人望而却步。
有没有哪个时刻 Claude 让你感到惊讶?
Simon:有没有哪个时刻 Claude 让你感到惊讶?就是模型做了你以为它做不到的事情的时候?
Thariq:我发过很多关于 Claude 视频剪辑的内容,但最近我在 ACM Agentic 会议上做了一场演讲,我问:“嘿,各位,你们有剪辑好的视频吗?我想把它发出来,和我的传播团队分享。”他们说:“哦,剪辑花了太长时间。”于是我要了原始文件。他们把我上台讲话的视频、演示文稿的视频和音频文件发给了我,说:“祝你好运。”我把这些连同我的 HTML 演示文稿一起交给了 Claude,说:“嘿,你能把这些剪辑到一起吗?”它做的事情真的令人难以置信——我已经准备好发布了。它转录了整段视频。它注意到我的演示文稿视频有时有点奇怪——中间弹出了一个自动更新窗口——然后它说:“哦,我可能不应该用你演示文稿的视频。我要做的是把它切开,弄清楚你讲到哪张幻灯片了,然后用 HTML 源码来代替。”所以它显示的是 HTML 源码。然后它有我讲话的视频,但我只占了舞台的一小部分,所以它会动态裁剪到我所在的位置——而且我在来回走动,所以它会随着我的走动追踪我。它还在转录我说的话。
Simon:这是 Fable,对吧?
Thariq:对,是 Fable。这是一个很好的提示词,但只是一次性的提示词。然后我让它添加一些有趣的动画和图形,我简直被震撼了。它能用 ffmpeg,也能用 Remotion。
这是 Thariq 的视频,展示了他如何用 Fable 剪辑 Fable 自己的发布视频,这是那个发布视频。
它还有什么做不到的?
我不好意思承认,我发现自己很难想出 Fable 5 和 GPT-5.6 这类前沿模型无法完成的任务。
Cat 仍然不认可它的 UX 设计能力:
Simon:它做不到什么?有哪些地方仍然让你失望——哪些地方你在等 Claude Fable 6 帮你解决?
Cat:我希望它在设计和用户体验品味上能更好。现在已经到了这样一个程度:如果我写出一段带有详细功能行为规格说明的提示词,它通常就会按那个方式表现。但内边距可能不对,或者界面还不够令人愉悦。它会借鉴应用设计的现有最佳实践,但对于前沿 AI 产品来说,还有太多我们尚未设计出来的全新交互体验。
Simon:存在一种 Opus 美学——你看到某个东西就能说:“对,这是 Opus 设计的。”如果我们能超越这一点就好了。
Cat:是的。我非常期待未来的模型能成为交互设计的思想伙伴。
Thariq:它做不到什么?我很希望看到它更多地与真实世界互动。它能解决科学问题吗?它能编排实验吗?这中间涉及一定量的编码工作,但它还需要另一种对更广阔世界的品味。
Anthropic 文化的哪些部分应该被其他公司借鉴?
我觉得这会是一个很好的收尾问题:
Simon:你们认为 Anthropic 公司文化中哪些部分独特地帮助了 Anthropic 在使用这些工具时保持高效,而其他公司应该借鉴的?人们应该从你们这里采纳哪些文化诀窍?
Cat:我分享一个关于 Claude Tag 的。Claude Tag 在公共频道中使用时效果最好,而且当你大多数频道都是公开的时候效果最佳。Claude Tag 能够搜索所有公共频道,获取尽可能多的上下文,从而给你最高准确度的答案——而它只有在能访问所有内容的情况下才能做到这一点。
Thariq:我在主题演讲中提到过这一点,但它对我太重要了,我想再强调一次。联合创始人说我们不会自己跟自己谈判,我认为这非常重要。你可以在脑子里想象各种权衡取舍,然后说服自己放弃做某件有雄心的事情——或者你也可以直接尝试去做那件有雄心的事情。我们经常问的是:如果我们直接做了会怎样?这真的是一个真实的权衡吗?如果是,那为什么——有什么证据证明这是一个真实的权衡,而不只是听起来合理?让权衡取舍自己向你显现。尽可能保持雄心。
你用 Claude 做过最离谱、纯粹因为“能造出来”就造了的东西是什么?
这个我也忍不住要放进来。
Simon:你用 Claude 做过最离谱、纯粹因为“能造出来”就造了的东西是什么?
Thariq:我正在做一款 2D 街霸格斗游戏,里面有一个角色是我自己——还有我的朋友们。它用 Claude Code 来给 Gemini 写提示词——说实话 Seedance 模型相当不错——用来生成视频动画。效果非常好;它在写提示词方面太强了,而且它还能校验帧,判断一段动画是否合格。
Simon:你生成的是街霸 2 那种级别的 2D 精灵图吗?
Thariq:对,没错——就是 2D 精灵图。动画看起来非常惊艳。而且它还能自己算判定框——它会说“哦,你的拳头在这里,我来把 JSON 格式的判定框画出来。”简直不可思议。
Cat:我的就简单多了。我是个狂热的攀岩爱好者,我很多朋友也攀岩,所以我们用 Claude Code 做了个小应用,用来记录我们正在攻克的所有攀岩路线。我们还经常一起出去野外攀岩,所以就让 Claude 用 workflows 做各种调研。Workflows 太棒了——我们把它定位成编码工具,但它在旅行深度调研方面也极其好用。我还负责规划团队团建,它很擅长找能容纳我们所有人的场地。我用 workflows 调研所有我们可能想去的攀岩目的地,以及从我们各自所在地有哪些直飞航班。它会去 Mountain Project 上找所有符合我们难度等级的攀岩线路,还会找 Airbnb。而且我不喜欢徒步,所以我特别在意攀岩路线要非常短——从停车点到岩壁本身的步行距离要非常短——它也会按这个条件筛选。用现有应用的话,我得手动在 Mountain Project 上一个个点,但有了这个,我只要把所有偏好输进去,它就成了为我们量身定制的应用。
Simon:所以你基本上是在用 vibe coding 给攀岩做一个 Jira。
Cat:没错。
观众:有没有计划做评测构建工具和智能体可观测性方面的东西?
最后我们留了几分钟给现场观众提问。
听众:你们近期有没有计划构建更多的评测工具,方便我们构建评测数据集,以及更多的可观测性工具来监控智能体和工作流的性能?
Cat:我们考虑过构建评测工具,但我认为真正的瓶颈往往在于,客户需要花很长时间才能构建出真正高质量的评测。所以我觉得工具本身并不是主要制约因素,更重要的是如何构建出色评测的技能。这是我们既愿意在内部投入、也希望对外分享一些最佳实践的领域。
听众:记忆功能目前是如何设计的——你们会不会从文件转向数据存储?
听众(Sai):我对记忆和多人在线功能很感兴趣。记忆功能目前是如何设计的?我猜是基于文件的。其次,你们有没有考虑过一个正交方向,即这些记忆实际上需要数据存储而不是文件,以便更好地扩展?
Thariq:目前 Claude Tag 中的记忆是按频道区分的。该频道中的每个 Claude 都有共享记忆,实例有会话——但会话可以回写到主记忆。我们做了很多记忆方面的研究,而正确的记忆实现方式有时并不那么直观。我们一直在进行记忆实验。目前 Claude Tag 中的实现方式是每个频道对应一个 markdown 文件。
2026 年 7 月 21 日
Earlier this month I hosted a fireside chat session at the AI Engineer World’s Fair with Cat Wu and Thariq Shihipar from Anthropic’s Claude Code team. We talked about Claude Code, Claude Tag, Fable, coding agent security, evals, tool design, and how Anthropic use these tools themselves.
The full video of the session is now available on YouTube. Below is an edited copy of the transcript, with extra links and my own bolded highlights.
A few top-level notes if you don’t want to watch the video or wade through the whole transcript:
- Claude Tag (Claude’s new collaborative Slack integration) now lands 65% of the product engineering PRs for the Claude Code team.
- Claude Code ships features to Anthropic employees first, and only ships the features that demonstrate user retention with that cohort
- Critical changes to Claude Code are still reviewed manually, but the team increasingly relies on automated code review for the “outer layers” of the product.
- Adding examples to a system prompt is no longer best practice for models like Fable 5 or even Opus 4.8. The Claude Code system prompt recently reduced in size by 80%.
- Likewise, lists of "don’t do X and don’t do Y" can reduce the quality of results from the latest models.
- Dogfooding inside Anthropic is called "ant fooding".
- Anthropic really believe in their auto mode, and see that as an enabling technology for Claude Tag.
- Thariq advises offsetting coding-agent-induced Deep Blue by "being more ambitious" with the work you take on.
- Fable is competent at editing video, and Thariq used it to edit its own launch video.
- Anthropic’s culture of working (internally) in public is key to their success, as demonstrated by the way they use Claude Tag in their public Slack Channels.
How has what you do day-to-day changed in the past year?
Simon: Claude Code came out in February of last year — it’s under a year and a half old, and it was originally just a bullet point on the Claude Sonnet 3.7 launch. How has what you do on a day-to-day basis changed in the past year, now that we have these coding agents that actually work for us?
Cat: I remember when we first came out with Claude Code and Sonnet 3.7, you would give it a task and you would have to closely monitor every single little thing it tried to do. I would read every permission prompt extremely carefully. I would frequently say no — no, no, no, did you check this file? Did you check that file? And now it’s been incredible with every model generation. I feel like we’ve all gotten a chance to take a step back and delegate a lot more of the menial implementation to Claude. It’s freed up a lot of our time to think about more creative work, like: what is the right experience that we should be providing to our users, now that we know Claude Code can implement a lot of it? And now with Fable it’s a totally different step change improvement. We see for a lot of our use cases that you can actually one-shot a ton of features with Fable now.
Thariq: I remember the first text I got about Claude Code. One of my best friends was like, “You need to go try Claude Code.” It was about when Opus 4 came out, and I tried it and I was like, “Oh, shit. I need to work at Anthropic now.” And that was Opus 4 — great model, but you were reading permission prompts. It’s kind of crazy how much amnesia we have, where I’m like, oh, auto mode has always been here, right? I don’t even remember pressing yes and allow. For me, the big thing I’m trying to push myself on is that we have to do higher quality work than we’ve ever done before. The outputs are incredibly high quality. I’ve been using it to edit videos a bunch, and I’m like, okay, it has to meet the very exacting demands of our brand team in a couple of hours or we just can’t do it. That’s how I’m trying to shift with Fable: the best work we’ve ever done, faster than we’ve ever done it before.
What piece of conventional software engineering no longer holds?
Simon: What’s a piece of conventional software engineering that was true a year ago that you don’t think holds anymore in this new world?
Cat: One of the biggest shifts we’re seeing in the eng skill set: two years ago it was pretty typical for a product manager to go talk to a bunch of customers, align over the course of six months with cross-functional teams on some PRD, and write a thorough spec on exactly how we’ll implement this before the first line of code gets written. Now things are completely turned the opposite way. For a lot of engineers, the push I would give to folks in the room is to develop more of your business sense and product sense on what it is we should build, because the timeline between having an idea and building it is so much shorter — it’s down from six to twelve months to maybe even a week. That means all of us need to have better taste on what is worth building, what will actually inflect the businesses we’re working on. So it’s an increase in value on product taste and business sense, and a bit lower on execution in most product domains. Of course, for infra there’s still a very heavy emphasis on making sure all the details are right.
Thariq: For me, it’s that rewrites are now good.
Simon: The worst thing you could do is now actually fine!
Thariq: Exactly. All the Mythical Man-Month stuff — never rewrite — I’m pro-rewriting now. If you have a good test suite — and I think the rewrite actually forces you to make sure you have a good test suite — but I think what people undercount is that a codebase is a spec, and maybe it’s the only copy of the spec that you have, because no one knows every branching part of the codebase. You can take this as an artifact and distill it or create other versions of it. We rewrote Bun in Rust and it works great — it’s live for me right now.
Simon: You’re not shipping Claude Code on Bun-in-Rust yet, right?
Thariq: Internally we have.
(Actually it looks like Anthropic started shipping Claude Code on Bun-in-Rust to everyone on June 17th.)
What kind of things are non-engineers doing with Claude Tag?
Simon: The other big launch recently was Claude Tag — that’s what, a week old now, at least for the rest of us. I understand it’s being used at Anthropic by non-engineers a great deal. What kind of things are non-engineers doing with Claude Tag?
Cat: Claude Tag is a Claude that lives in your team’s collaboration tools. We launched it last week within Slack. The thing that’s different about Claude Tag is it’s multiplayer by default. Once you add Claude Tag to a Slack channel, you can chime in, your teammates can chime in, and you can collaborate together on the PR. The other big difference is that it’s proactive instead of reactive. You can tell Claude Tag, “Hey, monitor every bug report in this channel, put up a PR to fix it, and tag the engineer who last touched this part of the codebase,” and it’ll do it for the lifetime of the channel without you having to manually tag it in. And the third big shift is that we’ve added team memory into this. If you tell Claude Tag your preferences in the channel, it’ll remember them for every future post. If you always want it to debug outages but you don’t want it to debug warnings, just tell it that in natural language in the channel and it’ll remember it for you and everyone else on your team.
Internally, we see Claude Tag as the evolution of Claude Code. We see this as a large shift in how we work internally. Claude Tag currently lands 65% of our product eng PRs.
Simon: For all of Anthropic, or just for Claude Code?
Cat: This is just for our product engineering team — our internal version of Claude Tag lands 65% of our product PRs right now. And this is a huge shift; this is more than 50% of our PRs. The way we see people split work between Claude Code and Claude Tag is: Claude Code is still the best place for your most complex tasks, when you’re interactively iterating with the agent. But Claude Tag is great for having it work proactively on your behalf, so you no longer need to manually kick off Claude Code for all the bug reports that come up for features you’re working on.
Thariq: And for non-coding cases: for example, before this talk we asked Claude Tag, “Hey, when is Fable releasing?” We wanted to make sure we’d line it up with the announcement. Claude Tag would search our Slack and look at who’s been saying what. As a search engine for your company, it’s really valuable. It has all the context for your product, so you can ask it metrics-related questions — often when you’re making decisions you want them informed by what the metrics say, so you hook it up to your event store. I’ve seen our marketing team do things like, "Hey, tell me about this feature." They’re not programmers, but Claude is a programmer — it can clone the codebase and say, "This is the feature, this is what it looks like, this is a recording of me using the feature." It enables a whole wide variety of things, and I think we’re still early in figuring that out.
Claude Tag as the team collaborative layer
Simon: One of the problems I’ve had with coding agents is that I get how to use them as an individual, but I’m not really clear on how to use them in a team environment. It sounds like Claude Tag is your current answer to that team collaborative layer for this stuff.
Cat: Exactly. And a large percentage of our sessions are actually multiplayer right now. Maybe I say, “Hey, I think we should implement this new feature in Cowork,” and I’ll tag in Claude Tag to do a first pass at it. Then I’ll tell Claude Tag, “Share a recording of your final implementation,” and I’ll tag in design to take a look. They’ll nudge it, then pass it on to eng to take it to the finish line and get it out to prod. It’s been this very fluid experience. We’re still trying to iron out what the social dynamics are for steering the same session, but we’ve found that people just observe how others use it and follow those social norms — it’s been pretty intuitive for us to integrate Claude Tag into our teams.
Thariq: It’s great for teaching people, and also for reducing slop, because the fact that everyone is seeing you use Claude together sort of levels up how you use Claude as well.
This reminded me of how Midjourney solved the challenge of teaching people advanced image prompting by enforcing prompting in public in their Discord channels.
How do you decide which features are worth building when building is so much cheaper?
Something I’ve found really hard myself is knowing when a feature is worth shipping now that the cost of actually building features has dropped so much.
Simon: How do you deal with the hardest problem in all of engineering — prioritization? How do you decide which features are worth building and shipping when building a feature is so much more inexpensive now?
Cat: This is the hard thing. There are a few ways we approach it. One is we dogfood our products every single day. Whenever there’s something we want to be able to do in our products that we’re not able to, instead of finding a different solution we fix our product so it can support that case. We have a very heavy dogfooding culture internally. Before we share our products with everyone in the world, we share them with everyone within Anthropic, and with some early customers who give us very honest feedback about it — the more brutal the better — and we iterate until people love it. We have an internal bar for the number of active users and the amount of retention a feature has to have before we share it with the world. Because this bar is very clear, every engineer knows what they’re trying to hit. I think this also levels up our polish, because if the feature isn’t polished, people will churn — and then we shouldn’t ship that feature.
Using internal user-retention to decide if a feature should ship makes a whole lot of sense to me.
Do you have an example of a feature which surprised you?
Simon: Do you have an example of a feature which surprised you? You rolled it out and the engagement was off the charts — something unlikely to be shipped that turned into a real product thing.
Cat: I do have one. A lot of folks on our team love remote control. Remote control lets you use your mobile device, or Claude in the web browser, to connect to a local Claude Code session running in your CLI. I never have this need, because I just kick off the task directly on mobile and it runs in a cloud session without using my local environment — I think because I’m doing very easy coding tasks. It was something I didn’t totally understand; I was like, hey, people should just set up remote dev environments. But in practice, once we rolled out remote control, so many people I talk to told me that what they do every night is plug their laptop into a power charger, open a bunch of remote control sessions, lock the screen, and then use their mobile phone from their couch to control Claude Code. So this has become a flow we’re now leaning into that I didn’t originally get — but now I do.
Does a human review every line of production code in Claude Code?
One of the over-arching themes of the conference was review: how much attention to people spend to reviewing code written for them by coding agents. I was very keen to hear the Claude Code team’s take on this!
Simon: How does code review work? Does a human being review every line of production code that makes it into Claude Code? And if not, what are you doing — how do you keep the quality up?
Thariq: It varies on the task a lot. For important areas we have code owners. The system prompt is an example where we have a code owner — you really need to get their approval.
Simon: So the code owner is directly responsible for the quality of that area of the code.
Thariq: That’s right.
Cat: And they need to approve any PR that touches it.
Thariq: We have our code review GitHub bot review everything — that goes on every PR, and often it’s doing the bulk of the review. Something I’ve seen on the team is that for more complex PRs you might make an artifact to explain the PR so that other people can then review. And we invest a lot into verification, CI/CD, things like that, to make sure that any time anything fails we have a test. We have a really robust environment where Claude can control Claude Code and test it. So there’s a multi-pronged approach to code review.
Cat: In general, we are trying to move to a world where humans don’t need to be in the loop. For the most critical changes to the core of Claude Code, and the cores of other products, there is always a code owner and they do manually review all the changes. But increasingly, for the changes at the outer layers, we actually have Claude code review fully review those. That sounds pretty scary, but we’ve had a six-plus-month-long process to get here, and there are baby steps that you take to build up trust with code review. In the beginning we had human review for everything, and then increasingly we would say, okay, for code changes that touch these files, code review is catching 100% of the issues there — so we actually don’t need a human manually reviewing those. And when we have incident review, we look at the PRs that caused the incident and say, okay, how do we update code review to catch that? — and we take those PRs and add them to an eval set to make sure our future changes to code review never regress that metric. Removing humans from the code review loop is a big step forward. It can sound scary, and it’s not something you can do overnight, but it is something you can do through many months of investment in the infrastructure to give you the confidence that code review is catching everything you care about.
So the key seems to be constantly iterating on the automated review systems themselves, in order to build trust in them over time.
How does a new model affect your intuition for what it can and can’t do?
We got deep into evals—another hot topic throughout the wider conference.
Simon: I know that Opus 4.8, if I ask it to build me a JSON endpoint that runs a SQL query and outputs JSON, is just going to get it right — that’s not something I have to review closely. But then a new model comes along and I don’t know how to build trust in Fable quickly, that it’s not going to mess things up that Opus didn’t. How does the new model affect your intuition for what it can do and what it can’t do?
Cat: The main reason we’re building up this eval base over time is so that new models can be a drop-in replacement. When we have a new model, we run the whole eval set and make sure that, for example, Fable is strictly better than Opus 4.8 — and that gives us the confidence to drop it in.
Simon: Are those model evals for Anthropic as a whole, or Claude Code team-specific?
Cat: We have both. We have evals on our team, and we run code review across every repo within Anthropic, so we have evals for that. And for things like auto mode, we not only have evals across every user within Anthropic — we’ve also commissioned multiple external testers to red team it, to create environments with prompt injections and malicious inputs, and make sure that auto mode doesn’t let any of those pass.
How do you build confidence that a system prompt tweak results in better output?
Simon: I want to know if the system prompt improvement I made actually improved the product — that’s the most basic form of product-specific eval, and I still don’t have a great feel for how to do that. Is that something you’re doing such that you have complete confidence that a tweak you’ve made to the system prompt results in better output?
Cat: We don’t have complete confidence, but we do a lot to make sure that we don’t regress performance. The starting point is a suite of external evals that we trust, and we complement that with an even larger suite of internal evals that we trust. To start, we mainly optimize for capability: given a complete definition of a task and the full codebase, does Claude make the right decisions, fully fix the bugs, and pass all the tests? That’s the starting point and the thing we optimize for, because it’s most directly what users want. But there are a lot of behaviors that impact how users feel when they work with Claude Code. For example, people really don’t like it when Claude Code says it’s time to go to sleep. Or people really don’t like it when it says, “Hey, I finished two out of five parts — do you want me to continue?” Yes, please continue. So we’re building up a set of behavioral evals to catch these. And as we get user feedback — please be loud with us about your user feedback — we rank the priority issues and go down one by one and build evals for each of them. It’s not 100% coverage, but it is a priority for us to increase the coverage.
How much interaction is there between the Claude Code team and the model training teams?
Simon: How much interaction is there between the Claude Code team and the teams at Anthropic who are training the models in the first place? Is that quite a close collaboration?
Cat: Across Anthropic, we all work quite closely together. We meet often to talk about what we expect the next generation of models to be able to do. Our research team has also been amazing about showing this publicly — we often talk in our blog posts about how we’re targeting ever-increasing longer-horizon work, and how we train Claude itself to be honest, harmless, and helpful. We also put a lot of effort into making sure it’s aligned with your intent, even if your intent is expressed in a fuzzy way. Of course, try your best to be specific about what you want, so Claude has all the context — but even when you’re not specific, we teach Claude to make good assumptions. It’s been a productive partnership.
The system prompt has been reduced by 80% — what have you been able to drop?
So many useful prompting tips in this section!
Simon: Thariq, you mentioned this morning that the system prompt for Claude Code has been reduced by 80% because of Claude Fable. Can you go into a little more detail? What kind of things have you been able to drop?
Thariq: It wasn’t just Fable — it was Opus 4.8 as well, and going forward, future models. We have different system prompts for different models now. One of the patterns we saw is that we were over-constraining Claude. The initial, maybe Opus 4-ish models wanted a lot of examples, and removing examples was extremely helpful, because it was just more creative than the examples we gave it.
Simon: That’s really interesting, because one of the top prompting tips I give people is: give it examples. If that’s no longer true, that kind of breaks my prompting model a little bit.
Thariq: Same here — I was surprised to hear that. I think now it’s more about the shape of what you give it — the tools you give to Claude, your system prompt, things like that. The other thing we did is try to give it more context and fewer “do not do this” instructions, because that’s a very strong impulse for Claude, and especially if it conflicts with user instructions later on, that can be extremely confusing to Claude — “I’ve got this skill that says this and the system prompt says this.” So we try to have fewer hard constraints, more context, and fewer instructions overall. It’s definitely a science — it took a bunch of evals to build.
Cat: In general, when you’re prompting these models, you should always think: are there edge cases to the instruction that I’m giving it? When we went back and reviewed all the instructions in the Claude Code system prompt, we found a few cases where yes, this statement is 90% true, but there’s a real 10% of cases where it’s not true. We didn’t want to constrain the model, or confuse it into thinking it should always do this. One good example is verification. Everyone here wants Claude to verify its work, and we had some instructions in the prompt that said: if you make a front-end change, always verify. But there’s a limit to it. If it’s changing copy from one string to another string, and the user says “just make a quick fix and update the test,” maybe you don’t want to verify. So we’ve adjusted our wording from “always verify, verify, verify” to something like: most of the time when you’re doing front-end work you can’t fully understand the experience by hitting the backend endpoints, so when you make larger changes to the user experience, please run the app locally. And in fact, that instruction probably isn’t even good either, because what is a large change? Maybe it should test small changes too. In general, whenever you give a prompt to the model, you should think about the ways in which it could be misinterpreted by a well-intentioned human, in order to better understand how the model might interpret it — and soften the prompt so that it’s actually 100% accurate, because you’re giving this prompt to the model 100% of the time.
Simon: What’s fascinating about that is you’re relying on the model’s judgment — and that’s got to be an Opus/Fable-level thing. Models a year ago did not have the level of judgment necessary to decide whether they were going to test a change or not. But that does break down if you’re building for a wide range of models and trying to run the cheaper models for cheaper tasks.
Cat: We actually have a different system prompt per model now, for this very reason. It’s only our most frontier models that have this 80% token decrease — the older models still have the full system prompt.
Simon: Do you think Fable and Opus are smart enough to prompt Haiku with more details, because they understand that Haiku has less judgment, less taste?
Cat: We haven’t been able to eval it — we don’t have any hard data to show it.
Thariq: There’s a tough thing with smaller models sometimes, because sometimes the larger models can be more token-efficient on a hard problem than the smaller models. So there’s a bit of intuition to build there — sometimes you really just want frontier intelligence almost all the time. The Pareto curve shifts, and it’s hard to find.
Simon: A year ago I did not trust a model to write a prompt. Today the good models are very good at prompting — a lot of my prompts are written by models, which feels absurd but works really well. What helped me come to terms with that was thinking about subagents, which are entirely about a Claude model setting up a prompt for another Claude model.
Thariq: Workflows are actually a really good example of this, because it’s Claude not just prompting a single subagent, but prompting the orchestration of many subagents, and each one of them gets a very detailed prompt. It’s almost a level above just spawning a subagent. I’ve also been using it on my personal machine, giving it the Gemini API and saying: here, generate images. It’s way less lazy than I am at prompting an image model. It’s just Claude prompting Claude all the way down.
Cat: I think Claude also wrote the prompt for the workflow tool.
Simon: I’ve read that prompt — it’s a good prompt. That’s actually a frustration I have with Anthropic generally: you publish the prompts for Claude Chat, but you don’t include the tool prompts and the Claude Code prompts. I still have to run a proxy to intercept them. I would love it if the Claude Code prompts were deliberately published — they’re the documentation. They’re how you know what the tool can do and how it works.
Cat: I’ll write down that feature request. I’ll have Claude Tag do it.
Interesting to note that OpenAI’s prompting best practices for GPT-5.6 includes similar advice for their latest models:
Favor leaner prompts
Removing repeated instructions and examples and simplifying tool descriptions can improve task performance and token efficiency. In a sample of internal coding-agent eval runs, configurations with leaner system prompts improved evaluation scores by roughly 10–15% while reducing total tokens by 41–66% and cost by 33–67%.
What’s your bar for introducing a new tool?
Simon: Claude Code is basically a big bag of tools. What’s your bar for introducing a new tool? How do you decide when it’s worth doing that additional engineering at that level?
Cat: Do you want to take it? You introduced one of the best tools we have.
Thariq: My career peaked when I introduced the ask user question tool. It’s really hard. Especially for some tools — ask user question is Claude’s tool to ask you — so it’s hard to eval, and sometimes it’s more of a user preference thing. Back then we had fewer evals, so it was very dogfooding based — or “ant fooding,” our ant version of that. But overall we’ve been trying to trend towards fewer tools. The last set of tools we introduced was the task tool, I think — and we try to give Claude more general versions to do things.
What’s the latest evolution of your file editing tool?
I have a long-running fascination with file editing tools—they were the subject of the old Aider code editing leaderboard, and I’ve watched with interest as they’ve evolved in different coding agents from search-and-replace based to line-number-based to more complicated patterns.
The Claude API docs describe a text editing tool that’s recommended for building against the API, but Claude Code seems to use slightly different approaches here.
Simon: One of the most interesting tools is the file editing tool — you can have file editing as a tool, or you can tell it to use sed and grep and do things that way. What’s the latest evolution of your file editing tool?
Thariq: We still have one, but for example we removed our grep and other search tools — glob tools — in favor of native bash. Like I said in my talk earlier, the models are kind of more of a biology than a physics, and tool design especially is quite hard. I’m not sure if Cat disagrees and thinks there’s a science to the eval of it, but I think tool design is more of an art, maybe — or a biology.
Cat: I largely agree, but in general as we introduce more tools, we try to keep the cardinality pretty low and make sure that every tool we add has a distinct function from every other tool, so that Claude can very easily distinguish when to call each. For file edit, the reason we have it is actually because we can render it. We show people when Claude makes a file change, and there’s this nice dedicated UI that says: do you approve this edit to this file? The reason we had a dedicated file edit tool was so that we could deterministically know that Claude was making a file change, so we could show people this nice UI. A lot of new users onboarding still really like this experience, so we’ve kept it around. But for a lot of us who are on auto mode right now — hopefully you’re not on YOLO mode — I don’t think it actually matters, and we could probably just remove file edit and be totally fine.
What’s the advice within Anthropic for safely running Claude Code?
It’s the prompt injection question! Who better than Anthropic employees to explain how Anthropic sees the risk of prompt injection attacks causing their Claude Code instances to run amok?
It turns out they really trust their auto mode—and see that as the feature that enabled Claude Tag.
Simon: Let’s talk about safety and security. I am deeply aware of the risks of prompt injection, and there are so many bad things that can happen if somebody else tells my Claude Code what to do. I still mostly run Claude Code in YOLO mode and feel incredibly guilty about it. What’s the advice within Anthropic for safely running Claude Code?
Cat: Why not auto mode?
Simon: I am starting to use auto mode, but I don’t understand it enough to get how safe it is. As of maybe three weeks ago, I’m defaulting to auto mode.
Cat: Broadly within Anthropic, almost every single person uses auto mode. It is the best way to do long-running work in Claude Code while being safe. We’ve done extensive bashing. We have thousands of evals. We’ve commissioned many red teamers to create adversarial environments in order to trick Claude Code into doing bad actions, and we’ve mitigated every single issue that they found. We’re going to publish some evals in the coming weeks, but we’ve pretty much mitigated every attack.
Simon: That is a big claim.
Cat: We’ll share the evals for it so folks can assess, but we’ve been extremely diligent about identifying all the ways in which Claude might mess up and then updating auto mode to counter it. It doesn’t catch 100% of things — that would be way too strong a claim. But for the main categories of risks that we’re concerned about, like prompt injection and data exfiltration, the risks are far lower than the average human reviewer.
I am very much looking forward to learning more about their evals and approach to verifying auto mode.
Thariq: A little on how auto mode works — it’s useful to build this mental model. Whenever Claude is doing a turn, or a bash call, there’s a Sonnet classifier that is judging the tool call and also the context of the conversation — your instruction. There are some things around permissions that are dependent on your request: you don’t want to give git push permissions all the time, but if you say “push this to GitHub,” you want it to do it — and if you say “don’t push,” you want it to deny it. Auto mode will do that. That particular thing happens to me a lot, where Claude tried to do something because it’s very helpful and proactive, and auto mode saw “don’t do this” and surfaced it. So it’s good at the dynamic permissions that you yourself give inside the prompt, which I think is really important. It also works well with our sandboxing infrastructure, because sandboxing is one of those things where there are so many different edge cases that it’s hard for us to deterministically follow them. We have a sandbox, and when something needs to escape the sandbox — like a network request — auto mode can look at that request and ask: does this make sense? — and allow it.
Simon: I hadn’t realized auto mode is interacting with the networking sandbox as well.
Cat: It interacts with any permission prompt the user would otherwise see.
Simon: How old is auto mode? As a feature I had access to, it’s only a couple of months old, right?
(It was first made available to the public on March 24th.)
Cat: We’ve been using it within Anthropic since January, so we’ve been hardening it for quite a while. Anthropic is extremely focused on safety and security, and we’ve been working broadly across our alignment and safeguards teams to enable the rollout internally, build out these evals, and make auto mode even more robust before sharing it with the world.
Thariq: This is also the reason Claude Tag is so good — Claude Tag uses auto mode. I’ve heard a lot of build-versus-buy questions about a Slackbot, and I’m like: please, you probably shouldn’t build your own AI Slackbot. There are so many attack vectors. You have a feedback channel that users can post feedback into, and now your bot is reading it. The work we’ve put in with auto mode — and we have a general Swiss cheese defense for security; we also RL against this stuff — I think this is really what makes Claude Tag work. It works seamlessly with your permissions, and you don’t want to be prompt injected in your Slack.
Are there more security things in the pipeline beyond auto mode?
Simon: Are there any more security things in the pipeline that go beyond auto mode?
Thariq: I think we’re very secure. With Claude Tag you can provision your own credentials for Claude, so it doesn’t need to act on your behalf — you can have Claude as an identity, and that also makes it easier to audit and inspect what Claude is doing.
Simon: Because Claude Tag is influenced by anyone who can talk to it — it’s got a much wider pool of people telling it what to do.
Thariq: That’s right. And of course we have probes as well with Fable, which is a downstream effect of our safety and research work. I think this is the moment where you see Anthropic being an AI safety company really paying off: we really want Claude to be able to run in an aligned way over long periods of time, and auto mode has to be basically flawless for this to work — it’s all downstream of our being an AI safety company.
Cat: We also launched trusted devices for the remote control users out there who want to be safer. And for all of our remote environments, we support credential injection. If you want Claude Code to be able to access Datadog, but you don’t want Claude Code itself to hold the Datadog credential, you can set up our identity and credential management system so that the Datadog credentials are only usable by the agent but not accessible by the agent — we insert them on the fly when the agent tries to make a Datadog request.
I really like that credential injection pattern, where Claude Code can access an API via a proxy and that proxy both audits the request and injects the relevant API key—so Claude can access authenticated endpoints without having access to the API credentials itself.
How has the past year and a half changed how you think about your own craft?
Thariq talked about a sense of grief brought on by Fable-class models in his keynote in the morning, and we dived further into that as part of our conversation. I’ve been calling this Deep Blue.
Simon: Let’s talk a little bit about the human element. A lot of people are feeling a sense of loss now that so much of what they considered to be their role in building software is being subsumed by the models. How do you think about that? How has the past year and a half changed the way you think about your own craft and the value that you add?
Thariq: Cat and Boris are such good reminders that you have to be more ambitious. They’re always like: we’re growing so fast, we have to be on the edge, we have to do the best work we can. That’s a constant reminder for me — any time I’m slow on something, I’m like, okay, can I do it faster? Can I be more ambitious here? And oftentimes the answer is Claude, because Claude is getting better as you go — the last time I tried this, it was with the previous model. On your point about loss: I think this is real. If you’re only trying to do the same work you were doing before LLMs, and now it’s a prompt, it is, I think, kind of a sad feeling. And the way you offset that is by being more ambitious. I think Jared is such a good example — he hand-wrote all of the Zig code in his Oakland apartment in about a year, barely left his house, and had so much fun doing that. Now I see him rewrite all of Bun into Rust and he’s having so much fun doing that — it’s so much more ambitious, and that’s how he offsets it. Generally it’s asking how do I do the bigger thing and do more — I think success is fun. It’s changing your ambition.
“The way you offset that is by being more ambitious” neatly captures where I’ve landed on this issue myself as well.
Simon: And Cat, what does that look like from a product management perspective?
Cat: I feel like the product role just changes every single month. All the PMs on our team are this mix of engineer, designer, PM — most of them actually used to be full-time engineers. For us it really means plugging in whenever there’s any kind of gap. If we have an idea and we didn’t inspire any engineer to go build it, then we should just build it, put it into a notebook, and inspire people to take it to production. If the designs look a little off, let’s take a page that’s similar, do a first-pass design, and tag in someone who’s very detail-oriented to fill in the gaps. Or if we notice that our team and product adoption is bigger within the company, and more people need to know what’s coming down the pipe for Claude Code, Claude Tag, and Cowork — let’s automate figuring out our whole launch calendar, let’s automate getting those status updates asynchronously so we’re not bugging people, and make sure our updates in our internal announce channels are fully detailed and to the point. For us it’s very much understanding what the gap is right now between a great idea and getting something to our customers, and how do we automate it as much as possible.
This reflects something I’ve noticed: when you can produce code so much faster, time spent blocked awaiting a decision from someone else becomes a much more notable bottleneck. Engineers who can make product decisions can move a whole lot faster, and the cost of getting one of those decisions wrong is much less prohibitive.
What’s a moment when Claude has surprised you?
Simon: What’s a moment when Claude has surprised you? When the model did something you didn’t think it would be able to do?
Thariq: I’ve posted a lot about Claude video editing, but most recently I gave a talk at the ACM Agentic conference, and I asked, "Hey guys, do you have the edited video? I’d love to post it and share it with my comms team." They said, "Oh, it’s taking so long." So I asked for the raw files. They sent me the video of me talking on stage, the video of the deck, and the audio file, and said, "Good luck." I gave this to Claude, along with my HTML deck, and said, "Hey, can you just edit this together?“ And what it does is honestly incredible — I’m ready to ship it. It transcribes the entire video. It notices that sometimes the video of my deck is a little weird — there’s a popup of an auto-update in the middle — and it goes, ”Oh, I probably shouldn’t use the video of your deck. What I’m going to do is slice it up, figure out which slide you’re on, and use the HTML source instead." So it displays the HTML source. Then it’s got video of me, but I’m only taking up a small part of the stage, so it’s cropping dynamically to where I am on the stage — and I’m pacing, so it’s tracking me as I pace. And it’s transcribing what I’m saying.
Simon: This was Fable, right?
Thariq: This was Fable, yeah. It was a good prompt, but it was a one-shot prompt. Then I asked it to add some interesting animations and graphics, and I was just blown away. It does ffmpeg, it does Remotion.
Here’s Thariq’s video on how he used Fable to edit Fable’s own launch video, and here’s that launch video.
What can’t it do yet?
I’m embarrased to admit that I’ve been finding it quite hard to come up with tasks that frontier models like Fable 5 and GPT-5.6 are unable to accomplish.
Cat still doesn’t rate its UX design skills:
Simon: What can’t it do? What are the things where you’re still disappointed — where you’re waiting for Claude Fable 6 to figure it out for you?
Cat: I want it to have better design and UX taste. It’s now at the point where if I write out a prompt with a detailed spec of how I want a feature to behave, it will usually behave that way. But the paddings might be off, or the interface just isn’t delightful yet. It leans on existing best practices for how apps are designed, but for frontier AI products, there are so many new interaction experiences that we have yet to design.
Simon: There’s an Opus aesthetic — you can look at something and go, “Yeah, that was designed by Opus.” It’d be good if we could move beyond that.
Cat: Yeah. I’m very excited for future models to hopefully be interaction design thought partners.
Thariq: What can’t it do? I would love to see it interact more with the real world. Can it solve science? Can it orchestrate the experiments? There’s some amount of coding that goes into that, but there’s also this other taste of the broader world that it needs.
Which parts of Anthropic’s culture should other companies steal?
I figured this would make a great closing question:
Simon: Which parts of Anthropic’s company culture do you think uniquely help Anthropic be productive with these tools, that other companies should steal? What are the cultural hacks people should be adopting from you?
Cat: I’ll share one for Claude Tag. Claude Tag works best when you have it in a public channel, and when most of your channels are public. Claude Tag is able to search across all public channels to get as much context as possible to give you the highest-accuracy answer — and it’s only able to do this if it has access to everything.
Thariq: I mentioned this in my keynote, but it’s so important to me I want to re-emphasize it. The co-founders say we don’t negotiate against ourselves, and I think this is really important. You can imagine trade-offs in your head and talk yourself out of doing something ambitious — or you can just try to do the ambitious thing. We’re so often asking: what if we just did it? Is this a real trade-off or not? And if so, why — where’s the proof that it’s a real trade-off, and not just something that sounds reasonable? Make the trade-offs show themselves to you. Be as ambitious as you can.
What’s your favorite absurd thing you’ve built with Claude, just because you could?
I couldn’t resist throwing in this one as well.
Simon: What’s one of your favorite absurd things that you’ve built with Claude, just because you could build it?
Thariq: I’m working on a 2D Street Fighter fighting game with me as a character — and my friends as well. It uses Claude Code to prompt Gemini — and honestly the Seedance model is pretty good — to make video animations. It works great; it’s so good at prompting, and it can verify the frames to check whether an animation was good.
Simon: Is this Street Fighter 2-level 2D sprites you’re generating?
Thariq: Yeah, exactly — 2D sprites. The animation looks amazing. And it can also figure out hitboxes — it can be like, “Oh, your fist is here, I’ll draw the JSON hitbox.” It’s incredible.
Cat: Mine is much more simple. I’m a big rock climber and a lot of my friends climb, so we have this little app we built with Claude Code where we log all the projects we’re working on. We also go outdoors together a lot, so we have Claude do all this research with workflows. Workflows is amazing — we brand it as a coding tool, but it’s amazing for doing deep research for travel. I also plan our team offsites, and it’s good at finding venues that can fit all of us. I use workflows to research all the climbing destinations we might want to go to, and what has direct flights from where all of us are located. It goes to Mountain Project and finds all the climbs at our grade level. It finds the Airbnb. And I don’t like hiking, so I care a lot about it having a very short approach — very short walking distance from where the car parks to where the rock actually is — and it filters for this. With existing apps I have to manually click through Mountain Project, but with this I just put in all of our preferences and it’s a custom app for us.
Simon: So you’re basically vibe coding Jira for mountain climbing.
Cat: Exactly.
Audience: Any plans for eval-building tools and agent observability?
We had a few minutes at the end for questions from the audience.
Audience: Do you have any near-term plans to build more eval tools for us to build eval datasets, and more observability tools to monitor the performance of agents and workflows?
Cat: We’ve considered building eval tools, but I think the limiting factor actually tends to be that it takes a long time for customers to build really high-quality evals. So I think the tooling is less of the constraint, and more the skill set of how you build a great eval. That’s an area where we’re excited to both invest internally and hopefully share some best practices externally.
Audience: How is memory designed today — and would you move from files to a data store?
Audience (Sai): I’m interested in the memory and the multiplayer. How is memory being designed today? I assume it’s around files. And second, have you thought about an orthogonal direction where you would actually need a data store for these memories, instead of files, to scale it better?
Thariq: Right now for Claude Tag the memory is channel-specific. Every Claude in that channel has a shared memory, and the instances have a session — but the session can contribute back to main memory. We do a lot of memory research, and it can be kind of unintuitive what the right way to do memory is. We’re always running memory experiments. How it works right now in Claude Tag is a markdown file per channel.
21st July 2026