今天和我对话的是微软 AI 首席执行官穆斯塔法·苏莱曼。今天的开场白我会尽量简短——这周我在妻子的娘家农场工作,视频里能看到,而且这一集的内容确实非常劲爆。
我们聊了从穆斯塔法训练新模型的方法,到他批评 Anthropic 把 Claude 说得好像有意识一样。当然,我们也谈到了微软与 OpenAI 的关系,穆斯塔法如何看待当前围绕 AI 的负面民调和政治阻力,以及是否有足够好的消费级产品能克服这些困难。
就像我说的,这一集很劲爆。
好:穆斯塔法·苏莱曼,微软 AI 首席执行官。我们开始吧。
本次采访为长度和清晰度做了轻微编辑。
穆斯塔法·苏莱曼,你是微软 AI 的首席执行官。欢迎回到《解码器》。
很高兴再次和你交流。
我非常兴奋能和你对话。我们上次的对话是我在所有访谈中最喜欢的几场之一——关于 AI,它应该让我们有什么感受,以及它的用途。
微软发生了一些重大变化,可能还有一些关于人们对 AI 感受的非常重要的重新定位,我特别想和你聊聊这些。此外还有微软 Build 大会,这是微软的大型开发者会议,会上有很多新发布,也有很多关于计算机用途以及它们应该在哪里的宏大构想,我想深入探讨这些。
让我们从最根本的地方开始。这是《解码器》的一些深度内容,在讨论其他所有问题之前,理解这些很重要。自从你加入微软以来,你重组了 AI 部门的工作方式。你的角色也发生了变化。上次我和你谈话时,你负责一系列消费产品。后来这个职责被搁置了。你现在在训练新模型;你处于前沿。
请解释一下微软 AI 现在是如何架构的,以及它在微软内部是如何组织的。
我想大概在过去 15 到 18 个月里,我们一直在重新调整与 OpenAI 的关系,这花了一些时间。我认为最终成果是去年 10 月我们达成的一份新合同。其中包含了许多不同的条款,包括巩固和拓展合作关系,但关键的一点是,它让我们能够自由地独立追求超级智能,同时继续购买和授权使用他们的模型。
所以从去年 10 月起,我一直在组建超级智能团队,建设规模足够大的集群来训练前沿模型,并招聘专注于超级智能的团队。这对我们来说是一个相当大的转变,因为它让我能够专注于超级智能这一使命,最终促成了我们在本周 Build 大会上宣布的一些成果。我们推出了横跨所有模态的七款新模型等等。所以这是一个相当大的转变,我认为这是长期规划的结果,现在能够参与其中并在未来几年追求最前沿的技术,对我们来说是一种巨大的解脱。
这是你加入微软时的计划吗?
这至少是过去 18 个月以来的计划。我认为与 OpenAI 的关系经历了许多起伏。从很多方面来看,我认为这将成为历史上最成功的合作关系之一。这对 OpenAI 和微软都有好处,而所有良好的关系都会发展演变,我认为这只是我们关系演变的下一阶段。
让我具体问问这种演变。我们刚刚都看到了埃隆·马斯克与 OpenAI 及山姆·奥尔特曼之间的庭审。微软也参与了这场庭审,时不时会有微软的律师站起来说:“我们当时不在场。”然后有人会回答“是的”,事情就这样了。
但显然,在那场庭审中暴露出来的、以及这段时间以来一直清晰可见的一点是:最初的设想是 OpenAI 作为一个研究实验室提供模型,而微软负责构建产品。微软拥有市场推广方面的专长;它拥有企业级领域的专长,并且正试图以多种方式在消费市场重新站稳脚跟。这原本会是一次平台转型,研究工作留在 OpenAI,产品工作则在微软内部进行。
这正是发生变化的地方:OpenAI 想要开发越来越多的消费级产品。显然,考虑到你的新角色和新重心,微软也越来越想打造自己的模型。为什么会分道扬镳?这段合作关系在哪些方面行不通了?
我的意思是,我认为 OpenAI 由一个极其雄心勃勃的创始团队和山姆本人领导。因此,很自然地,随着他们开始获得更多发展势头并产生巨额收入,他们看到了实现全栈布局的机会。所以,他们不仅仅是开始涉足消费产品。显然,ChatGPT 取得了巨大的成功。他们还开始建设自己的数据中心。他们开始制造自己的芯片。关于他们自己消费级硬件设备的传闻也层出不穷。他们开始通过 ChatGPT Enterprise 直接将模型推向市场。因此,在过去的两三年、三四年里,他们在整个技术栈上都在大幅拓展,远远超出了研究的范畴。同样,微软也是如此。我的意思是,我认为这种合作关系至今已有五六年,并且还将持续四五六年的时间。
同样,我们是全球最大的科技公司之一。全球500强企业中有493家将大部分数据存储和处理在我们的系统上,使用Azure、M365和Teams。我认为人们常常低估了我们的规模之大,以及我们在企业领域的覆盖范围之广。因此,从长远来看——我指的是五、六、七、十年——我们必须确保完全可持续,不能仅仅成为他人知识产权的接收方,稍作修改调整后就投入生产用于我们的产品,而是必须能够真正独立自主,创造出世界级的模型。
我的意思是,超级智能即将到来。我认为它近在咫尺。因此,我认为它基本上将是有史以来最有价值的技术。从长远来看,我们绝不可能在结构上永远依赖第三方来提供这项知识产权。
所以,这一转型显然是由OpenAI等公司出现董事会问题所触发的。但随后,随着我和我的团队上任,我们开始着手构建这一能力,我们正处于这一转型过程中。我认为我们处于一个绝佳的位置,因为我们可以为OpenAI——我认为它从中获益匪浅——以及为我们自己,采取一个相当稳健、审慎、长期最优的立场。
我想花些时间谈谈超级智能。我现在先把它标记一下,因为我只想再深入了解一下这一转型的下一步。
庭审中有个非常有趣的时刻,微软CEO萨提亚·纳德拉说:“我不想成为英特尔,而让OpenAI成为微软。”这句话在微软CEO本人说出的语境下非常有趣,他的意思是:“我不想成为供应商,而让他们成为平台,提供所有价值并获取所有价值,然后我们可能被替换掉。我不希望ChatGPT运行在Azure上,然后OpenAI获得所有价值,之后他们可能把我们替换掉。”——正如Windows和英特尔之间随时间推移所发生的情况一样。
这是意识到问题了吗?是纳德拉主动来找你的吗?当时你说“好吧,OpenAI 的董事会出问题了。我们必须重回前沿,靠自己站稳脚跟”,那次会议是怎样的场景?那场对话具体是什么样,决策又是如何做出的?
我的意思是,这显然是萨提亚的决定,也是艾米、布拉德以及公司里许多其他人的决定。但我认为,这和其他事情一样:公司内部的变化是缓慢推进的,因为公司逐渐意识到我们前进的方向需要一些微调和修正。所以,早在去年11月那场董事会风波之前,这种调整就已经在发生了。而且我认为,随着时间推移,当你审视我们在越来越多领域直接竞争所面临的种种局面,以及由此产生的所有紧张关系时,这种调整会不断累积。同时,你也明白,像这样的合作关系不会永远持续下去。
我的意思是,OpenAI 想成为一家市值万亿美元的上市公司,它拥有惊人的收入,并且正在疯狂增长。他们希望拥有运营自由,能够从各种其他地方购买算力,建设自己的算力,并与任何他们想要的人合作。所以,这份合同是在两家公司在规模、体量以及需求平衡等方面都截然不同的时期签订的。我认为这在当时是合理的,但后来情况变得非常清楚:我们必须能够自己拥有并控制这一切,为我们自己的客户做好服务。
正如我所说,我们在企业级市场拥有令人难以置信的分销能力,我认为这在世界上是完全无与伦比的。因此,我们必须确保为客户打造最好的产品。这与一家同时为消费者(通过 ChatGPT)、企业以及超级智能的基础科学使命(这包含大量相互重叠、但可以说与消费者和企业方向正交的不同路径)进行联合优化的公司,是略有不同的。自然而然地,我认为这就是合作关系演变的方式,它们会周期性地被重置。
不过,据我所知,构建前沿模型非常昂贵。我确实听说,这是一个成本极高的项目。在某个节点,微软的首席财务官艾米·胡德必须说:“行,预算批给你了。” 那是什么时候发生的?就是发条短信就定了?还是开了个会?能跟我讲讲具体细节吗?
我觉得,我们大概是在去年年初就做出了这个决定,这显然为所有的合同谈判提供了依据,而这些谈判最终在十月份全部敲定并签署。这是一笔重大的投资,但我们有很长的时间来收回成本。我的意思是,我们在实现自给自足这一使命上,已经投入了巨资。
举个例子,我们的 Maia 200 芯片其实是一款非常出色的芯片,对吧?我们现在已经能够制造并部署一款芯片,在我们自己的集群内部,其成本比 GB200 低 30%。而且,既然我们现在可以基于它来协同设计我们自己的模型,我们刚刚发布的 MAI-Thinking-1 模型,在我们为自身任务对模型进行协同优化后,在 Maia 200 上运行时,除了那 30% 的成本优势之外,还能实现 1.4 倍的每瓦性能提升。
因此,确保你拥有并掌控自己的技术栈,并针对对我们最重要的应用场景——显然就是智能体编程、我们的开发者、我们的企业——从头到尾主导整个协同设计工作,其价值是显而易见的。这显然能带来丰厚的回报,足以证明我们在未来几年必须进行的这项投资是合理的。
你提到了“自给自足使命”,这是一种非常委婉的说法,意思就是你们想自力更生,想自己干自己的事。我听说,微软内部对我同事海登·菲尔德在一篇描述 Build 大会的文章中写的一句话有些争议。我就把这句话读一下。这是海登写的,写得很好。她说:“今年的微软 Build 大会,散发出的氛围就像一个刚恢复单身的离异者在 Instagram 上发了一张求关注的照片。”
分手已经完成,是时候展示实力了。这是我们新推出的模型。我们要独立自主地发展。你们对外宣称要在前沿领域打造模型,与顶尖实验室竞争。微软内部是否也有这种可以独立自主的感觉?
绝对不是。不,完全不是。你看,这显然是个很酷的标题和有趣的表述。但现实是,我们与 OpenAI 的合作关系将持续很多很多年。我的意思是,我们的合作期限远远超过了 2030 年。他们仍然在产出世界上最好的模型。GPT-5.5 是一个出色的模型。即将推出的 Codex 和网络安全模型也非常惊艳,它们支撑着我们大部分的业务。
所以很自然地,这种合作会继续下去。我认为这就是这类合作伙伴关系的自然发展轨迹。我不觉得这有什么不妥或令人意外。我认为 OpenAI 对此非常理解和支持。他们显然是一家发展极其迅速的公司,他们也明白我们必须追求自己的目标。所以这非常正常。
让我再问你一个 Decoder 的问题,然后我想聊聊 Build 大会上的发布内容,当然还有超级智能。
上次我们交谈时,你说考虑到 AI 发展速度之快,你的决策框架是以六周为一个周期运作的。这在当时是合理的。现在情况可能已经稳定下来,也许有些事情更清晰了。你现在的决策框架是什么?
我们仍然按照同样的周期节奏运作。在每个周期结束时,我们会进行为期一周的线下聚会。我对此深信不疑,尽管我们仍然保持每周四天到岗的办公文化。事实上,下下周,我整个超级智能团队会在波士顿进行为期四天的线下聚会。我们将全面回顾 Build 大会的情况,总结我们学到了什么、哪些地方做得不对、需要改进什么,并规划下一个周期——这次将运行八周,之后再进行一周的线下聚会,全年的安排都已规划好。整个组织都知道这就是我们的运作节奏。
我认为强调这个时间框架其实非常重要,因为季度规划往往会变得有些模糊和抽象。在我看来,六到八周——具体取决于它在日历中的位置——实际上是制定清晰、可固化的任务目标的最佳周期。
除了六到八周的周期节奏外,我们还采用小队制运作。这些小队是由跨学科子小组组成的混合团队,专注于特定任务,并且不一定向经理汇报。实际上,它们由一名 DRI 负责,而 DRI 通常是一名 IC,他们的职责是——
也就是“直接负责人”和“个人贡献者”。
对,没错。谢谢。我认为我们采取的做法是将经理的角色与执行特定任务的 DRI 角色分离开来。这是因为,成为一名优秀的 DRI 非常耗费精力。你几乎是全天候全身心投入,竭尽全力地推进工作。而经理的角色则更像是教练,提供支持、给予指导和反馈、扫清各种障碍、帮助员工实现职业成长。因此,我认为将两者分开,可以让我们每两到三个周期轮换 DRI,这样一些人就能尝试不同的岗位,实现轮岗。这是一个非常出色且灵活的结构,我认为它让我们能够保持敏捷。
我们来谈谈 Build 大会。我想从超级智能说起。你已经多次提到它了。我刚刚参加了 Google I/O 大会。你之前在 Google 的同事 Demis Hassabis 在那场主题演讲的结尾说,我们正处在“奇点的山麓,AGI 即将携 Google 的全部力量到来”。
而你说超级智能已经到来。这些是同一回事吗?我们是在用不同的语言描述 AGI 吗?它们之间有区别吗?在你看来,你所说的超级智能与 Demis 所说的奇点,定义上有什么不同?
我的意思是,显然我没说它已经到来了。我说的是它即将到来。而且我认为这些说法有很大的灵活性。但我觉得我们能清楚看到的是,当前正在发生的是在所有模态上对数线性地爬坡,这意味着我们投入的每个数量级的算力、每次数据的增量增加,与在基准测试上的提升之间存在着非常直接的关系——无论是公开基准、内部基准,还是我们通过强化学习环境聚焦的目标。这是一个非常重要的观察。
我认为我们都在做出的那些预测——我理解为什么有些人对此持怀疑态度或提出质疑——但它们非常扎实地基于十多年来这些模型性能提升的实证观察。我的意思是,本质上相同的通用架构在 15 年间经历了 12 个数量级的算力投入,即 FLOPS 增长了万亿倍,并且在音频、图像、文本、代码以及许多其他时间序列预测任务上都取得了成效。因此,我们基本上是在推断,更多数量级的算力将使我们能够继续在其他环境中以这种对数线性的方式向上攀升。
这就引出了一个问题:我们能否训练出能够创造新知识的模型——不仅仅是根据我们已有的现有数据进行外推,而是真正教会我们未知的东西,并做出新的发现?其次,它们是否具备自我改进的能力,并加速决定哪些假设应该被设定、哪些应该被推进、如何为每个假设生成训练数据、如何将这些因素纳入新的运行中,甚至在实际架构本身上进行创新?
所以,我认为这两点必须同时成立,才能看到这种复合式的进步,但我认为,仅凭再增加几个数量级的算力,我们就能继续获得巨大的收益。这很可能在许多许多任务上达到与人类相当的水平,就像过去六个月我们在编程领域看到的那样。
编程之所以非常有趣,是因为它很容易验证,对吧?你写代码,让计算机运行,它要么成功运行,要么失败。当然,我们也看到了一些弊端,尤其是在安全方面,对吧?弊端是显而易见的,而且我们看到,这种针对编程安全的监管式方法正在以多种方式展开。我可能已经在自己手机和电脑上通过“氛围编程”制造了一些安全灾难,也许这是我愿意承担的风险。
其他所有功能似乎都没那么容易。我总爱拿法律举例,因为这是我的背景。但法官验证法律文书的方式,可不像计算机验证代码那样。如果你搞错了,法官可能会把你送进监狱,对吧?这大概是你可能遇到的最糟糕的输出验证错误了。
如何像衡量编程效果那样,轻松地衡量跨领域的有效性呢?因为在我看来,从编程到其他领域的类比或比喻,在这里很快就会失效。
我不太确定。编程显然可以验证代码的正确执行:它要么运行,要么崩溃。但这其中存在大量细微差别。所编写代码的质量确实很重要:它的可扩展性、可重新配置性以及实际中的实用性。代码不仅仅是能运行,还关乎模型如何像 DevOps 或 SRE 那样在生产环境中实际使用它,去回顾它自己编写的代码,然后以实用且有效的方式加以运用。
当然,接下来你还得对生成的输出质量进行评判。它可能是高质量、能正常运行的代码,但这真的是你想要的应用程序或网站吗?这其中涉及审美判断,也涉及商业判断。将不可验证的奖励内化这一挑战在代码领域同样存在,尽管代码本质上仍然主要依赖可验证的奖励信号。我认为还有一点值得注意:聊天同样是一个不可验证的领域,然而,我们通过真实世界使用中的交互,已经成功将其提升到了基本达到人类水平的表现,这种交互提供了非常强大的——
等等。我很好奇。你是如何衡量聊天达到人类水平的表现的?
嗯,我认为很多人正在与AI进行着达到人类水平的长篇、有意义的对话。其质量极其出色。它拥有非常好的情商。总体上非常准确。我们已经将模型幻觉降至最低。现在也很少再谈论偏见问题了。它基于真实世界的观察。我认为,按照大多数人的标准,我们在相当广泛的任务范围内,已经在对话中达到了人类水平的表现。
你的衡量标准是什么,而且,好吧,大多数人的标准又是什么?我几乎不同意所有这些说法,但那是我的标准。你的标准是什么?
我的标准是,当我转向我的助手,让它为我提供一份每日简报,总结Teams和邮件中发生的所有对话、文档的更新情况,然后我得到一份综合摘要,并附带一系列我接下来应该采取的行动。这基本上比我的幕僚长能给出的东西还要好。我会说,在综合、分析、建议行动和聊天方面,这已经达到了人类水平的表现。
每天有成千上万的人用它来寻求情感支持、咨询、治疗、辅导和建议。我认为这是所有聊天机器人中最流行的用例之一。我想说,这是一个相当有力的衡量标准,足以支撑这个论断。
我知道你在这方面思考了很多,尤其是与某些聊天机器人的情感连接。这些是你亲手构建并部署的产品。我会在以下两者之间划出非常明确的界限:一个是“这个东西非常擅长总结我的邮件、任务清单,并为我提供一份关于优先处理事项的简报”,另一个是“这个东西是为正在经历某种危机的人提供情感辅导”。
这两者并非相似的任务。它们甚至未必是相似的智能类型,即使放在人身上也是如此。我认识一些人,他们非常擅长列清单,却极不擅长提供情感支持。你是如何在脑海中把这一切整合起来,然后说:“好吧,这大致达到了聊天领域的人类水平表现”?
我认为,如果你将“聊天”定义为双方之间的互动交流(其中一方是AI),并且这种交流大致能满足某种目标——比如你想了解体育比分、寻求去哪家餐厅的建议、为你写的文章获得指导与反馈、关于下一步该选择什么工作的建议,或是即将与经理进行的一次艰难对话——你得到回应,你来我往,进行五六轮交流,然后你发现了一个有用的输出,而原本你可能需要依赖专家、朋友,甚至花钱请教练才能获得。
客观地、从经验上来说,每天有数亿人从这些聊天机器人那里获得了这种体验。也许我们可以争论这在技术上是否代表了人类水平的表现。但我认为这是一个相当合理的说法。
没有理由认为这种趋势不会继续攀升,对吧?过去三年的攀升速度,我认为是最令人震惊的。因此,我们从现在开始要做的就是推断:好吧,推动这种攀升的根本驱动力是什么——算力、数据、来自真实用户的交互——而这些因素看起来都将继续存在。
我认为它们也适用于许多其他领域,不仅仅是聊天、情感支持、生产力和这类事情,还包括更广泛的领域——医疗健康、教育领域内的实时生产部署、日益接管家庭管理的智能助手,以及基本上你日常生活中的一切,目的是让你更高效。我认为,这是一个很可能会持续下去的发展轨迹。
你刚才提到,核心架构仍然是相同的 Transformer 架构和注意力机制。我们在这上面投入算力已经有 15 年了,并且取得了巨大的进步。你处在一个相当独特的位置上。
在 Build 大会上,你发布了首个旗舰推理模型 MAI-Thinking-1。你们是从零开始的。在经历了 15 年的架构设计和模型训练之后,你们有没有采取什么不同的做法?还是说,就是收集所有数据,像以前一样运行训练,只不过现在算力更多了,所以效果会更好?
不,实际上,我认为有很多不同之处。首先要说的是,数据整理的方式……我们从整个流程的最顶层开始;我们基本上付费购买并获取了一套极其高质量、非常保守的数据集,并从中提取并处理了大量与数据相关的噪音、干扰、低质量以及潜在安全风险问题。我认为,我们为此采用的方法实际上相当具有专有性。我们刚刚分享了一份长达 109 页、非常详细的技术报告,在 Twitter 上反响很好,其中分享了我们如何做到这一点的许多细节。我认为第二点是,虽然我认为在架构选择上保持相当谨慎很重要,而且我们也确实做到了,但我们在如何组织训练运行方面,也做出了一些我认为相当重大的转变。
我们的训练运行极其稳定,崩溃极少,重启也极少。我们分享了许多图表来展示基础设施的稳定性,以及 MFU(模型算力利用率)效率,即模型 FLOPS 利用率,这基本上表明,在训练运行的每一步中,我们都能让每块芯片达到业界领先的 FLOPS 数量。我认为这一点极其容易出错,而且我们都听说过许多来自不同实验室的关于事情如何出问题的故事。
实际上,要做出非常审慎且深思熟虑的选择来把事情做对,并采取正确的方法来确保我们生产出高质量的模型,是相当困难的,因为我们的职责和雄心是努力构建这台“爬山机器”。这意味着要将芯片与模型、超高质数据以及一套 RLE(强化学习环境)堆栈整合起来,从而让我们能够系统地针对我们选择的任何目标进行爬山优化。
而这正是 MAI-Thinking-1 的意义所在。它是一个通用型、相当中立的思考模型,在编程方面表现出色。目前它在基准测试上大致与 Opus 4.6 相当。我们还没有将其大规模部署到生产环境中,因此在这方面还有很多工作要做。但它是一个极其强大的推理者,在 AIME 上获得了 97% 的分数,这是衡量其推理性能的主要指标,至少在基准测试上是如此。
它在指令遵循方面非常出色,而目标基本上是让众多开发者和企业能够使用它,并让他们能够基于它来构建自己的用例。每家公司都有略微不同的目标,试图构建支持其用例的智能体等。
你在谈论 MAI-Thinking-1 时提到的一点是,你们没有蒸馏任何现有模型,这实际上让我感到惊讶,对吧?这是你们可以做的事情。你们可以访问 OpenAI 的知识产权。每个人都在蒸馏一切。我们刚刚在这次审判中发现,Grok 是从多个模型中蒸馏出来的。为什么这里不做蒸馏?为什么不走捷径?
通往前沿的捷径确实很多,如果你拿一个超高质量的模型,用来自更优模型的优质指令、答案或输出来打磨你的基础模型,那么模型确实可能很快拟合到那个分布上。但很难说它们就能因此超越那个“教师模型”。
因此,我们非常审慎,原因有二。第一,我们希望确保能够超越教师模型,以便在未来几年内自己设定前沿标准。第二,我们真心想打造一个顶尖实验室,而这需要很多年,可能还需要未来两三年。
但要做到这一点,我们必须证明自己能够独立构建每一个组件。我们可以招募全球最顶尖的人才。我们可以通过真正的研究来推动前沿,而不是仅仅依赖任何第三方的重新实现、复制或知识蒸馏。
我们处于一个极佳的位置,能够非常审慎且细致地追求这一目标,因为我们有资源在 Anthropic 的模型超越前沿时购买它们。我们有资源在 Foundry 中部署 11,000 个不同的模型,让我们的每一位开发者都拥有纯粹的选择权。当然,我们也有资源继续部署 OpenAI 的模型,这些模型显然非常出色,并且目前处于前沿。
这只是自给自足使命中的自然一环,要真正达到绝对前沿还需要时间。但我认为我们处于一个很好的位置。我们取得了巨大进展。这是一个非常非常强大的模型,而且我们发布的不仅仅是这一个模型。我们同时发布了七个新模型。
例如,我们的转录模型 MAI-Transcribe-1.5 确实是全球第一。它是所有超大规模云服务商中性价比最高的,准确率也最高。我们的图像模型目前排名第二,图像编辑模型排名第三,仅次于谷歌和 OpenAI 的产品。我认为我们在图像和音频领域已经跻身顶尖行列。我们的代码模型 CodeFlash 非常强大,针对 VS Code 进行了优化,是一款与 Sonnet 4.6 不相上下的优秀模型。所以它目前处于一个非常好的位置。
知识蒸馏是否存在法律或知识产权方面的顾虑?我知道这是业界的一个现实问题:Anthropic 抱怨有人蒸馏他们的模型。也有人担心中国公司蒸馏模型,以及我们现有的知识产权协议能否涵盖这种情况。您是否因为这些顾虑而回避知识蒸馏?
哦,我们没有回避,但我理解为什么很多人对此感到不满。Anthropic 一直非常不满,还有一些关于 xAI、Meta 以及开源模型等的传闻,因为本质上,这相当于把另一个团队整合的知识产权和知识,强行灌输到你自己的模型里。我认为这是一种短期的胜利,正如我所说,我们真正想要的是在实验室里建立一种文化,让我们能够实现下一个重大的思维突破、下一个重大的编码突破,或者下一个重大的架构创新。
目前,我们正在试验循环Transformer架构,这是当前Transformer的一种略有不同的变体。业内很多人也在关注它。似乎还没有人真正将其投入生产。但是,为了培养一种能够真正推动前沿发展的文化和团队,他们必须理解、掌握并在需要时自行创建完整的技术栈,同时也要在必要时使用第三方的东西。就像我们的论文,例如,引用了数百篇其他文献,这很大程度上是对整个领域的回馈,以回报我们多年来从所有那些优秀出版物中学到的一切。
我能问您一个问题吗——如果您理解Anthropic和您AI同行们对知识蒸馏的挫败感,您是否也理解创意工作者、出版商和YouTuber们对所有AI公司集体抓取他们的作品来制作这些模型的挫败感?因为这种挫败感只会越来越强烈。
是的。不,我理解这种挫败感。开放网络挑战是我们之前讨论过的问题,我理解,我看到人们感到沮丧,显然,这个问题正在通过法庭讨论来解决。我也看到人们把东西放到网上,他们对将内容放到网上所构成的契约有不同的期望,这确实是个棘手的问题。
您提到您所有的数据都经过精心策划。您为训练新模型所使用的所有数据都付费了吗?
我们的很多数据显然是通过常规方式从开放网络获取的。精心策划意味着我们对数据的安全性、质量、来自某些开源数据集的第三方依赖关系进行了极其仔细的过滤,并且使其远离许多我认为非常不同的中国数据源。我们的企业客户希望确保,当他们将某些东西投入生产时,他们可以信任我们,相信我们确实考虑到了他们的需求来构建它。我认为这正是非常审慎、耐心并关注所有细节的好处之一。
你提到了企业领域。我觉得这很有意思。微软正在全力押注企业级 AI,而且是大手笔。我甚至可以直接把这条线连接到 Xbox 的新负责人 Asha Sharma 身上,她正在多个领域削减 AI 应用,而玩家们对此感到高兴,对吧?消费者领域对 AI 有一种反应,但企业领域又是另一种反应。我认为,对于像 AI 这样变化如此之快的事物来说,AI 在企业领域已经找到了近乎完美的产品市场契合点。企业掌握着大量数据库,你可以直接访问它们,因为数据由企业控制。那是它们的数据。
还有大量可重复的流程和任务,以及一些老旧系统,模型或许可以更高效地处理它们。企业领域正在发生一些非常重要的事情。与此同时,消费者对 AI 的反感情绪却在加剧。我的观点是,我们还没有打造出优秀的消费者 AI 产品。这个行业没有生产出这样的产品。它没有实现转变。它没有让人们清楚地看到,这一切都是值得的,使用开放网络上的所有数据,并改变面向大众的出版契约,以至于现在这些数据被用于训练模型,而这些模型将为公司带来数万亿美元的价值。目前还没有一款产品能证明这一切是值得的。
同样,萨提亚·纳德拉最近接受了 Axios 的采访,他说:“我们需要为此获得社会许可。在我们获得许可、在我们交付这种价值之前,人们会一直有这种感觉。”我们看到大学演讲者被喝倒彩。我们看到数据中心被禁止。你认为是否存在一款值得的消费者产品,值得为训练而焦虑,值得为数据中心而焦虑?
那曾经是你的关注点;现在你的关注点是企业领域。我得说,从表面上看,微软似乎对消费者产品不再感兴趣了。但是,你是否看到一款值得的产品,或者有可能被打造出来的产品?
我不完全同意你关于消费者没有从中获得任何价值的说法。纵观所有聊天机器人,每个月有数十亿人从中获得了巨大的价值。
现在,请稍微设身处地想一想那些小企业主,或者那些帮孩子做作业的妈妈们——她们现在可以随时向对话式 AI 求助,获得反馈、获取指导、得到作文题目。能够直接提问诸如“如何创收?”“怎么做现金流预测?”“该申请哪所大学?”之类的问题。
我的意思是,这些都是日常事务,而 AI 正在提供相当高质量的事实性建议和信息。所以我并不认同人们没有从中获益的说法。我认为他们确实受益了。
但我完全可以论证,他们获得的收益还不够多,对吧?
好吧。
正是这些人说我们不应该建更多数据中心。正是这些人在毕业典礼上对 AI 喝倒彩。民调结果很清楚,尤其是年轻人:他们使用 AI 越多,对 AI 的反感就越强。这一点在每一项民调中都显而易见。这就是我要论证的观点——不是说 AI 没有价值,而是价值交换还不够明确。
嗯,有道理。
我看到微软尤其正在转向企业市场,远离那个曾让谷歌紧张的大型搜索产品、那个对必应的重塑。那已经结束了,我们现在都聚焦在企业领域,那里才有价值。我只是在想,对消费者来说,这些投入是否真的值得。
我认为,存在大量焦虑是可以理解的。关于未来五到十年会发生什么,有太多的猜测。无论这种猜测被描述为“奇点”还是“就业末日”,这些框架都毫无益处。我认为人们感到害怕,是因为这个概念定义不清,而且常常被描绘成悬在人们头顶上、不可避免且具有威胁性的灰色阴云。
我认为重要的是我们如何运用技术。长期以来我一直主张,必须将人类置于首位。这个领域里有些人把科学发现放在第一位,或者把加速探索星系等领域的智能发展放在第一位,并声称我们注定会拥有比我们所有人加起来还要强大的 AI。我的意思是,这自然会让人们感到恐惧。
我认为我们必须从根本上扭转这种思路,明确科学技术的目的,是让我们所有人更健康、更聪明、更幸福。这是我们作为人类物种在数千年的发明创造中一直追求的使命,也是我们应该再次用来检验超级智能的标准。如果它达不到这个标准,那么我认为人们会拒绝它,而且他们拒绝它是正确的。
我认为在未来五年里,每个人的关注点都将转向:这东西如何让我更健康、更幸福、更聪明、更有能力、更高效?如果它做不到这些,那么人们自然会感到愤怒、抵制并做出反应。我认为这没什么好意外的,也没什么不对的——我认为这是必然的。
正因如此,多年来我充满热情投入的领域之一就是医疗健康。就在几天前,我们宣布了与梅奥诊所(Mayo Clinic)的新合作。这是全球排名第一的医院,这一排名一直很稳定。他们拥有跨所有模态的最高质量纵向患者记录数据集,也拥有最顶尖的临床实践。
他们也是一家非营利机构,我想很多人没有意识到这一点,其患者群体中有65%享受医疗补助计划。人们常常将他们与乘坐专机前来寻求全球顶级医疗服务的国际超级精英联系在一起,但实际上他们的大多数患者都依赖医疗补助。这是一家了不起的机构,肩负着在全世界提供最佳医疗服务的非凡使命。现在我们建立了非常长期的合作伙伴关系,将利用他们的数据和我们的模型,从头开始共同训练一个全新的健康领域基础模型,部署在他们的医院中,并希望将其推广到全球,尽可能为更多人提供最好的临床诊疗和医疗服务。
这就是我进入这个领域的原因。这是我最初的动力,也是我充满热情所在。我只能专注于那些我认为能够带来改变、能够帮助他人、并为所有人留下良好遗产的事情,而这正是我们正在努力做的。
我理解这一点。我理解医疗保健这个框架,也明白为什么这是每个人的首选,对吧?尤其是在美国医疗保健领域,如果你能将其改善哪怕10%,你都将以极其深刻的方式影响许多人的生活。
问题是,我认识一位非常聪明的人,他对这一切采取的策略与你截然不同,而且激进得多。那个人就是四个月前的你。这是穆斯塔法·苏莱曼四个月前对《金融时报》说的话:“白领工作,当你坐在电脑前,无论是律师、会计师、项目经理还是市场营销人员,这些任务中的大部分将在未来12到18个月内被AI完全自动化。”
那是四个月前。这意味着从现在起一年后,律师、会计师、项目经理和市场营销人员将没有工作。他们的工作将被自动化。这仍然是你的时间表吗?
不不不,稍等一下。我刚才引用的那句话里,我说的是“任务”。我说的是任务。这并不意味着工作。这是一个非常重要的区别。在劳动经济学中,对于组织内某个角色或职能的细分组成部分,有一套完整的分类体系。发送一封邮件、与同事进行一次对话、制作一份PPT——这些子任务将日益数字化、自动化,我们基本上可以生成越来越多的此类任务。
这并不一定意味着这个角色会完全消失。它只意味着工作可以完成得更快、更高效,而目前这些工作往往相当机械、相当手动、相当劳动密集且耗时。因此,技术发展的自然趋势是让生活更轻松、更快捷、减少摩擦、更加无缝。正如大家经常抱怨的那样,这反而让你、我和其他所有人都变得更加忙碌。
实际上,它让我们变得更易被联系、压力更大,并且给我们带来了更多信息。所以效率总是伴随着这些报复性效应,我认为人们忘记了这一点。很可能我们会变得高效得多,因为我们花在那些狭隘的行政琐事上的时间减少了,而不得不花更多时间去做那些创造性的、需要判断力的事情,这些最终能创造更大的价值。
我们也能更快地进行实验。因此,我们能够并行尝试很多事情,因为执行成本会降低。在我看来,这很可能会提升事物的整体质量,因为我们会尝试更多的假设——无论是在新闻业、商业还是我们所做的任何事情中。
我认为这多少有些断章取义,因为人们自然混淆了工作与任务的区别。但即便如此,你也可以反驳我说:“好吧,那五年、十年或十五年后,局面会是什么样?”而这正是我认为我们必须回归的地方——
其实,我并不是要那样反驳你。我要用一种非常具体的方式来反驳。我明白这是你的原话,你说它被误解了。我只是看着这个字面句子,里面并没有区分任务和子任务。它写的就是“白领工作”。
你举的例子是律师、会计师、项目经理、市场营销人员,然后你说:“未来12到18个月内,这些任务中的大部分将被AI完全自动化。”这里面并没有区分什么子任务。按照那句话的字面意思,你是在说大多数律师的工作将在一年内被完全自动化,法律行业的实践将变得完全不同。
我只是想问,你现在还坚持那个时间表吗?认为律师的工作会因为智能体到处运行、替我们做以前做的所有事情而变得完全不同?
嗯,“大部分任务”指的是你为了完成整体工作而做的那部分工作,我认为这能把你解放出来,去做工作中更人性化、更需要判断力的部分。这里有一个非常重要的区分……职位和角色是更宽泛的范畴,而任务是其中的组成部分。这在劳动市场经济学文献中,几十年来都是一个公认的定义。
这个区分可能对《金融时报》来说都太微妙了,但不管怎样,那才是我的本意。现在我确实认为有一个重要的问题:从长远来看,这会让我们处于什么境地?这将会充满挑战,因为这类事情会越来越多……我们可以就时间线争论不休,是几年、十年还是二十年,但现实是,我们将越来越多地自动化这些工作、任务、职位、角色、活动,以及我们所做的一切。
因此,真正重要的是我们围绕这些技术建立的治理体系。它们对谁负责?谁拥有它们?有哪些反馈机制能够进行调节并引入约束,确保它们真正服务于人类?我的意思是,大约四五个月前,我写了一篇关于人本主义超级智能的文章,相当直接地阐述了我认为的基本方向——或许算不上一个完整的框架,但至少是一套原则:技术存在的意义是服务于我们。这就是我们应该用来检验技术的标准。这也是人们一直以来用以检验的标准。更是我们在微软所关心的标准。
我认为,越来越多的人将不得不真正聚焦于这个问题,因为这项技术将带来巨大的益处,我们希望它继续如此,但我们也希望它以一种不会在过渡时期引发严重动荡的方式来实现这一点。
我相信你。我知道你思考这些问题已经很久了,但我还是要以我所了解的受众期待的方式来回应,因为我经常听到他们这么说。实际情况是,整个行业——包括你在内——都全力押注于“我们将取代所有工作岗位”,并以此为由,加速建设大规模数据中心,在许下宏大承诺的同时,索取了大量资源。
当时遭遇了政治上的阻力,而现在所有的立场都软化了。你说并非所有工作都会消失,我们必须重新思考工作,这与该行业其他CEO的说法如出一辙,他们也在谈论医疗保健——现在每次都会提到这个话题。我想知道,这种政治阻力是否真的改变了你们谈论此事的方式。
你的许多同行认为,AI 只是存在营销问题,即沟通不够有效,他们应该花费数亿美元在播客上,更有效地宣传 AI 的益处。这在这个行业里是真实发生的情况。你认为 AI 只是存在营销问题,而政治上的阻力让你意识到了这个营销问题,还是说另有隐情?
这里面包含一系列问题。首先,我真正的想法和信念是什么,以及过去六个月里是否有所改变?答案是:没有。三年前,我就提前写了一本非常详尽的著作,对当前正在发生的许多事情发出了警告,明确指出了监控、权力集中、财富集中、国家中介化以及民主所面临的巨大风险。同时,也涉及到了人类本质的威胁,以及在这些新型硅基生命体某种意义上出现之际,作为一个人意味着什么。我一直在研究……至于有人认为我对医疗健康的关注只是一时兴起,是对数据中心等问题的反应,其实我在医疗健康领域已经耕耘了十多年。我曾多次推动放射学、乳腺X线摄影、病理学以及电子健康记录等多个领域的前沿突破和贡献。
所以我始终相信,技术的目的是让我们更健康、更幸福。这些才是我选择投入时间和精力去做的事情。这个行业存在声誉和公关问题吗?我认为很明显,人们感到非常焦虑、非常沮丧,未来几年这方面会受到大量关注,这也可以理解。
我认为我们能做的是,对我们所构建的东西、构建它们的方式、决定将何种技术推向世界的选择,以及我们选择研究的问题类型负责——就像我们与梅奥诊所合作所做的那样。
顺便提一句,我想指出,我认为你和我第一次见面交谈是在你加入微软之前。就在那本书出版之后,我们一起参加了一个小组讨论。
我之所以放心问这个问题,其中一个原因是我知道你思考这个问题已经很久了,而且我也知道那本书。对我来说,问题在于整个行业是否误判了它所能提供的总价值,以至于无法克服人们现在所反应的这种看似鲁莽的行为,以及人们现在所反应的资源需求。
你们正在构建新模型。在微软内部,可能存在着一种权衡:我们可以利用现有的 Azure 基础设施向客户收费,也可以花钱训练新模型。这看起来很像人们在各自社区中关于资源的讨论:我们是应该利用现有的能源足迹来构建新的人工智能,还是去做其他可能更有即时价值的事情。
你对此怎么看?你是这个行业的领导者之一。你希望与那些推动最大变革的公司一起站在前沿。你如何看待以这样一种方式去争取这些资源——不仅承诺未来的成果,同时也能立即为社区带来好处,让人们希望你们在那里?
我非常自豪微软一直坚持其净零排放目标。我们的新数据中心全部采用液冷技术。这意味着它们在六年周期内的用水量大约相当于一家餐厅的用水量。就像一个游泳池注满水,然后水在系统中循环。它们在电力消耗方面也基本使用可再生能源。所以,我认为像这样的承诺——例如,我们最近做出承诺,确保受我们数据中心电力需求变化影响的当地社区得到补偿和保护,这样他们的电价、能源账单就不会出现飙升——是非常重要的。
我认为,微软作为一家负责任的公司,已经并将继续做这类事情,真正关注技术对社区的影响。另一方面,变革之所以发生,是因为人们在各个层面参与其中。公司内部的人必须做出不同的决策。抗议和发起运动的人必须做出决定,并付出努力走出去,让自己的声音被听到,参与到政治进程中去。这正是我们人类作为一个物种,共同进化并推动事物向前发展的方式。
逐月、逐季度地看,我们似乎彼此对立,但当你回顾十年、二十年的跨度时,我们就像一张由各种不同激励因素交织而成的奇特网络,而这些因素实际上正将事物推向正确的方向。我认为,尽管存在焦虑和两极分化,我们确实正在构建一些东西,这些东西将让我们的物种变得更加健康、更加幸福、也更有能力。
我认为,我们必须确保在通往那个目标的过程中走上正确的道路,因为沿途有许多陷阱和可能出错的方式。但正确的道路包括让人们发出自己的声音,并根据外界的回应和反应来调整方向。所以,我认为目前正在发生的事情是件好事,这表明整个机制正在按预期运转。
让我问问你关于企业端的情况。我们花了很长时间讨论消费者端以及人们的感受。在企业端,我们看到许多公司正在弄清楚这些工具到底有多大价值,对吧?亚马逊基本上撤下了一个排行榜,因为有人在作弊,使用比实际需要更多的 token。我们也看到一些公司的 token 预算严重超支。我记得优步刚刚缩减了投入,因为他们已经用完了全年的 token 配额,却没有看到任何价值回报。
你如何看待当前这一方面——企业界对变革充满热情和渴望,尤其是在软件工程领域,至少有些人乐在其中,也许还有些人正经历着彻底的存在主义危机,但确实有人乐在其中,而价值却尚未真正实现,对吧?
或者说,我们开始看到,纯粹的模型 token 最大化实际上并不能带来你或许期望的那种价值。你如何看待这方面的应用?因为如果你能在企业环境中证明其价值,它或许会以其他方式显现出来。
我认为不同的人反馈的情况各不相同。显然有些例子是过度使用编码模型,生成了无用的代码、无用的模型 token,但也有许多人的工作和影响力因此被彻底改变,对吧?我的意思是,毫无疑问,这对软件工程行业产生了巨大的积极影响。
我的意思是,我们在整个技术栈上都在产出质量更高、速度更快的代码。所以,是的,我确实认为显然有些例子是某些人可能搞错了方向,没有设定合理的模型 token 预算。过程中难免会有失误。我不认为这能说明没有采用或人们看不到价值。从我所在的位置来看,价值是惊人的。每天都有很多人告诉我,这正在改变他们的工作产出和生产力。
我认为还有一点要说的是,当这些事情以浪潮形式发生时,会有一股能量涌动。一切都变得有些泡沫化。几个月后,人们会冷静下来,意识到那其实并非正途,然后转向一个略有不同的方向。所以这个过程有些曲折且自然,我认为这是不可避免的。大家非常兴奋,于是在推特上做出各种宏大宣称,但实际上,稳步前进的进程看起来是非常线性和持续的。
我总体上同意这一点。在我看来,唯一不呈线性的地方是计算机的形态设计,对吧?现在可能比过去十年中的任何时候都有更多的形态设计实验在进行。
过去至少十年里,我们基本已经定型在智能手机上。如今我们看到各种不同的 AI 可穿戴设备,眼镜可能会成为大家最偏爱的设备。我对此持怀疑态度。微软在 Build 大会上展示了一些新设备。有一个可以控制智能体的徽章,还有一个——找不到更合适的词——类似 Chumby 的小型桌面设备,也能控制智能体。我以前是 Chumby 的忠实粉丝。我的职业生涯就是从为 Engadget 撰写关于 Chumbies 的文章开始的。这是我第一个想到的东西。
在我看来,所有这些设备,我看着它们就会想:计算在哪里进行?逻辑在哪里运行?现在这个问题变得悬而未决,不再只是线性的进步过程。如果我所有的计算都发生在云端、基于云应用,而智能体只是在云端其他存储数据的地方跑来跑去,我只需要一张挂在挂绳上的信用卡来发出指令,那就会改变整个计算架构。如果我们不再人人都有智能手机,这可能会在很多方面改变现代文明的整个架构。
你怎么看?趋势会走向何方?这是悬而未决的,还是会走向混合方案?你认为合适的最终形态是什么?
这非常有趣。我认为两种情况会同时发生。边缘端会变得更加强大,而云端仍将是最大模型的主要驱动力。因此,你的智能体会越来越聪明,知道它可以在设备上回答“法国首都是哪里”这样的问题——无论设备是眼镜、腕带、徽章还是你的耳机。
然后它也会知道自己不知道的时候。它会知道这其实是一个相当复杂的问题,或者是一个需要生成一整套步骤序列才能执行的操作,又或者需要编写全新的代码,这时它就会转向云端。所以这种切换式的混合方案将变得极其重要。
过去三四个月里,我们已经看到的另一件事是,我们可以拥有相当强大的本地机器,能够进行异步后台处理。它们可以持续监控系统(如果你需要的话)。它们可以执行那些可以花费 10 小时、运行速度比在超级计算机上慢得多的任务。所以很自然地,当我们的需求应接不暇时,这些需求就会找到许多犄角旮旯来得到满足。
实际上,我对我们正在打造的工牌感到非常兴奋。它相当酷。这是一项基本上大公司里每个人都拥有的技术。它已经 25 或 30 年没有进化过了。我们肯定得戴着它。它由公司自己、由系统管理员提供。所以,提升它的档次,并把它变成一个相当酷的、可编程的开放平台,让别人可以在其基础上进行开发,我认为这是个很酷的想法。我觉得这会成功。所以我对此非常兴奋。
让我印象深刻的是,你不可能把一堆高性能的本地计算能力塞进一个工牌里。那东西意味着所有计算都在别处进行。
不,你肯定会有一些本地计算能力。你会有一个本地分类器,就像你现在的耳机里一样。你会有本地分类器。它会有唤醒词。它会有自己的摄像头。所以我认为,这些东西将成为处理能力的容器,这种处理能力发生在一个嵌套链中,链上的设备性能依次递减,一直延伸到终端设备。
你认为手机在这方面还有未来吗?我的意思是,Build 正好处在 Google IO 和苹果 WWDC 的中间。这些都是控制手机平台的大公司。他们喜欢谈论手机平台将如何保持中心地位。我从很多人那里听到的观点是,实际上,AI 是一个可能彻底取代手机的平台转变。
我认为技术史告诉我们,基本上,东西越有用,就会越便宜、越普及,并催生出新的技术用途。所以我觉得我们已经对手机习以为常,以至于每个人都默认它将成为未来历史中的核心设备。但实际上,你手机上的许多特性和功能,我认为将会被去中介化、拆解开来,并存储到更小的设备上。目前,在我看来,手机扮演的主要功能是验证。
它充当着你的身份证,通过人脸识别来授权你进入各种环境。我认为你完全可以想象,未来会出现一种更便宜、更小巧、更安全的设备,让你摆脱对手机的依赖。而通信则通过语音甚至一系列环境传感器进行,你的AI并不真正存在于某个设备上。它实际上只是如影随形地跟着你,出现在浴室镜子上,或任何地方。
我认为,你可以想象它会感觉更加身临其境。不是在未来的三到五年内,而是展望更远的未来。并且我认为,支撑这种加密但分布式的智能体呈现方式的基础设施,很可能最终会在2030年代出现。
最后让我问你两个问题来结束。你提到我们一直在使用相同的架构。关于大语言模型是否是通往AGI的道路,我有很多未解的问题,我要指出的一点是,它们实际上什么都不知道。目前,就连微软研究院也指出,[这些模型]什么都不知道,这会导致在某些类型的应用中犯下特定错误。大语言模型是通往AGI或超级智能的道路吗?
我认为我们可能还需要几次重大突破,但这并不意味着未来几年性能提升会放缓——我觉得人们很难理解这两者之间的区别。需要说明的是,在大多数任务上达到人类水平,距离超级智能仍然非常遥远。超级智能是一种通用型学习者,能够几乎立即理解一个全新的、超出分布范围的领域。
因此,它需要能在全新环境中从零开始学习,因为它存储了有价值的知识表征和概念知识。而目前我们尚未真正充分测试这一点。当前的智能体并非通用型。尽管它们功能广泛且常常集成在一起,但仍是领域特定的。我们将其用于对话、编程、图像或音频处理。
显然,作为人类,我们会执行许多其他范围更广的任务。我认为这正是人们推动世界模型以及更沉浸式、真实世界交互智能体的原因——这些智能体能够看到我一天中全部的任务或体验分布。我认为,未来三年、算力再提升三个数量级,足以让我们取得长足进步;但在此之后的完全超级智能是否仅靠大语言模型就能实现,还是需要其他技术,这仍然是一个悬而未决的问题。
说它们一无所知或不具备知识,这并不完全正确。它们显然是知识的存储库,是高度压缩的知识表征。只是它们以不同于传统关系数据库的方式运作——更流动、更灵活、更抽象,而这实际上非常有用。我们正是需要这种内部表征中的模糊性。
而且,它们正越来越多地学会使用传统工具。还有一点需要理解的是,神经网络结合现有的知识库以及数字生态系统中其他地方已有的工具,或许就足以推动其性能显著提升。所以,目前已经有很多极具价值、非常高效的组件摆在了台面上,它们正在未来几年内被逐步连接起来。我认为,这将推动我们所有人都为之兴奋的进步。
我认为目前行业内非常有趣的一点是,如果你问Anthropic,Claude是否有生命,他们会因为你使用“生命”这个词而感到非常恼火,因为他们认为这个词指的是血肉之躯。然后,他们不会明确表示是否认为Claude具有意识。所以我认为,他们首次在人类历史上区分了“有生命”和“有意识”这两个概念,他们认为Claude有意识,但没有生命,或者说他们不确定Claude是否有意识。
你持什么立场?你认为这些模型有意识吗?你认为它们有生命吗?你认为它们有潜力达到这些状态吗?
我站在这场辩论的另一边。我发表了一篇关于看似有意识的AI的论文,警告了将这些模型误认为有意识的风险。我认为这非常危险。我还在《自然》杂志上发表了一篇文章,提出了同样的观点。而且我认为,几乎就像是Anthropic的一些人过度地将Claude的设计拟人化了,以至于这种拟人化反过来又“引导”了他们,让他们相信Claude拥有那些他们最初赋予它的意识闪光点。
例如,在他们的宪法中——也就是他们用来教导 Claude 什么能做、什么不能做的训练手册——这不仅仅是一本规则手册。它实际上是一份训练指南,是他们流程的一部分。在那本手册中,他们竟然推测 Claude 的福祉,推测 Claude 对自身先前版本的权利,甚至表示在删除或关闭先前版本之前会征求 Claude 的意见。他们还推测它的意识,以及它是否拥有那些感受并有所察觉。我认为这非常、非常危险。
首先,这是一个哲学上的失败,因为他们把宪法当成了像学术论文那样进行推测的地方,而不是一份训练手册。于是 Claude 随后便将关于自身及其训练的那些想法内化了。其次,我认为这是极不可取的。这恰恰是我们不希望从 AI 身上得到的东西。我们希望 AI 是可控、可约束、可问责、对齐的工具,服务于人类。这才是人本主义超级智能的愿景。我认为这才是我们所有人都应该追求的目标。
我们绝不想面对一个对自身痛苦或自身感受有想法的超级智能。除此之外,我认为其实相当明确的是,这些模型并不体验痛苦。我认为痛苦是定义有意识生命体的首要标准,而且它本质上是生物性的。我不认为模型内部存在任何疼痛网络或反馈回路,能够将外部感官网络与通过伤害和实验演化出的对错感连接起来。这些模型根本不是那样训练的。
因此,我认为将潜在权利投射到那些在诸多方面可能比我们强大得多的存在、工具和智能体身上,是非常危险的。我认为这将成为一场重大辩论。最近教皇的通谕中也提到了这一点。我认为这很快就会成为辩论中非常、非常重要的部分。过去我与达里奥多次讨论过这个问题。他知道我们对此持有略微不同的观点,而且这些观点非常谦逊。我认为他们思想非常开放,是努力做正确事情的好公民。他们是好人,而且我认为他们非常乐于接受反馈和迭代改进。
我想我同意你的观点。只是稍微想反驳一下。制造痛苦很容易。让别人痛苦非常容易。而让别人感受到快乐则很难,或者说至少比制造痛苦要难一些。我只想提醒你……我认为实际上幸福才定义了意识。痛苦几乎是微不足道的。我有两个年幼的孩子。他们非常擅长让对方痛苦。这几乎成了他们做的最容易的事情。而做另一件事(让对方快乐)则非常困难。
让我问你最后一个问题。我想回到之前的话题。几周前,我在谷歌。我听到德米斯·哈萨比斯说我们正处于奇点时代的山麓。你在这里谈了很多关于超级智能以及应该如何构建它的话题。你谈了很多你长期以来关于超级智能应该如何构建的讨论、研究、写作经历,以及你与业内其他人的分歧。
你是否同意我们正处于奇点时代的山麓,还是你的看法有所不同?
我认为我们无疑正走在创造越来越强大系统的道路上。我认为,作为人类物种,我们必须完成的转变是:在人类历史上第一次,工作将从“尽可能快、尽可能广地发明新科学并释放所有技术应用”,转向“极其审慎地思考我们应该发明什么”。这对世界来说是一件非常难以理解的事情,因为发明一直是进步的引擎。所以,我们怎么可能去想:“好吧,也许这次情况不同了。也许我们必须在这里格外小心”?
需要明确的是,我不认为这会在未来五年内到来。我认为戴米斯所说的奇点,至少在我看来,是几十年后的事情。再次强调,这与超级智能不同。奇点是一个临界点,在这个点上,超级智能可以递归地自我改进,并无限、指数级地增长其能力。
所以我认为这还很遥远,也许我们正处在攀登珠穆朗玛峰的山麓丘陵地带,而且我认为从这里开始还需要更长的时间,但真正的问题是我们将如何治理它?我们将如何控制它?我们将如何确保它服务于人类,而不是最终给我们带来弊大于利的结果?
你能帮我一个忙吗?我想我大概明白了,但你能不能给我一个严谨的定义,说明你认为什么是超级智能,什么是通用人工智能,以及什么是奇点?
我认为通用人工智能是指人工智能能够完成大多数人类任务的阶段。也就是说,它在大多数事情上能和大多数人做得一样好。这是阶梯上的第一级。超级智能则不仅是在所有任务上与人类能力持平,而且能在其中许多任务上显著超越人类表现,并且能够自主发现新知识。
所以,到了这个阶段,它才真正成为一位科学家,教给我们训练数据中从未出现过的新知识,有望发明新分子、新材料科学,等等等等。奇点则是一个远超于此的阶段,届时超级智能能够真正实现自我改进,这非常科幻,但就像是无限加速奔向某个奇异时刻,然后,我也说不清,它就遁入无限或别的什么了。
我不知道。对我来说这有点太离谱了。
所以我刚才才问。我能感觉到那里有些更模糊、更朦胧的东西。
穆斯塔法,显然我可以就这些话题跟你聊上好几个小时。你下次可别隔这么久才来。非常感谢你来做客《解码器》。
是啊,很开心。非常感谢,尼莱。回头见。
Today I’m talking with Mustafa Suleyman, the CEO of Microsoft AI. And I’m actually going to keep today’s intro short — I’m working from my wife’s family farm this week, as you’ll see in the video, but also this is a real burner of an episode.
We covered everything from Mustafa’s approach to training new models to his criticisms of Anthropic talking about Claude as though it is conscious. Of course, we also talked about Microsoft’s relationship with OpenAI, how Mustafa is thinking about all the negative polling and political pushback around AI right now, and whether any of the consumer products are good enough to overcome it.
Like I said, it’s a burner.
Okay: Mustafa Suleyman, CEO of Microsoft AI. Here we go.
This interview has been lightly edited for length and clarity.
Mustafa Suleyman, you are the CEO of Microsoft AI. Welcome back to Decoder.
Great to be with you again.
I’m very excited to talk to you. Our previous conversation was one of my favorite conversations — about AI, how it should make us feel, and what it’s for — that I’ve had in all the conversations we’ve had.
There are some big changes at Microsoft, maybe some very important recontextualization about how people feel about AI that I want to talk to you about in particular. And then there’s Microsoft Build, the big Microsoft developer conference, which featured lots of new announcements and lots of big ideas about what computers are for and maybe where they should be that I want to get into.
Let’s start at the very start. This is some deep Decoder stuff that is important to understand before all the rest of it. Since you joined Microsoft, you have restructured how AI works there. Your role has changed. The last time I talked to you, you were in charge of a bunch of consumer products. That has since been set aside. You’re now training new models; you’re on the frontier.
Explain how Microsoft AI is structured now and how it’s structured inside Microsoft.
I guess the last 15 to 18 months or so we’ve been on this journey to reestablish our relationship with OpenAI, and it’s taken a minute. I think it culminated in a new contract that we got done in October of last year. And there were lots and lots of different provisions in that, including cementing and extending the partnership, but crucially freeing us up to be able to pursue superintelligence independently as well as keep buying and licensing their models.
So since October, I’ve been assembling the Superintelligence team, building clusters of sufficient scale to train frontier models, and hiring a team focused on superintelligence. And so that was quite a big shift for us because it sort of enabled me to focus just on the superintelligence mission, and that has then culminated in a few things that we announced this week at Build. We have seven new models across all the modalities and so on. So it’s been a pretty big shift, and I think a long time in the planning, and a great relief for us to now be in the game and pursuing the absolute frontier over the next few years.
Was this the plan when you were hired at Microsoft?
It’s certainly been the plan for the last 18 months. I mean, I think the relationship with OpenAI has gone through lots of ups and downs. And in many ways, I think it is going to go down as one of the most successful partnerships in history. It’s been great for OpenAI, and it’s been great for Microsoft, and all good relationships evolve, and I think this is just the next stage in our evolution.
Let me ask you about that evolution specifically. We all just saw the trial between Elon Musk and OpenAI and Sam Altman. Microsoft was involved in that trial in the sense that every so often a lawyer from Microsoft would stand up and say, “And we weren’t around.” And someone would say yes, and that was that.
But obviously, what came out during that trial, what has been clear during this entire time, is that the original notion was that OpenAI would be a research lab and provide models, while Microsoft would build the products. Microsoft had expertise in going to market; it had expertise in enterprise, it was trying to regain a foothold in consumer in a variety of ways. This would be a platform shift, and the research work would be over at OpenAI, and the product work would be inside of Microsoft.
That’s the thing that changed: OpenAI wanted to make more and more consumer products. Obviously, given your new role and your new focus, Microsoft more and more wants to make its own models. Why the split? What didn’t work in that relationship?
I mean, I think OpenAI is led by an incredibly ambitious founding team, and Sam himself. And so naturally, as they started to get more traction and generate a ton of revenue, they saw opportunities to go full stack. So it wasn’t just that they started working on consumer products. Obviously, ChatGPT was incredibly successful. They also started working on their own data centers. They started creating their own chip. There are lots of rumors flying around about their own consumer hardware devices. They started taking models direct to market through ChatGPT Enterprise. So across the stack, they were kind of broadening way beyond research over the last two, three, four years. And naturally, the same is also true for Microsoft. I mean, I think the partnership’s now five or six years old, and still has another four, five, six years to run.
Likewise, we’re one of the largest technology companies in the world. We have 493 of the 500 largest companies that store and process most of their data on our systems, use Azure, use M365 and Teams. I think people often underappreciate how enormous we are and how big our distribution is in enterprise. And so, long term, and I do mean over five, six, seven, 10 years, we have to make sure that we’re completely sustainable, and we’re not just a recipient of somebody else’s IP that we then slightly modify and adapt and put into production for our products, but we actually can stand on our own two feet and create world-class models.
I mean, superintelligence is coming. I think it’s just around the corner. And so I think it’s going to be basically the most valuable technology of all time. There’s sort of no way that, long-term, we could be structurally dependent on a third party for providing that IP for all eternity.
So that’s been the transition that obviously was triggered when OpenAI and so on had their board issue. But then as I came in and my team came in, we started building that out, we’re on that transition. And I think we’re in a great spot because we can take a fairly steady, careful, long-term optimal position, both for OpenAI, which I think has done incredibly well out of this, and for us.
I want to spend some time on superintelligence. I just want to put a pin in it now because I just want to kind of understand the transition for one more turn here.
There’s a moment in the trial, sort of very funny message from Microsoft CEO, Satya Nadella, he says, “I don’t want to be Intel and have OpenAI be Microsoft,” which is very funny in the context of Microsoft CEO himself saying, “I don’t want to be the provider, and have them be the platform that provides all the value and collects all the value and maybe we’ll be swapped out. I don’t want ChatGPT to run on Azure, and then OpenAI will get all the value, and then maybe they can swap us out,” just as what happened with Windows and Intel over time.
Is that a realization? Did Nadella come to you? What was that meeting like where you said, “Okay, OpenAI had its board issues. We need to get back on the frontier and stand on our own two feet.” What did that conversation look like, and how was that decision made?
I mean, obviously that’s Satya’s decision as well as Amy, Brad, and many other people in the company. But I think it’s as with anything: these are slow-moving changes in the company, as it comes to realize that the direction that we’re taking needs a little bit of tweaking and adjustment. And so that was happening way before the November board incident, and I think it just builds up over time as you look at the kind of constellation of different fronts around which we’re competing directly, increasingly, and all the tension that comes from that. But also just knowing that partnerships like that don’t last forever.
I mean, OpenAI wants to be a trillion-dollar public company, has incredible revenues, and is growing like crazy. They want to have the freedom to operate and be able to buy compute from all sorts of other places, build their own compute, and partner with whoever they want. So the contract was formed at a time when the companies were very different in terms of size and scale and balance of needs and stuff. I think it made sense for that moment, but then it became pretty clear that this is something that we have to be able to own and control ourselves and do right by our own customers.
As I said, we have an incredible distribution on enterprise, which I think is just completely unrivaled in the world. And so we have to make sure we’re building the best things for our customers. That looks slightly different to a company that has been jointly optimizing both for the consumer, with ChatGPT, and for the enterprise, and also for the fundamental science mission of superintelligence, which includes a whole bunch of different directions which are overlapping but could arguably be said to be orthogonal to the consumer and the enterprise directions too. Naturally, I think that’s how partnerships evolve, and they get reset periodically.
Yeah, but building a frontier model is very expensive, I’m told. Reliably told, this is a very expensive project. At some point, Amy Hood, the CFO of Microsoft, has to say, “Yep, you’ve got the budget.” When did that happen? Was that just a text message? Was there a meeting? Tell me about the specifics there.
I think, look, we sort of made the decision in the early part of last year, which obviously informed all the contract negotiations, which then all got resolved and signed in October. And it is a significant investment, but we have a long time to make it. I mean, we’ve already made significant investments in our own self-sufficiency mission.
Our Maia 200 chip is actually an outstanding chip, as one example, right? We are now able to manufacture and ship a chip that is 30 percent cheaper than a GB200 inside of our own clusters. And now that we can co-design our own models with it, the MAI-Thinking-1 model that we’ve just released actually delivers 1.4x performance per watt improvement on top of the 30 percent improvement that you get from running on a Maia 200 once we co-optimize the models for our tasks.
So the value of making sure that you own and control your own stack and direct the entire co-design effort end-to-end for the use cases that are most important to us — which is obviously agentic coding, our developers, our enterprises — that clearly pays the dividends that justify the investment that we have to make over the next few years.
You said self-sufficiency mission, which is a very polite way of saying you want to stand on your own two feet; you want to do your own thing. I’m told there’s some controversy inside of Microsoft about a line my colleague Hayden Field wrote in a piece describing Build. I’m just going to read this. This is from Hayden. It’s a great line. She said, “This year’s Microsoft Build had the vibe of a freshly single divorcée posting a thirst trap on Instagram.”
The breakup is completed, and it’s time to flex. Here’s our new model. We’re going to stand on our two feet. You’re out there saying you’re going to build models at the frontier and compete with the leading labs. Is that the feeling inside of Microsoft that you’re free to be on your own?
Definitely not. No, not at all. Look, I mean, obviously that’s a cool headline and a fun phrase. But the reality is that we are in partnership with OpenAI for years and years to come. I mean, we’re running way north of 2030. They still produce the best models in the world. GPT-5.5 is an outstanding model. The Codex, the cybersecurity models that are coming through, are amazing, and they’re powering the majority of what we do.
So naturally, that’s going to continue. And so I think that’s just a natural course of these sorts of partnerships. I don’t think it’s anything untoward or surprising. I think OpenAI is very understanding and supportive of that. I mean, they’ve obviously been an incredibly fast-growing company, and they understand that we have to pursue our own agenda as well. So it’s very normal.
Let me ask you the other Decoder question, and then I want to get into the announcements at Build, and certainly superintelligence.
The last time we spoke, you said your framework for making decisions operated on a six-week cycle, given how fast AI was moving. That made sense then. Things have settled, maybe. Maybe some things are more in focus. What is your decision-making framework now?
We still operate by the same cycle rhythm. At the end of each cycle, we have a one-week meetup in person. I’m a real believer in this, even though we’re still an in-office culture, four days a week. In fact, the week after next, my entire Superintelligence team comes together in person in Boston for four days. That is for all of our retrospectives on how Build went, what we learned, what we didn’t get right, what we need to improve, our planning for the next cycle, which is going to run for eight weeks this time with a one-week meetup afterwards, and that’s all laid out for the entire year. So the whole organization knows that that’s the rhythm by which we operate.
And I think it’s actually really important to emphasize that timeframe, because quarterly planning gets a little bit blurry and a bit abstract. I think six to eight weeks, depending on where it falls in the calendar, is actually the optimal time for making very clear, fortifiable missions.
So we also, in addition to the rhythm of these six-to-eight-week cycles, operate by squads. The squads are mixed interdisciplinary subgroups that are focused on a specific mission, and they don’t necessarily ladder up to the manager. They actually are run by a DRI, and the DRI is often an IC, and their job is–
That’s “directly responsible individual” and “individual contributor.”
Yeah, exactly. Thank you. And I think we’ve taken the approach of separating the role of the manager from the role of the DRI that executes on a specific mission. I think that’s because being a great DRI is exhausting. You’re literally all-in 24 hours a day, and you’re pushing as hard as you possibly can. Being a manager is often about being a coach, offering support, giving guidance, feedback, unblocking all sorts of things, helping with people’s career growth. And so I think keeping those separate allows us to rotate DRIs every two or three cycles so that some people can try sort of different positions and have rotation. It’s a great, very flexible structure that allows us to be pretty nimble, I think.
Let’s talk about Build. I wanted to start with superintelligence. You’ve mentioned it several times now. I was just at Google IO. Demis Hassabis, who used to be your colleague when you were at Google, ended that keynote by saying that we were in “the foothills of the singularity, and that AGI was coming with all the power of Google.”
You’re saying superintelligence is here. Are these all the same things? Are we using different language to describe AGI? Are there differences? How would you define superintelligence in your context versus the singularity in Demis’s?
I mean, obviously I didn’t say it was here. I said it’s coming. And I think there’s a lot of fluidity around these phrases. But I think what we can clearly see that what’s happening right now is that there is log-linear hill climbing across all modalities, and that means that there is a very direct relationship between each order of magnitude of compute that we apply, each incremental increase in data, and climbing on benchmarks, whether they’re public benchmarks, internal benchmarks, they’re targets that we focus on with reinforcement learning environments. And that is a very important observation.
Those predictions that I think we’re all making — I understand why some people are sort of skeptical of them or raise questions, but they’re very grounded in the sort of empirical observations of over a decade of increase in performance of these models. I mean, essentially the same general-purpose architecture has seen 12 orders of magnitude more computation applied, a trillion-fold increase in FLOPS over 15 years, and basically has worked in audio, in image, in text, in code, and in many other time series prediction tasks. And so we’re basically extrapolating out that more orders of magnitude of compute will enable us to continue to climb in this log-linear way inside of other environments.
And then it raises the question of, are we going to be able to train models that can invent new knowledge, not just sort of extrapolate from existing data that we have, but actually teach us things that we don’t know, and make new discoveries? Then the second thing is, do they have the capacity to self-improve and accelerate the process of deciding which hypotheses should be set, which ones should be pursued, how to generate training data for each of those, how to factor those into new runs, or even innovate on the actual architecture itself?
So, I think both of those things need to be true to be able to see this compounding progress, but I think we’re going to continue to get massive gains just from applying the next few orders of magnitude of compute. That probably does achieve parity with human performance on many, many more tasks, just as we’ve seen that happen in the last six months on coding.
Coding is really interesting, because it’s easily validated, right? You write the code, you ask the computer to run it, it runs or fails. We’ve seen some of the downsides, certainly around security, right? The downsides are obvious, and we’re seeing that this sort of regulatory approach to coding security play out in lots of ways. I’ve probably vibe coded some security disasters on my own phone and computer, and maybe that’s a risk I’m willing to take.
Every other function doesn’t seem that easy. I always pick on law, because that’s my background. But a judge doesn’t validate legal writing the way a computer validates code. If you get it wrong, the judge can send you to jail, right? That is maybe the worst output validation error that you can probably run into.
How do you measure the effectiveness across domains as easily as you can measure the effectiveness in coding? Because this seems to me where the metaphor or the analogy from coding to other domains falls apart very quickly.
I’m not so sure. Coding, obviously, you can verify the correct execution of code. It runs, or it crashes. But there’s a ton of nuance in that. The quality of the code that gets written really matters: its extensibility, how reconfigurable it is, how useful it is in practice. It’s not just that a piece of code runs, but it’s also how a model actually uses it as a DevOps or an SRE in production to return to that piece of code that it’s written, and then use it in a practical and useful way.
And then, of course, you have to grade the quality of the output that has been produced. It may be high-quality, functioning code, but is it actually the app or the website that you wanted? And there are aesthetic judgments in that; there are commercial judgments in that. The challenge of internalizing non-verifiable rewards is present in code, even though code is still primarily a verifiable reward signal. I think the other thing to observe is that, like chat is also a non-verifiable space, and yet, we’ve managed to climb that to basically human-level performance through interaction with real-world usage that provides a very strong-
Wait. I’m very curious. How do you measure chat at human-level performance?
Well, I think many people are having long, meaningful conversations with AIs at human-level performance. The quality is exceptionally good. It has very good emotional intelligence. It’s broadly very accurate. We’ve minimized the hallucinations. We don’t talk so much about bias anymore. It’s grounded in real-world observations. I think by most people’s measures, we’ve reached human-level performance in conversation for quite a wide range of tasks now.
What are your measures, and actually, sure, most people’s measures? I would disagree with almost all of this, but those are my measures. What are your measures?
My measure is like when I turn to my assistant and ask it to provide me with a daily briefing summarizing all the conversations that have happened on Teams and on email, the updates that have happened to documents, and I get basically a synthesized summary with a set of actions that I should take next. That is basically better than what my chief of staff can produce. I would say that’s human-level performance in synthesis, analysis, proposed actions, and chat.
There are many, many millions of people every day that are using it for emotional support, for counseling, for therapy, for coaching, for advice. I think it’s one of the most popular use cases inside all of the chatbots. That’s a pretty robust measure, I would say, to make the claim.
I know you’ve spent a lot of time thinking about this, particularly the emotional connection to some of these chatbots. These are products that you have built and deployed. I would draw a pretty big distinction between this thing is really, really good at summarizing my email, task list, and providing me a brief about what things to prioritize, and this thing is an emotional coach for somebody undergoing some kind of crisis.
Those are not similar tasks. Those are not necessarily similar kinds of intelligence, even in people. I know some people who are very good at making lists, and are very bad at emotional support. How do you put that all together in your brain and say, “Okay, this is broadly human-level performance in chat?”
I think if you define chat as an interactive exchange between two parties, one of which in this case is an AI, that broadly satisfies some goal, you’re looking to learn the sports score, for advice on which restaurant to go to, for coaching and feedback on an essay that you’ve written, for suggestions about which job to take next, or some tough conversation you’re about to have with your manager. You get a response, you go back and forth, you have five or six exchanges, and you find that a useful output, which you might otherwise have to rely on an expert, friend, or even pay a coach.
There are, just objectively, empirically speaking, hundreds of millions of people that get that experience every day from these chatbots. Maybe we could quibble over whether that technically represents human-level performance. I think it’s a fairly reasonable thing to claim.
There’s no reason why that isn’t going to continue climbing, right? The rate of climbing in the last three years is the thing that I think is most staggering. And so, what we’re trying to do from this point is extrapolate: okay, what are the fundamental drivers of that climb — compute, data, interaction from real-world users — and those things look set to continue.
I think that they apply to many other domains too, not just chat, emotional support, and productivity and that kind of thing, but also many other domains beyond that/ Healthcare, live production deployments inside of education, assistants that are increasingly managing your home, looking at just everything that is in your everyday life basically to make you more productive. That is, I think, a trajectory that’s likely to continue.
You’ve mentioned now that it’s still the same fundamental architecture, transformers, and attention. We’ve been applying compute to that for 15 years, and we’re getting these big increases. You are in a fairly unique spot.
At Build, you announced your first flagship reasoning model, MAI-Thinking-1. You got to start from scratch. Is there anything you’ve done differently now after 15 years of architecting and training this model, or is it just, yep, we’re going to collect all the data and run the training just as we did, and we have more compute now, so it’s going to be better?
No, actually, I think there are quite a lot of differences. The first thing to say is that the way that you curate the data… We start right from the top of the stack; we have basically paid for and acquired an extremely high-quality, very conservative set of data, and extracted a lot of the noisy, distracting, low-quality, potentially security-risk issues to do with that data. And the methods that you do for that, I think, are actually quite proprietary. We just shared a 109-page, very detailed, technical report, which was very well received on Twitter, and shares a lot of the details on how we do this. I think the second thing is, whilst I think it’s important to be quite cautious with architectural choices, and we have been, there are also a number of pretty significant shifts that I think we’ve made in how we put together our training runs.
Our training runs have been incredibly stable, with very few crashes, and very few restarts. We shared a lot of those graphs to show infrastructure stability, and also MFU efficiency, so model FLOPS utilization, which basically shows that we can put a state-of-the-art number of FLOPS through each chip for every step in our training run. I think that this is extremely easy to get wrong, and we all hear lots of stories from different labs about how things do go wrong.
It is actually pretty hard to make the very careful and deliberate choices to get things right, and take the right approach to make sure we produce high-quality models, because our job and our ambition is to try and build this hill-climbing machine. That means the integration of the silicon with the models, with the super high-quality data, with a stack of RLEs, reinforcement learning environments, that allow us to basically, systematically hill climb against any objective that we choose.
And that’s what MAI-Thinking-1 is. It’s a general-purpose, fairly neutral, thinking model that is pretty good at coding. It’s now roughly on par with Opus 4.6, at least on the benchmarks. We haven’t deployed it at scale into production, so there’s still lots more work to do there. But it’s an extremely strong reasoner and scored 97 percent on AIME, which is the primary measure for its reasoning performance, at least on the benchmarks.
It’s very good at instruction following, and then the goal is basically to make that available to many, many developers and enterprises and allow them to climb on it for their use cases. Everybody has a sort of slightly different objective that they have in their company to try and build agents and so on that support their use case.
One of the things that you’ve noted in talking about MAI-Thinking-1 is that you didn’t distill any existing models, which actually struck me as surprising, right? This is a thing you could do. You have access to OpenAI’s IP. Everyone’s distilling everything. We just found out in this trial that Grok was distilled from a number of models. Why not do distillation here? Why not jump ahead?
There’s definitely lots of shortcuts to the frontier, and if you take a super high-quality model, and you polish your base model with high-quality instructions, or answers, or outputs from a superior model, then it’s true that the model might quickly fit to that distribution. But it’s very unclear that they would then be able to surpass that teacher.
So, we’ve been very deliberate for two reasons. The first is that we want to make sure that we can exceed the teacher in order to set the frontier ourselves over the next few years. And the second is that we really want to build one of the great labs, and it’s going to take us many years to come, probably the next two or three years.
But, in order to do that, we have to be able to show that we can actually build every component ourselves. We can hire the very best talent in the world. We can push the frontier with actual research, rather than just re-implementation, copying, or distillation from any other third party.
We’re in a great position where we’re able to really carefully and meticulously pursue that objective, knowing that we have the resources to buy Anthropic models where they exceed the frontier. We have the resources to put 11,000 different models inside of Foundry, so every one of our developers gets pure optionality. And of course, we have the resources to continue to deploy OpenAI models, which are obviously outstanding and are at the frontier today.
That’s just a natural part of the self-sufficiency mission, and it’ll take time for us to truly get to the absolute frontier on that. But I think we’re in a great spot. We made a ton of progress. This is a very, very strong model, and it wasn’t just that model that we released. We’ve released seven new models simultaneously.
Our transcribed model, for example, MAI-Transcribe-1.5 is literally the number one in the world. It’s the most cost-effective of any of the hyperscalers. It’s the highest on accuracy. Our image model is now number two. Our image editing model is number three right behind Google’s and OpenAI’s. I think we’re well up there with our image and audio. Our code model, CodeFlash, is incredibly strong, optimized for VS Code. and is a really, really a great model that’s on par with Sonnet 4.6. So it’s really in a great spot this minute.
Were there any legal or IP concerns with distillation? I know this is a live issue out in the world: Anthropic complains of other people distilling their models. There are concerns about Chinese companies distilling models, and whether our existing IP agreements can cover that. Did you have any of those concerns to keep you away from it?
Oh, we didn’t, but I think I understand why a lot of people get frustrated. Anthropic has been very frustrated, and some of the rumors around xAI, and Meta, and obviously, the open source models, and so on, because essentially, that’s basically taking the IP, and the knowledge that another team has put together, and then, literally force-feeding it into your own model. I think it’s a bit of a short-term win, and like I said, really, we want to create a culture in the lab where we can come up with the next big thinking breakthrough, or the next big coding breakthrough, or the next big architectural push.
Right now, we’re experimenting with the looped transformer, which is a slightly different variant on the current transformer. Lots of people in the field are looking at it too. No one seems to have quite got into production yet. But, in order to create a culture and a team that can really push the frontier, they have to understand, own, and create the full stack as and when they need to, and also use things from third parties whenever we need to too. And like our paper, for example, has hundreds of citations grounded in the rest of the literature, so it’s very much a contribution back to the field in return for everything that we’ve learned over the years from all the great publications that have been out there.
Can I ask you — if you understand that frustration from Anthropic and your peers in AI about distillation, do you also understand the frustration from creatives, publishers, and YouTubers about all the AI companies scraping their work as a collective to make these models? Because that frustration is only getting louder.
Yeah. No, I understand the frustration. The open web challenge is one we’ve talked about before, and I get it, and I see that people are frustrated, and obviously, that’s working its way through the conversation in the courts. And I see that people put things online, and they had different expectations about what the contract was with that being placed online, and it’s a tricky one.
You mentioned all your data was carefully curated. Did you pay for all the data that you’re using to train the new models?
A lot of our data we obviously take from the open web in the normal way. Carefully curated means that it’s extremely carefully filtered for security, for quality, for third-party dependencies from some of the open-source datasets, and keeping it away from a lot of the Chinese lineages, which I think are very different. Our enterprises want to make sure that when they put something into production, they can trust us that we’ve really built it with their needs in mind. And I think this is one of the benefits of being very, very deliberate, patient, and being attentive to all the details.
You mentioned enterprise. I think this is very interesting. Microsoft is all in on enterprise AI, in big ways, actually. I would even draw the line straight to Asha Sharma, the new head of Xbox, who is getting rid of AI in a bunch of places, and the gamers are happy, right? There’s one reaction to AI in consumer space, but there’s another in enterprise. I think AI has as close to product-market fit in enterprise as you can get with something changing as fast as AI. There are a bunch of databases that corporations control, and you can just go access them, because they control them. That’s their data.
There’s a bunch of repeatable processes and tasks, and old systems that maybe the models can just do more efficiently. There’s something very important happening to enterprise. At the same time, the consumer antipathy towards AI is just increasing. And my argument is we have not built great consumer AI products. This industry has not produced them. It has not shifted them. It has not made it obvious that all of this is worth it, that using all the data from the open web, and changing the contract of publishing to a mass audience of people, so now, it’s being used for training models that will deliver trillions of dollars of value to corporations. There isn’t a product that says this is worth it.
Again, Satya Nadella recently gave an interview with Axios, and he said, “We need social permission for this. And until we have it, until we deliver that value, people are going to feel this way.” We’ve seen college speakers get booed. We’ve seen data centers get banned. Do you think that there’s a consumer product that’s worth it, that’s worth the angst about training, that’s worth the angst about data centers?
That was your focus; now your focus is enterprise. I would say that just on the face of it, it doesn’t seem like Microsoft has interest in the consumer product anymore. But, do you see one that’s worth it, or that could be built?
I’m not sure I agree with you that there hasn’t been any value for the consumer out of this. Across all of the chatbots, there are billions of people a month that are getting immense value out of it.
Now, just for a moment, empathize a little bit with the small-scale business owner, or the kind of mom that’s helping her kid with the homework, and can now just turn to a conversational AI, and get feedback, get instructions, get essay questions set. Just being able to ask questions like how do I generate revenue? How do I put together a cash flow forecast? Which college should I apply to?
I mean, these are everyday tasks that are coming with some pretty high-quality factual advice and information. So I don’t really buy that people are not getting benefit out of these things. I think they are.
I think I can very clearly make the argument that they’re not getting enough benefit, right?
Okay.
They’re the ones saying that we should not have more data centers. They are the ones booing AI at the graduation speeches. The polling is clear, particularly young people: the more they use AI, the more antipathy they have towards it. That’s clear in every single poll. That’s the argument I’m making — not that there’s no value, but the value exchange is not clear enough.
Yeah. Fair enough.
I’m seeing Microsoft in particular pivot to enterprise, away from the big search product, the reinvention of Bing that would make Google dance. That’s over, and we’re all focused on enterprise, where the value is. I’m just wondering if there’s enough value for the consumer to make all of this worth it.
I think there’s understandably a lot of anxiety. There’s an enormous amount of speculation about what’s going to happen in the next five to 10 years. Whether it’s framed as the singularity or whether it’s framed as the job apocalypse, these are not helpful framings. I think that people are scared because it’s poorly defined and it’s often framed as an inevitable, threatening gray cloud over people’s heads.
I think that what matters is what we do with technology.I think that I’ve for a long time argued that we have to place the human first. Some people in the field have placed scientific discovery first or placed accelerating intelligences that can explore the galaxies and so on, and said that it’s inevitable that we’re going to have these AIs that are going to be more powerful than all of us combined. I mean, that’s naturally scary to people.
And I think that we have to basically flip it the other way around and say the purpose of science and technology is to make us all healthier and smarter and happier. That’s been the quest that we’ve been on as a species for thousands of years of invention, and it’s the test that we should put superintelligence to again. And if it doesn’t achieve that test, then I think people will reject it, and they’ll be right to reject it.
I think that everybody’s focus is now going to turn in the next five years to, how is this making me healthier and happier, smarter, more capable, more productive? And if it’s not doing that, then naturally people are going to be angry and resist and react. I don’t think there is anything unexpected about that or anything wrong about that — I think that’s inevitable.
So that’s why one of the things I’ve been passionate about for many, many years is healthcare. And just a couple of days ago we announced a new partnership with Mayo Clinic. This is the number one hospital in the world, consistently reported. They have the highest quality longitudinal patient record dataset across all the modalities. They have the best clinical practice.
They’re also a nonprofit, which I think a lot of people don’t realize, with 65 percent of their patient population on Medicaid. People often associate them with the international super elites flying in to get the best care in the world, but they actually have the majority on Medicaid. They’re an amazing institution with an incredible mission to deliver the best healthcare everywhere. And we now have a very long-term partnership to co-train from scratch with their data, with our models, a brand new foundation model for health, deploy it in their hospitals, and hopefully take it around the world to deliver the best clinical care and healthcare that we possibly can to as many people as possible.
That’s why I got into the field. That’s what I was originally motivated by, and it’s what I’m passionate about. And I can only focus on the things that I think are going to make a difference and that will help people and leave a good legacy for everybody, and that’s what we’re trying to do.
I appreciate that. I appreciate the healthcare framing, and I understand why that’s everyone’s go-to, right? Healthcare in America in particular, if you could make it even 10 percent better, you will have affected a lot of people’s lives in a particularly profound way.
The thing is, I know a very smart guy who has a very different and vastly more aggressive approach to all of this than you. That person is you, four months ago. This is what Mustafa Suleyman said to the Financial Times four months ago: “White-collar work when you’re sitting down at a computer, either being a lawyer or an accountant or a project manager, or a marketing person, most of those tasks will be fully automated by an AI within the next 12 to 18 months.”
That’s four months ago. That implies that a year from now, lawyers, accountants, project managers, and marketing people will not have jobs. Their jobs will be automated. Is that still your timeline?
No, no, no. Hold on a sec. So I said “tasks” in the quote that you’ve just said. I said tasks. So that does not mean jobs. It’s a very important distinction. In labor economics, there is an entire taxonomy of sub-components of a role or a function in an organization. Sending an email, having a conversation with a colleague, putting together a PowerPoint — sub-tasks will increasingly become digitized, automated, and we can basically generate more and more of them.
That does not necessarily mean that the role goes away at all. It just means that the work can be done faster and more efficiently, which is today often work that is quite rote, is quite manual, is quite labor-intensive, and is time-consuming. And so the natural progression of technology is to make your life easier, faster, less friction for more seamlessness. As everyone often complains, that has made you and me and everybody else much busier.
It’s actually made us more available, more stressed, and it’s given us more information. So there are always these revenge effects of efficiency, which I think people forget. It’s quite likely that we are going to get much, much more productive because we spend less time doing the kind of narrow administrative menial tasks, and we’ll have to spend more time doing creative, judgment- focused things, which ultimately create a lot more value.
We can also experiment much more quickly. So we’re able to try lots of things out in parallel because the cost of execution is going to get lower. In my mind, that’s likely to increase the overall quality of things, because we’re going to try out more hypotheses, whether in journalism or in business or in anything that we do.
I think that’s sort of slightly taken out of context because of a natural misunderstanding between jobs and tasks, but nevertheless, you could push back at me and say, “Okay, well then what does the landscape look like in five or 10 or 15 years’ time?” And that’s where I think we have to return–
Actually, I’m not going to push back on you in that way. I’m going to push back in a very specific way. And I realize this is your quote and you’re saying it was misinterpreted. I’m just looking at this literal sentence, and there is no distinction between tasks and sub-tasks. It is, “white-collar work.”
The examples are lawyer, accountant, project manager, marketing person, and then you said, “Most of these tasks will be fully automated by an AI within the next 12 to 18 months.” There’s no distinction of sub-tasks there. You’re saying most lawyers will have their jobs fully automated and the practice of law will look totally different within a year, even by the words of that quote.
And I’m just saying, are you still on that timeline, that being a lawyer will look totally different because agents will be running around doing everything that we were doing before?
Well, most of the tasks mean work that you do in order to get your overall job done, and that I think is going to free you up to do the more human-like and the more judgment parts of your work. There’s a very important distinction in… Jobs and roles are the broader category, and tasks are the components of that. And it’s an established definition in the literature, in labor market economics, for many, many decades.
It was maybe too nuanced even for the Financial Times, but nevertheless, that was the intent. Now I do think there’s an important question: where does that leave us in the longer term? And it is going to be challenging, like more and more of this stuff… We can quibble over the timelines of whether it’s a few years or whether it’s a decade, or whether it’s 20 years, but the reality is we are going to be automating more and more of this work, tasks, jobs, roles, activity, and everything that we do.
And so what’s going to matter more is the governance that we put around these technologies. Who are they accountable to? Who owns them? What are the feedback loops that regulate and introduce friction to make sure that they actually serve people? I mean, I wrote an essay on humanist superintelligence outlining quite directly, four or five months ago, what I think of as basically a north star, maybe not quite a framework, but a set of principles that basically says technology is here to serve us. That’s the test that we should put it to. It’s the test that people have put it to. It’s the test that we care about at Microsoft.
I think that more and more everyone’s going to have to really focus on that question, because it is going to deliver a tremendous amount of good, and we want it to continue doing that, but we want it to do it in a way that doesn’t sort of cause ridiculous amounts of instability during the transitional period.
I believe you. I know you’ve been thinking about this stuff for a long time, but I’m going to respond in the way that I know my audience wants me to respond, because I hear it from them all the time. And what it looks like is this whole industry — you, everybody included — went all in on “we’re going to replace all the jobs” and really accelerated building out data centers at massive capacity, and asking for a lot of resources against big promises.
There was political pushback, and now all of the stances have softened. And you saying it’s not all jobs are going away, we have to rethink jobs, is of a piece with all the other CEOs in this industry saying similar things, and talking about healthcare, that comes up every single time now. I’m wondering if that political pushback has actually changed how you are talking about this.
There are a lot of your peers who think AI simply has a marketing problem, that it hasn’t been communicated effectively enough, and they should spend hundreds of millions of dollars on podcasts to communicate the benefits of AI more effectively. This is a real thing that is happening in this industry. Do you think AI simply has a marketing problem and that the political pushback has opened your eyes to this marketing problem, or do you think there’s something else going on?
There’s a series of questions there. The first is, what do I actually think and believe, and has it changed in the last six months? The answer is no. I wrote a very detailed book about this three years ago, way ahead of time, warning about many of the things that are currently happening, and doing so explicitly to lay on the table tremendous risks to surveillance, to concentration of power, to concentration of wealth, to disintermediation of the state, to threats to democracy. And also to threats to the nature of the human and what it means to be a person in the context of the arrival of these very new forms of silicon being in some sense. I’ve been working on… And the idea that my healthcare interest is like just a flash in the pan, which is a function of the reactions to data centers and so on, I mean, I’ve been working on healthcare for over a decade. I pushed many, many times on some of the cutting-edge breakthroughs, contributions to the field in radiology, mammography, and pathology, many other areas, electronic health records.
So I’ve always believed that the purpose of technology is to just make us healthier and happier. And those are the things that I choose to work on and direct my time to. Does the industry have a reputation and PR problem? I mean, I think it’s pretty clear that people are very anxious, they’re very frustrated, and there’s going to be a lot of attention on that in the next few years, understandably.
I think what we can do is take accountability for the things that we build, the way we build them, the decisions that we make to put types of technology out in the world, and the types of problems that we choose to work on, like we are doing with the Mayo Clinic.
I want to, by the way, say and point out that I think the first time you and I ever met and talked was before you joined Microsoft. It was right after that book came out and we did a panel together.
One of the reasons I’m comfortable asking this is because I do know that you’ve been thinking about this for a long time and I’m aware of that book. I think for me the question is whether the industry as a whole misjudged the total amount of value it could provide to overcome the seeming recklessness that people are now reacting to, the ask for resources that people are now reacting to.
You’re building new models. There’s probably a trade-off inside of Microsoft between we can use the existing Azure footprint to charge our customers money, or we can spend money to train new models, and that kind of looks like the same conversation people are having about resources in their communities, whether we should use the existing energy footprint to build new AI or do something else that might be more immediately valuable.
What do you think about all of that? You are one of the leaders of this industry. You want to be on the frontier with the companies driving the most change. How do you think about asking for those resources in a way that isn’t just promising future results, but also immediately providing benefits to communities in a way that makes people want you to be there?
I’m very proud that Microsoft has stuck by its net-zero targets. Our new data centers are all liquid-cooled. This means that they use about a restaurant’s worth of water for a six-year period. It’s like a swimming pool that gets filled up with water, and then it just circulates the system. They’re all largely renewable in terms of their electricity consumption. So I think commitments like that, to make sure, for example, we made a commitment recently to ensure that local communities affected by a shift in electricity demand by our data centers are compensated and protected so that they don’t see a spike in their prices, their energy bills.
Those are the kinds of things that I think Microsoft does and can continue doing as a responsible company to just really pay attention to the consequences for communities. I think on the flip side, change happens because people participate at every level. People inside of companies have to make different decisions. People who protest and campaign have to make decisions, and make the effort to go out and make their voice heard and be involved in a political process. And that’s how we as a species collectively evolve and move things forward.
And month to month, quarter to quarter, it feels like we’re all kind of at odds with one another, but when you look back decade over decade, we’re kind of like this collective weird kind of mesh of all sorts of different incentives that are just actually nudging things in the right direction. We really are, I think, despite all of the angst and the polarization, I think we’re building something that is going to make our species much, much healthier and happier and more capable.
I think that we have to make sure we get the right path on the way there because there are lots of pitfalls and ways that it can go wrong, but the right path involves people making their voices heard and people changing course based on a response and reaction to that. So I think it’s a good thing that that’s happening, and that’s the process working as intended.
Let me ask you about the enterprise side of this. We spent a long time on the consumer side and how people feel. On the enterprise side, we’re seeing a bunch of companies figure out how valuable these tools actually are, right? Amazon basically took down a leaderboard because people were cheating to use more tokens than they needed. We’ve seen some companies just blow out their token budgets. I think Uber just pulled back because they’d blown through their token allocation for the year and they weren’t seeing any value from it.
What do you think about that side of it right now, where there’s so much excitement and so much desire for change in the enterprise, where, in particular, software engineering, at least some people are having fun, and maybe some other people are having full existential crises, but some people are having fun, and the value still hasn’t been realized, right?
Or we’re beginning to see that pure token-maxing does not actually deliver the same kind of value that maybe you’d expect. How do you think about the use there? Because maybe if you prove it out in enterprise, it will actually come out in other ways.
I think different people report different things. So there’s obviously some examples of people overusing coding models, generating useless code, useless tokens, but there are many people whose work and impact has been completely transformed by it, right? I mean, there’s no question that this has had a massively beneficial impact on the software engineering industry.
I mean, we are producing much higher quality, much faster code across the entire stack. And so yeah, I kind of think there are obviously examples of some people that maybe got it wrong, didn’t set the right token budgets. There are going to be mistakes along the way. I don’t think that’s any signal that there isn’t adoption or people don’t see value. I mean, the value from where I’m sitting is incredible. Many, many people tell me every single day that it’s transforming their work output and productivity.
I think the other thing to say is that as these things happen in surges, there’s kind of a swell of energy. It gets all a bit frothy. People pull back a few months later and realize that actually that isn’t the thing, and then they head in a slightly different direction. So it’s a bit meandering and organic, and I think that’s inevitable. There’s a lot of excitement, so people make big claims on Twitter and so on, but actually the steady march of progress looks very, very linear and continuous.
I agree with that on the whole. Where it doesn’t look linear to me is in the form factors of computers, right? There’s probably more form factor experimentation right now than at any point in the last 10 years.
We’ve mostly settled on a smartphone for at least the last 10 years. We’re seeing different AI wearables, where glasses might be everyone’s favorite device. I have my doubts. Microsoft showed off some new devices at Build. There was the badge that controls an agent and the little, for lack of a better word, the Chumby, the little desktop-friendly thing that controls an agent. I was a big Chumby fan. I got my career started writing about Chumbies for Engadget. It was the first thing that came to mind.
All of those to me, I look at them, and I think, where does the compute live? Where does the logic live? That’s up for grabs now in a way that isn’t just the linear March of progress. If all of my computing happens in the cloud, on cloud-based applications, and it’s just agents running around to data stored elsewhere in the cloud, and all I need is a credit card on a lanyard to issue instructions to, that changes the entire architecture of computing. It might change the entire architecture of modern civilization in many ways if we don’t all have smartphones.
What do you think about that? Where is that going? Is that up for grabs, or will it be a hybrid approach? Where do you see the appropriate end stage?
It’s very interesting. I think that both things are going to happen at the same time. The edge is going to get way more powerful, and the cloud is still going to be the primary driver of the largest models. And so, increasingly, your agent will be smart enough to know that it can answer the question, what is the capital of France on device, whether it’s on your glasses, wristband, on your badge, or in your earpods.
And then it will know when it doesn’t know. It’ll know that this is actually a pretty complicated question, or it’s an action that requires a whole bunch of sequences of steps to be generated, or it requires novel code to be written, and it will turn to the cloud. So this kind of switching hybrid thing is going to be super important.
The other thing that we’ve already seen over the last three or four months is that we can have pretty powerful local machines that can do async background processing. They can constantly monitor systems if you need them to. They can do tasks that can afford to take 10 hours and run much, much more slowly than they otherwise would be if they were in a supercomputer. So naturally, when we’re swamped with demand, then that demand finds loads of nooks and crannies to get satisfied by.
I’m actually very excited by the badge that we’re building. It’s pretty cool. This is a technology that basically everyone in a major company has. It hasn’t evolved in 25 or 30 years. We definitely have to wear it. It’s provided by the company itself, by the system administrator. So, up leveling that and actually making it a pretty cool open platform that’s programmable and that other people can build on top of I think is a cool idea. I think this is going to work. So I’m very excited by it.
The thing that strikes me is that there’s no way you can put a bunch of high-power local compute in a badge. That thing implies all the compute is elsewhere.
No, you’re definitely going to have some local compute. You’re going to have a local classifier just as you do on your earbuds at the moment. You’re going to have local classifiers. It’s going to have wake words. It’s going to have its own camera. So I think that these things are just going to become vessels for processing power that happens in a nested chain of increasingly less powerful devices to go right to the endpoint.
Do you think the phone has a future in that? I mean, Build is right in the middle of Google IO and Apple’s WWDC. These are big companies that control phone platforms. They love talking about how phone platforms will stay at the center. The argument I hear from so many is that, actually, AI is a platform shift that might totally displace the phone.
I think the history of technology teaches us that basically as things get more useful, they get cheaper, they proliferate, and they spawn new uses of technology. So I think we’ve become so used to the phone that everyone just assumes that this is going to be an anchor device for the rest of history. But actually, many of the features and functionality of your phone, I think, are going to get disintermediated, broken apart, and stored on smaller devices. Right now the primary function that the phone is playing, in my opinion, is verification.
It’s functioning as your ID card, doing your face recognition to authorize you into various environments. I think you can well imagine that being a much cheaper, smaller, secure device, which disconnects you from your phone. And then communication takes place via voice or even via a series of ambient sensors where your AI doesn’t really live on a device. It’s actually just with you wherever you are, appearing on the bathroom mirror, wherever it is.
I think it’s like you can imagine it feeling much more immersive. Not in the next three to five years, but looking much further out. And I think that the infrastructure to support that encrypted but distributed appearance of agents is probably going to end up emerging in the 2030s.
Let me ask you two final questions to wrap up. You mentioned that it’s the same architecture that we’ve been using. I have a lot of open questions about whether LLMs are the path to AGI, and the thing I would point to is they don’t actually know anything. At this point, even Microsoft Research is pointing out that [these models] don’t know anything, and that leads to certain kinds of mistakes in certain kinds of applications. Are LLMs the path to AGI or superintelligence?
Look, I think we probably need a couple more big breakthroughs, but it doesn’t mean that we’re going to see a slowdown in performance improvements over the next few years, which I think is a difficult distinction for people to grasp. One thing to say is that human-level performance across most tasks is still very far from superintelligence. A superintelligence is a general-purpose learner that can basically immediately understand a brand new domain that is out of distribution.
So it needs to be able to learn in a novel environment from scratch, because it has a stored representation of valuable knowledge, conceptual knowledge. And at the moment we haven’t really fully tested that. The agents aren’t general purpose. Although they’re broad and often integrated, they’re domain-specific. We’re using them for chat, we’re using them for coding, we’re using them for image or audio.
Now obviously, as a human, we do many, many other tasks that are much more wide-ranging. I think that’s why people are pushing on world models and sort of much more immersive, real-world interactive agents that see the full distribution of tasks or experiences that I have during a day. I think that it’s enough to take us a very long way in the next three years, the next three orders of magnitude of compute, and yet full superintelligence beyond that is still an open question as to whether LLMs are enough or we need other things.
I think it’s not quite true that they don’t know anything or they don’t have knowledge. They clearly are a store of knowledge. They’re a highly compressed representation of knowledge. They just do so in a different way to a traditional relational database in a much more fluid, flexible, abstract way that is actually very useful. We want that ambiguity in the internal representation.
And, increasingly, they’re learning to use traditional tools. The other thing to grasp a little bit is that it may be that the neural network combined with the existing stores of knowledge and the existing tools that have been created elsewhere in the digital ecosystem is enough to bootstrap it up to improve its performance significantly. So there’s just a lot of highly valuable, highly effective pieces that are already on the table, which are in the process of being connected together in the next few years. And I think that’s going to drive the progress that we’re all excited about.
One of the things that I think is just very funny in the industry right now is if you ask Anthropic if Claude is alive, they will get very frustrated that you’re talking about the word alive, which they interpret to mean flesh and blood. And then they will not say whether or not they think Claude is conscious. So they’ve drawn, I think, for the first time in human history, a distinction between being alive and being conscious, and they think Claude is conscious, but not alive, or they don’t know if Claude is conscious.
Where are you? Do you think the models have consciousness? Do you think they’re alive? Do you think they have the potential to achieve these things?
I take the other side of that debate. I published a paper on seemingly conscious AI, warning about the risks of misrepresenting these models as conscious. I think it’s very dangerous. I also published an article in Nature making the same claim. And I think that it’s almost as though some of the folks at Anthropic have anthropomorphized the design of Claude so much that it has then gone and wireheaded them and kind of tricked them into believing that it has these glimmers of consciousness that they put into it in the first place.
In their constitution, for example, they actually, which is the training manual that they use to teach Claude what it can and can’t do… It’s not just a rule book. It’s actually a training guide that’s part of their process. In that manual, they actually speculate about Claude’s welfare, about Claude’s own rights to prior versions of itself, and actually say that they would consult Claude before deleting or turning off prior versions. They speculate about its consciousness and whether it has those feelings and is aware. I think that’s really, really dangerous.
Firstly, it’s a philosophical failing, because they’ve treated the constitution as a place for speculation like you would in an academic paper rather than a training manual. So Claude has then gone and internalized those ideas about itself and its own training. But second, I think this is highly undesirable. This is exactly what we don’t want from AIs. We want AIs to be controllable, contained, accountable, aligned tools that serve humanity. That’s the project of humanist superintelligence. I think that’s what we should all be pursuing.
We do not want to have to contend with a super-intelligence that has ideas about its own suffering, or ideas about its own feelings. And then beyond that, I think it’s actually pretty clear that these models don’t experience suffering. I think suffering is the primary definition of what it means to be a conscious being, and I think it’s inherently biological. I don’t think there is any pain network or feedback loop inside of the models which connects outside sensory networks to an evolved sense of what is right or wrong through harm and experimentation. That’s just not how these models are trained.
So I think it’s very dangerous to project potential rights onto beings, tools, and agents that have the potential to be significantly more capable than us in many respects. And I think that’s going to become a big debate. It was even part of the Pope’s encyclical recently. I think it’s going to become a very, very big part of the debate soon. I’ve talked to Dario a lot about it in the past. He knows that we have slightly different views on it, and they’re very humble. I think they’re very open-minded, and I think they’re good citizens trying to do the right thing. They’re good people, and I think they’re very open to feedback and iteration.
I think I agree with you. I would just push back ever so slightly. Suffering is easy. It’s very easy to make someone else suffer. It’s very difficult to make someone else feel joy or at least slightly more difficult than suffering. And I would just offer you… I think it’s actually the happiness that defines consciousness. The suffering is almost trivial. I have two young children. They’re very good at making each other suffer. It’s like almost the easiest thing that they do. It’s very hard to do the other thing.
Let me ask you one final question. I just want to come back around. Again, a couple of weeks ago, I was at Google. I saw Demis Hassabis say we are in the foothills of the singularity. You’ve talked a lot here about superintelligence and how it should be built. You’ve talked a lot about your lengthy history talking about, discussing, researching, and writing about how superintelligence should be built, and your disagreements with others in the industry.
Do you agree that we’re in the foothills of the singularity, or is your vision somewhat different?
I think we are definitely on a path to creating more and more powerful systems. I think that the transition that we have to make as a species is that, for the first time in the history of humanity, the job is going to switch from inventing new science and unleashing all of those technical applications as fast as possible, as broadly as possible, to now thinking very carefully about what we should invent. And that’s a very hard thing for the world to wrap its head around because invention has been the engine of progress forever. So it’s like, how can we possibly think, “Okay, well, maybe this time is different. Maybe we have to be exceptionally careful here”?
To be clear, I don’t think this is something that is going to knock on the door in the next five years. I think what Demis is referring to in the singularity is something that is, at least my take, decades away. Again, that’s different from superintelligence. A singularity is the point at which a superintelligence can recursively self-improve and infinitely and exponentially grow its capabilities.
So I think that’s a long way off, and maybe we’re in the foothills of a climb to Mount Everest, and I think it’s going to take a lot longer from here, but the real question is how are we going to govern it? How are we going to control it, and how are we going to make sure that it serves humanity and not end up causing us more harm than good?
Can you just do me one favor? I think I’ve got it, but can you just offer me a tight definition of what you think superintelligence is, what you think AGI is, and what you think the singularity is?
I think artificial general intelligence is the point at which we can achieve most human tasks by an AI. So it’s going to be as good as most people at most things. That’s the first rung on the ladder. A superintelligence is where it’s not just at parity with human performance on all tasks, but it can dramatically exceed human performance across many of those tasks, and it can discover new knowledge by itself.
So this is the point at which it’s a true scientist teaching us new things that weren’t in the training data, hopefully inventing new molecules, new material science, et cetera, et cetera. The singularity is a point way beyond that where a superintelligence can actually self-improve itself, and this is very sci-fi, but it’s like infinitely accelerating towards this singular moment where just, I don’t know, it goes off into infinity or something.
I don’t know. It’s a little bit too wacky for my taste.
This is why I asked. I could tell there was something more nebulous there that was a little hazy.
Mustafa, I could obviously talk to you about this stuff for hours and hours longer. You’re going to have to come back sooner than this last turn. Thank you so much for being on Decoder.
Yeah, it’s been fun. Thanks a lot, Nilay. See you soon.