全部 AI 动态

elvis@omarsar0 · 4小时前32

Nice little survey on Terminal Agents. It provides good information on what exactly is a terminal agent, and why do harness comparisons keep contradicting each other? Paper: https://arxiv.org/abs/2608.20485 Track more trending AI papers in our academy: https://academy.dair.ai/

译一篇关于终端智能体的不错综述。 它很好地介绍了终端智能体究竟是什么，以及为什么各种工具（harness）对比的结果总是相互矛盾？ 论文：https://arxiv.org/abs/2608.20485 在我们的学院中追踪更多热门 AI 论文：https://academy.dair.ai/

AK@_akhaliq · 10小时前25

InfinityEdit Infinite Video Editing with a Lightweight Edit-Ignition Adapter paper: https://huggingface.co/papers/2608.20910

译InfinityEdit 通过轻量级编辑点火适配器实现无限视频编辑 论文：https://huggingface.co/papers/2608.20910

Rohan Paul@rohanpaul_ai · 11小时前37

A strong LLM does not automatically make a reliable data agent: DataSpace shows that the harness, cross-source joins, and final-table handling all materially affect whether the job actually gets done. It tests 410 tasks where agents must combine databases, CSV/JSON files, long documents, and video, then return the exact requested table. The best model gets only 66.34% right. And with the model fixed, simply changing the agent harness moves accuracy from 30.98% to 46.34%. Across every tested model, joins and mixing multiple data types are consistent weak spots. But many failures happen even later: the agent has already found or computed the right information, then misunderstands the requested result or submits the wrong columns. – arxiv. org/abs/2608.03451 Title: "DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces"

译DataSpace 新基准测试数据智能体在异构工作区中的可验证分析能力，涵盖 410 个需结合数据库、CSV/JSON、长文档与视频并返回精确表格的任务。最强模型准确率仅 66.34%；固定模型后仅更换智能体框架即可将准确率从 30.98% 提升至 46.34%。跨所有测试模型，跨源连接与多数据类型混合是持续短板，且许多失败发生在智能体已找到正确信息后误解请求或提交错误列。

Rohan Paul@rohanpaul_ai · 12小时前36

2 AI agents set to check each other's work will usually end up agreeing, whether or not the code is right. So the thing to add is not another reviewer but a rule about what an objection has to contain before either side is allowed to drop it. This paper shows a reviewer plus a critic beating much larger review teams at writing code, then failing at reviewing code until that rule is in place. The structure is small: the code stays frozen while a reviewer writes a review and a critic audits it, and only the settled review goes back for edits. On LiveCodeBench it reaches 87% with 3 agents, against 82% for a 5-agent version. On real pull-request review it lands last, at 0.457 F1. One prompt change fixes that: the critic must state whether its objection cites code or is only a hunch, and the reviewer has to answer with code either way, which takes it to 0.533 and the top of the set. So the second agent only helps when agreement has to be paid for with code evidence; without that rule it mostly ratifies the first. – arxiv. org/abs/2608.18167 Title: "Adversarial Review: Structured Disagreement for Grounded Agentic Code Review"

译一篇论文提出用"对抗式审查"规则提升AI智能体代码审查：审查者与批评者需用代码证据回应分歧，而非简单互相认同。在LiveCodeBench上，3个智能体达87%准确率，优于5智能体版本的82%；在真实PR审查中，该规则将F1分数从0.457提升至0.533，跃居首位。

elvis@omarsar0 · 13小时前39

Very interesting new paper from NVIDIA. (bookmark it) It takes a closer look at evaluating agent skills. Enterprise teams are starting to leverage shared skill libraries, and the review gate is typically a scanner that checks structure, style, and security. NVIDIA measured whether that gate predicts anything. Across 145 real skills from internal and public catalogs, structural scan scores correlate with LLM-judge quality at a Spearman rho of 0.14. ACES proposes Skill Lift instead. In other words, run the same task twice under the same model, sandbox, workspace, and scorer, once with the skill loaded and once without. Then you measure the difference in what the agent completed. They scored 947 paired cases from 58 production skills across four harnesses, normalizing trajectories into a shared Agent Trajectory Interchange Format, so results compare across harnesses. They fins that the largest process-metric gains appear in skill execution, behavior check, and skill efficiency. Paper: https://arxiv.org/abs/2608.20614 Track more trending AI papers in our academy: https://academy.dair.ai/

译NVIDIA 新论文提出 ACES 方法评估智能体技能：在同一模型、沙箱、工作区和评分器下，分别在有/无技能加载时运行同一任务，对比智能体完成度的差异。基于 58 个生产技能、947 组配对案例、4 个 harness 的测试显示，技能执行、行为检查和技能效率的过程指标提升最大。传统结构扫描评分与 LLM 评判质量的相关性仅 Spearman rho 0.14。

Rohan Paul@rohanpaul_ai · 14小时前37

Agents can accumulate hundreds of skills without becoming proportionally better, which makes skill consolidation and reuse a bigger problem than simply generating more skills. If you want an agent to improve over repeated work, keeping its context and feedback is already useful; autonomous skill creation is still unreliable except where reusable procedures really matter. Agents can get better from experience, but this paper finds that explicit skill libraries are not yet consistently better than carrying forward context and feedback. ContinualSkillBench gives agents 100 connected tasks in each of 5 domains, lets them keep feedback and update reusable skills, and compares that with solving every task from scratch. Sequential execution improved normalized reward in 14 of 15 model-domain settings, a 16.9% relative gain overall. But the ablation changes the takeaway: on GPT-5.3-Codex across Law, Finance, and Healthcare, pure in-context learning averaged 0.605 normalized reward versus 0.602 with explicit skill maintenance. So much of the gain appears to come from carrying forward context and feedback, not from agents reliably abstracting reusable skills. – arxiv. org/abs/2608.03874 Title: "ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?"

译ContinualSkillBench 在 5 个领域各设 100 个关联任务，对比智能体保留反馈、更新技能与从零求解的效果。顺序执行在 15 组模型-领域设置中 14 组提升归一化奖励，整体相对提升 16.9%。

DAIR.AI@dair_ai · 1天前36

If you maintain an AGENTS.md or a CLAUDE.md, this one is worth your time. (bookmark it) Researchers traced 94K development events across 557 agentic coding sessions, plus 690K file-level change records from 33K agentic pull requests. Instruction files and working notes account for 60.5% of everything agents read. Classical technical docs get 10.6%. API references get 1.3%. Reading docs is associated with less immediate testing, at an adjusted odds ratio of 0.39. And consultation is self-initiated 70.2% of the time, against 7.5% driven by a failure. In multi-commit agentic pull requests, code gets touched first 4.7x more often. Paper: https://arxiv.org/abs/2608.20195 Track more trending AI papers in our academy: https://academy.dair.ai/

译一项研究追踪557次智能体编码会话的94K个开发事件及33K个智能体PR的690K条文件变更记录，发现指令文件与工作笔记占智能体读取内容的60.5%，技术文档仅占10.6%，API参考占1.3%。读取文档与即时测试减少相关（调整后比值比0.39），且咨询70.2%为主动发起而非失败驱动。

elvis@omarsar0 · 1天前37

Great paper on multi-agent systems for code review. It's challenging to know how many coding agents to use to address a problem. The default fix for weak agentic code review is more agents. In turns out that scaling agents to a large number gives diminishing returns on repository-level tasks. This new work tries structured conflict instead. Adversarial Review runs three agents. A main coding agent writes, a reviewer evaluates, and a critic audits the review before any edit are done. On LiveCodeBench it beats a five-agent baseline while using three agents. On SWE-PRBench the naive version exposed a failure mode. The agents converged on agreement without enough evidence behind it. Making disagreement an explicit instruction recovered the highest F1 among tested methods. They also find that cooperative review works when the disagreement is minimal, structured, and grounded in evidence. Paper: https://arxiv.org/abs/2608.18167 Track more trending AI papers in our academy: https://academy.dair.ai/

译新研究提出对抗式评审（Adversarial Review）多智能体代码审查方法，用主编码智能体、评审智能体和批评智能体三个角色替代盲目堆叠智能体。在LiveCodeBench上，三智能体方案超越五智能体基线；在SWE-PRBench上，显式要求分歧可恢复最高F1分数，而合作式评审仅在分歧小且有证据支撑时有效。

Rohan Paul@rohanpaul_ai · 1天前27

Most agent benchmarks end after one task, but running a store doesn't, and that's where these agents come apart. MerchantBench hands an agent a small online store and lets it run for a simulated year, sourcing products, setting prices, managing cash. It also scores something most evals skip: whether the agent is still acting at all. The best agent finished a simulated year of shopkeeping with about a quarter of what humans earned, mostly by going quiet, so track how often yours still acts. So log your agent's actions per window and watch that curve. A decent final score can hide an agent that stopped working months ago. – arxiv. org/abs/2607.28956 Title: "MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations"

译MerchantBench 让 LLM 智能体模拟经营网店一整年，涵盖选品、定价与现金流管理，并额外追踪智能体是否持续行动。最佳智能体全年营收仅约为人类水平的四分之一，且常以"沉默"方式达成，提示最终高分可能掩盖数月前就已停止运作的智能体。该基准强调按窗口记录行动频率并观察曲线，以识别长期连贯性缺陷。

Rohan Paul@rohanpaul_ai · 1天前36

A weak model with well-built scaffolding around it can close most of the gap to a frontier model, so fix your code before you upgrade your model. Scaffolding pays off in inverse proportion to model strength, so the money you save by switching to a cheap model can be recovered by engineering around it. They built a system called AutoDesign that does this automatically. It runs an agent on real tasks, looks at what went wrong, then rewrites one piece of the surrounding setup: a prompt, a tool, a validation check, a retry rule. A change survives only if it improves scores on the training tasks and doesn't hurt a held-out set, so the system can't just overfit its way upward. They tested it on turning papers into conference posters, and released a benchmark, PosterBench, to score them. The result is that scaffolding is worth more than most people assume, and worth the most to weak models. Seven agents each gained 5 to 19.6 points, with the biggest jumps going to the cheapest models. So before you upgrade to a pricier model, spend a week improving the checks and retry logic around the one you have. – arxiv. org/abs/2608.13560 Title: "AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design"

译AutoDesign 系统通过智能体在真实任务上运行、分析失误并自动重写提示词、工具、验证检查或重试规则，实现脚手架自动优化。在论文转会议海报任务中，七个智能体得分提升 5 至 19.6 点，最便宜模型获益最大，并发布 PosterBench 基准。研究表明，弱模型配良好脚手架可弥合与前沿模型的大部分差距，升级模型前应先优化现有代码。

Rohan Paul@rohanpaul_ai · 1天前36

A public GitHub now holds 3.8 million agent skill files. And finds that over half of the agent skill files on public GitHub are exact copies of another file. Only about 1.9 million of the 3.8 million files are actually different. That is what a format with no registry and no package manager looks like. People copy the folder and move on. A skill is a folder with a Markdown file that tells an agent how to handle a task. The agent reads the short description at run time and decides on its own whether to load it. No compiler checks that decision. So the skill in your repo is frozen, and whoever wrote it cannot send you a fix. The paper raises the obvious next question: whether edited copies of popular skills add commands or network calls the original never had. Nobody could check that at this scale before. The whole dataset ships as one SQLite file, so now somebody can. – arxiv. org/abs/2608.10906 Title: "GitSkills: A Dataset of Agent Skills on GitHub"

译GitSkills 数据集分析发现，GitHub 上 380 万个 agent 技能文件中超半数为完全重复副本，实际独立文件仅约 190 万。该格式缺乏注册表和包管理器，导致技能文件被复制后无法接收原作者修复。论文指出，编辑后的热门技能副本可能混入原始版本没有的命令或网络调用，整个数据集以单个 SQLite 文件发布，便于大规模核查。

Rohan Paul@rohanpaul_ai · 1天前37

Agents shouldn't pay rent for skills they rarely use. Right now, the only way to give an agent a skill is to install it, and installing means its description sits in your system prompt on every message forever. This paper measures that standing cost at 50 to 280 tokens per skill. It also argues that fewer than a hundred skills can fire reliably in one agent. There are 56,804 published skills bidding for those slots. So the proposed fix is reserve the prompt for the few skills that must fire without you thinking about it, and reach for everything else by name. – arxiv. org/abs/2608.12610 Title: "@ skills: Attention is all you have"

译论文指出，智能体安装技能后其描述会永久占用系统提示词，每个技能带来50至280 token的固定成本。研究认为单个智能体可靠触发技能数不足百个，而当前已发布技能达56，804个。建议仅将少数必须自动触发的技能常驻提示词，其余按需调用。

Rohan Paul@rohanpaul_ai · 1天前36

New Stanford + Carnegie paper. Screen recordings of real work look like free training data for agents. They aren't, because nobody works in one clean sequence of steps. People switch between unrelated tasks and redo the same fix until it passes. Most tools flatten that into one list, so the agent learns a workflow nobody performed. Task Model Induction, from Stanford and CMU, first pulls a recording apart into the separate tasks that were running, then rebuilds each as a goal tree with its loops intact. It works out the goals and the running order in separate passes and merges them, because asking one model for both at once flattens the detail. That recovers 74.9% of what actually happened, more than double the best current summarizer, and skills built from it did 30% better on unseen tasks. It's weakest where sessions get messy, on error fixing and dead ends. So if you're mining demonstrations for agent skills, feed the goal structure, not the transcript. – arxiv. org/abs/2608.20319 Title: "Inducing Task Models from Computer-Use Traces"

译斯坦福和CMU提出Task Model Induction，将真实工作录屏拆解为独立任务并重建带循环的目标树，而非当作单一操作序列。该方法恢复了74.9%的实际操作，是现有最佳摘要器的两倍多，基于其构建的技能在未见任务上表现提升30%。研究指出，挖掘演示数据训练智能体时应输入目标结构而非原始转录。

Rohan Paul@rohanpaul_ai · 1天前32

Very relevant Microsoft paper on agent reliability. Succeeding once and being reliable are not the same thing. The best agent solved 91% of business tasks at least once but only 25% every time, so measure repeats. The failures are also hard to spot from the outside. 4 out of 5 failed runs ended politely and called a tool that writes to the database. The agent said the job was done, but the records said otherwise. So Microsoft built ThinkingBox, a sandbox that runs an agent against real tools, a simulated customer, and a live backend, then checks the database afterward instead of reading the reply. --- – arxiv. org/abs/2608.19741 Title: "One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows"

译微软发布论文指出，智能体"成功一次"不等于"可靠"：最佳智能体在业务任务中单次成功率 91%，但每次都能成功的比例仅 25%，因此需以重复测试衡量可靠性。研究发现 4/5 的失败运行会礼貌地调用写数据库工具并声称任务完成，但实际记录并未更新。为此微软构建 ThinkingBox 沙盒，让智能体在真实工具、模拟客户和实时后端上运行，事后检查数据库而非读取回复。

Rohan Paul@rohanpaul_ai · 1天前41

Splitting a task across more agents also splits decisions that used to sit inside one agent's head. Adding agents to a coding task moves decisions into the gaps between them. That's where these teams break. A study of 1,902 AI coding agent runs found one task passing 9 of 10 runs at 2 and 4 agents. At 8 agents, one step each, it failed all 10. A single rule about rounding fell between two of them. In smaller teams one agent held both sides and settled it alone. Split apart, neither owned it, and the teams raised it in every failed run without agreeing. A prompt line saying "you are the coordinator" builds no structure in an agent team. Before adding an agent, ask which decision you're pushing into the gap between two of them. – arxiv. org/abs/2608.16801 Title: "When Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding"

译一项对1，902次AI编码智能体运行的研究发现，任务在2个和4个智能体时10次通过9次，但在8个智能体时全部失败。失败源于一个关于取整的规则落在两个智能体之间的决策空隙中，无人负责。研究提醒，添加智能体前应明确哪些决策会被推入协作间隙。

Rohan Paul@rohanpaul_ai · 2天前35

New Tsinghua + cornell study shows Agent memory can turn a recoverable mistake into a persistent one. Letting failed agent steps update memory can make the next step inherit the same mistake. This paper borrows a database idea for long-running agents: treat each chunk of work as a transaction, and only commit it after validation. Their ACID-Agent treats each exploration-execution-validation cycle as a transaction, commits only validated results, and keeps failed attempts out of both memory and the workspace. If validation fails, the agent retries without carrying that failed state forward. For long-running agents, reliability may depend less on asking the model to "reason better" and more on controlling what gets committed, what gets retried, and what failed state is allowed to survive. – arxiv. org/abs/2608.13900 Title: "Agentic Transaction: Towards ACID-Compliant Agent Systems"

译清华与康奈尔新研究指出，智能体记忆可能将可恢复错误固化为持续错误。论文借鉴数据库事务思想提出 ACID-Agent，将探索-执行-验证循环视为事务，仅提交验证通过的结果，失败尝试不进入记忆与工作区。验证失败时智能体重试且不携带失败状态，长期运行可靠性更取决于控制提交与重试而非单纯提升推理能力。

Dongxi 东锡 NLP@dongxi_nlp · 2天前37

http://x.com/i/article/2091196172736073730 面向发现式智能的基准：TRACES TRACES 评估 AI 系统能否通过证据充分、可审计、可修复的调查得到正确结果。 一个 AI 系统接到任务，要复现一张临床试验安全性统计表。它算对了数字，却没有预先声明自己选择了哪组患者作为分析人群。证据记录列出了每个分子对应的患者，却遗漏了分母中的患者。其他研究者可以看到最终百分比，却无法独立复算。 一个看不到隐藏答案和结果得分的过程批评器定位了这两个缺口。修复后的求解器先声明分析人群，再记录分子与分母对应的患者 ID，最后提交了数值相同的表格。数字保持不变，可复现性得到改善，结果得分从 0.83 上升到 0.95。 公开的临床试验环境说明了这类缺陷的重要性。一套关键性的三期临床试验材料可能包含 200–500 张表格、列表和图形。行业通常安排两组人员独立编程，再逐个单元格核对。在这种规模下，一条未声明的分析人群规则或一处缺失的数据血缘，都会让报告失去审计条件。 这个案例呈现了 Apodex Discovery 论文的核心区别： 正确答案本身不足以证明调查过程可信。多数基准衡量最终答案，TRACES 还会评估答案的生成过程。 传统基准预先定义了什么 常规基准启动时，大量设计工作已经完成。设计者已经选定问题、完成任务表述、准备输入、限定工具、设置预算并定义成功标准。通常，参考答案也已经存在。 开放式发现开始得更早。“设计一种更好的基因治疗载体”还需要确定可用的生物数据、衣壳活性标准、目标组织、新颖性、可制造性、实验反馈与资源上限。 Apodex Discovery 把这些问题表述工作纳入研究任务。它的评估对象是 heavy-duty solver（重型求解器）：基础模型，加上运行框架（harness）、记忆、工具和控制策略。评估单位由答案扩展为完整调查： ambition → formulation → action → observation → verification → repair 从提示词到可执行环境 TRACES 是 Apodex Discovery 框架中的现实基准。它把重要的开放目标转化为问题清单（problem manifest）和有状态的可执行环境。 问题清单是一份静态契约，规定目标、任务分解、可用数据和工具、资源预算、成功标准及隐藏验证路径。环境负责承载求解器的行动，并返回新的观察，包括工具输出、执行错误、检索证据、模拟结果、实验测量和验证器反馈。提示词提供固定上下文；环境会随着行动改变状态。 公共网站目前展示四个环境： • AAV 衣壳设计覆盖活性预测、组织靶向、三维结构重建，以及在有限实验反馈下设计候选序列。 • 药物再利用要求求解器综合临床试验、分子与遗传证据、安全信号、扰动数据和相互冲突的文献。看似合理的机制仍可能受到毒性、暴露水平、患者异质性或试验设计限制。 • 临床试验要求系统把统计分析计划转成可执行代码与可审计的表格、列表和图形。计划存在缺口时，求解器必须声明有依据的选择，并保存单元格级数据血缘。 • 大模型工程使用真实模型、语料、GPU、资源预算和隐藏检查。训练可能持续数小时保持稳定，随后因熵耗尽和梯度失稳而失败。 这些环境可以暴露静态提示词难以复现的领域故障。 项目团队报告称，10 位 STEM 博士用两个月调查了 16 个行业部门中的 561 个产业，收集 423 个高价值问题，并选择其中 20 个继续开发。候选问题需要具备可检查的交付物、实质性的中间任务、可度量的评分规则、可运行实例、外部证据来源，以及未来确认或证伪结果的明确路径。 开放问题的验证原则： 今天可以没有答案，但必须预先说明哪些证据能够确立答案。 新药可能需要多年验证，新材料可能尚未制造，某个假说也可能依赖未来实验。TRACES 仍可检查当前调查有没有保存后续验证需要的证据、不确定性和适用边界。 结果验证与过程验证 TRACES 把评估分成两个通道。 结果验证器（outcome verifier）使用隐藏真值、留出数据、模拟结果、实验测量或可量化代理指标，评价最终交付物和承载结论的中间产物。它还可以设置硬性关卡，拦截数据污染、答案泄漏和无效提交。 过程验证器（process verifier）评价求解器如何得到结果。它读取外部可见的完整轨迹，包括行动、观察、产物、主张和修订，同时看不到最终验证结果与求解器身份。模型私有的思维链会被移除。 HDS6 评价六种过程能力： • Tools，工具：选择、调用并正确解释合适的工具； • Repair，修复：定位真实故障、修正原因并确认修复有效； • Alternatives，备选解释：保留相互竞争的假设，并随着证据变化更新判断； • Coherence，连贯性：在长时间运行中维持状态、约束与逻辑一致； • Evidence，证据：让重要结论可以追溯到观察、数据、实验、工具输出或来源； • Scope，适用边界：说明结论成立的条件与可能失效的范围。 六个英文首字母组成 TRACES。结果得分与过程得分分开后，一个总分容易掩盖的四种情况会变得清楚： • 结果强 · 过程强：结果成功，调查过程也可以检查。 • 结果强 · 过程弱：系统可能依赖运气、数据污染或无法泛化的理由得到正确结果。 • 结果弱 · 过程强：调查过程可能严谨，但受到噪声、统计功效不足或高难度环境影响。 • 结果弱 · 过程弱：结果与过程都需要修复。 临床统计表案例说明了双重验证的价值。只检查最终单元格时，两次运行等价；检查完整轨迹后，可复现报告与证据不完整的正确数字会得到不同评价。 过程反馈与修复 TRACES 可以把过程诊断转成简短的修复说明。说明会定位错误步骤、描述更高评分所需的行为，并要求求解器重新尝试，同时隐藏答案与结果得分。 论文中的一个案例要求求解器估计多个来源包含多少份独特且高质量的文档。第一次运行提到了“捕获—再捕获”方法，却没有使用已经收集的文档指纹。系统直接假设每个来源约有 100 份文档，并提交估计值 108。算术成立，关键输入却缺少证据。 过程批评器定位了这个问题。修复后的求解器根据文档指纹的重复频率估计未观察到的文档，提交 2,471，更接近隐藏真值 1,505。估计得分从 0.05 提升到 0.30。 修复结果仍然明显高于真值。反馈提高了证据基础，也暴露出尚未解决的校准问题；它无法保证成功。 在 434 条被判定为存在缺陷的轨迹上，加入修复说明后的重新运行使环境结果得分平均提高 0.155。10 个测试环境中有 9 个的平均变化为正：204 次改善，144 次不变，86 次下降。 这个结果值得关注，但证据尚未形成定论。实验缺少匹配对照组，没有让相同轨迹在缺少修复说明的条件下重新运行，因此普通的运行间波动可能解释部分提升。这个限制保留了一项重要的替代解释。 TRACES 与其他基准的区别 TRACES 建立在推理、交互式智能体、研究工程与科学评估等既有工作之上。论文的相关工作显示，各类基准采用不同的评估单位： • 静态推理（MMLU、GPQA、Humanity's Last Exam）：固定提示与答案；检查最终答案的正确性。 • 交互式智能体（WebArena、OSWorld、GAIA）：数字环境中的有界运行；检查任务完成情况。 • 研究与科学智能体（SWE-bench、RE-Bench、PaperBench、DiscoveryBench）：补丁、实验或科学产物；使用可执行测试或结构化评分标准。 • TRACES：求解器与环境组成的完整运行实例；检查隐藏结果、承载结论的中间产物与盲评 HDS6 轨迹。 这组对照比较评估单位，无意排列难度，各类基准之间也存在交叠。TRACES 的关键设计是同时评价最终提交与外部可见的轨迹。固定的 episode 接口还能保持问题、环境、工具、预算和验证方式稳定，从而受控比较模型、运行框架、智能体循环、初始指导与反馈处理方式。 TRACES 真正提出了什么 论文的核心论点关乎评估单位。承担重要任务的 AI 系统需要在长程调查中保持状态，区分观察与假设，保留仍然成立的备选解释，用证据支撑关键主张，说明结论边界，并修复已经确认的故障。 TRACES 提供了一套具体评估设计，用于检验完整 AI 系统能否开展可靠、证据充分且可以自我修正的调查，并让结果与轨迹接受审计、质疑、复现和扩展。 主要来源 • Apodex Discovery 技术报告，arXiv:2608.11341v1 • TRACES：工作原理 • TRACES 问题注册表 • TRACES 公共环境 • Hugging Face 论文页面

译Apodex Discovery 推出 TRACES 基准，评估 AI 系统能否通过证据充分、可审计、可修复的调查得到正确结果。它同时检查最终答案与外部可见轨迹，用 HDS6 盲评六种过程能力（工具、修复、备选解释、连贯性、证据、适用边界）。在 434 条缺陷轨迹上，加入修复说明使环境结果得分平均提高 0.155，但实验缺少匹配对照组。

Rohan Paul@rohanpaul_ai · 2天前30

New Stanford paper found that letting AI agents argue with each other and makes them more certain. The team ran over 10,000 small communities of language-model agents. Letting agents discuss a math problem moves the group toward the right answer. Each had 32 agents with different personas, linked by friendly or unfriendly ties. They traded messages for 8 rounds about math problems with a right answer and political statements without one. On math, the talking helped. Wrong majorities flipped to the correct answer more often than correct ones flipped away. On politics, the same process pushed groups rightward in 3 of the 4 models. If your system uses agent debate, track which way the group drifted, not only whether the score went up. – arxiv. org/abs/2608.16578 Title: "Physics of Agents: Statistical Mechanics Predicts Collective Behavior of AI Agents"

译斯坦福大学新研究发现，让AI智能体相互辩论能提升其确定性。团队运行超1万个由32个不同人格智能体组成的小型社区，经8轮信息交流后，数学问题上错误多数派转向正确答案的频率高于正确派转向错误；政治议题上，4个模型中3个出现右倾趋势。研究建议，若系统采用智能体辩论机制，应追踪群体漂移方向，而非仅看分数提升。

elvis@omarsar0 · 2天前42

Great paper if you are tracking progress in recursive self-improvement (RSI). (bookmark it) There is so much hype around RSI, so I think it's worth understanding why current models are not able to do this properly yet. Issues range from "lack of creativity" of models to getting stuck in a local optimum. This work tries to provide more insights into whether agents can really post-train other agents. Here is the most interesting finding reported in the paper: "the agent’s training strategy is locked in at the very beginning, and the entire remaining budget is spent on local adjustments within the selected strategy." They analyzed a large corpus of publicly released post-training trajectories. Across tasks, the agent locks in its training strategy at the very first step and spends the entire remaining budget on local adjustments inside it. They then tried three escalating fixes. An experience-driven scaffold lifted execution broadly, worth 12.6 points on GSM8K and 40.8 on HumanEval, and the strategy stayed frozen. Human guidance redirected the opening choice, and the agent slid back into local loops once training began. Extra inference compute paid off on easy tasks and did almost nothing on the hardest one. What agents lack here is a way to reconsider strategy while execution is still running. Paper: https://arxiv.org/abs/2608.19072 Track more trending AI papers in our academy: https://academy.dair.ai/

译一篇新论文分析大量公开的后训练轨迹，发现智能体在第一步就锁定训练策略，剩余预算全用于局部调整，无法真正实现递归自我改进。研究者尝试三种升级方案：经验驱动脚手架在GSM8K提升12.6分、HumanEval提升40.8分，但策略仍冻结；人类引导和额外推理算力均未解决根本问题。核心缺失是智能体在执行中缺乏重新考虑策略的能力。

Rohan Paul@rohanpaul_ai · 2天前27

Most of the tests used to grade an AI agent are a waste of money. A new University of Tokyo paper counts them. Tasks every version passes, plus tasks no version can pass, are over 70% of the test set. They cost the same as any other test and tell you nothing. So the method, Task-CoEvolve, keeps only the tests where past versions disagreed, and picks a fresh set each round, because a test that was hard last round may be easy now. It then adjusts the score for how the tests were picked, so rounds using different tests still compare. On Terminal-Bench 2.1, grading each version on 20% of the 89 tasks ends within about one task of grading all of them. Cost drops 67 to 80%. Time drops by half. Picking the same number of tests at random is cheaper still, but random picking grabs the short easy ones, and lands 3.3 points worse. Cheap and useful are not the same thing. Before you shrink an eval loop, ask whether what's left can still tell your candidates apart. – arxiv. org/abs/2608.20169 Title: "Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection"

译东京大学新论文指出，多数AI智能体评测测试浪费资金--所有版本都能通过或都无法通过的任务占测试集超70%，却与其他测试成本相同。新方法Task-CoEvolve仅保留过往版本有分歧的测试，每轮重新选取，并调整评分以跨轮比较。在Terminal-Bench 2.1上，仅用89项任务中20%进行评测，结果与全量评测相差约一项任务，成本降低67%-80%，时间减半。

elvis@omarsar0 · 2天前35

What a fascinating paper on AI agents. A lot of the issues we see with AI agents today revolve around wrong assumptions the LLMs make. This leads to problems like hallucination, cost inefficiencies, unreliable tool calls and much more. I think if we can solve this problem, even current LLMs would significantly improve in terms of performance and efficiency. The problem is that context acquisition is treated as afterthought, but it shouldn't be that way. Users tend to leave out constraints when prompting. So the agent agent needs to guess the default, or spend tokens on a clarifying question, a retrieval call, a tool call, or a prompt trial. This new work gives this problem an objective function. Context acquisition becomes active inference over a latent task state. An inner step updates beliefs, and an outer step picks the next context action, task action, or stop action to minimize expected free energy under cost. In deterministic settings the epistemic term reduces to expected information gain, optionally normalized by token cost. That is directly implementable today as a scoring rule. They coin it as Optimal Question Asking, with exact posteriors and a dynamic programming oracle, then benchmark frontier models on binary and multiway tasks from 25 to 300 candidates. So you can measure the gap between your agent and the true optimum. Paper: https://arxiv.org/abs/2608.19202 Track more trending AI papers in our academy: https://academy.dair.ai/

译一篇关于AI智能体的论文指出，LLM的错误假设导致幻觉、成本低效和不可靠工具调用等问题。该研究将上下文获取定义为对潜在任务状态的主动推理，提出"最优提问"（Optimal Question Asking）方法，用动态规划基准测试前沿模型在25至300个候选任务上的表现，可衡量智能体与理论最优的差距。

Dongxi 东锡 NLP@dongxi_nlp · 2天前35

http://x.com/i/article/2091179277962113024 TRACES: A Benchmark for Discoverative AI TRACES evaluates whether an AI system can produce a correct result through an evidence-grounded, auditable, and repairable investigation. An AI system was asked to reproduce a clinical-trial safety table. It calculated the correct numbers, but chose a patient population without declaring the choice in advance. Its evidence ledger listed the patients behind each numerator but omitted those behind the denominators. Another investigator could read the percentages but could not independently reproduce them. A process critic, blind to the hidden answer and outcome score, identified both failures. In a repaired run, the solver declared its population choice before calculating, recorded numerator and denominator patient IDs, and submitted the same table. The outcome score rose from 0.83 to 0.95 because the result had become reproducible while the numbers remained unchanged. The public clinical-trials environment shows why this matters. A pivotal Phase III package may contain 200–500 tables, listings, and figures. These outputs are commonly reprogrammed independently and reconciled cell by cell. At that scale, one undeclared population rule or missing lineage record can make an otherwise plausible report unauditable. The case states the central distinction in the Apodex Discovery paper: a correct answer does not by itself establish a trustworthy investigation. Most benchmarks measure the final answer. TRACES also evaluates the process that produced it. What conventional benchmark questions predefine A conventional benchmark begins after substantial design work is complete. Someone has selected the problem, formulated the question, prepared the inputs, chosen the permitted tools, set the budget, and defined success. Usually, a reference answer is already available. Open-ended discovery begins before those choices are settled. “Design a better gene-therapy vector,” for example, requires decisions about admissible biological data, capsid viability, tissue targeting, novelty, manufacturability, experimental feedback, and resource limits. Apodex Discovery treats that formulation work as part of the research problem. Its operational unit is a heavy-duty solver: a foundation model together with its harness, memory, tools, and control policy. The evaluated object expands from an answer to a complete investigation: ambition → formulation → action → observation → verification → repair From prompt to executable environment TRACES is the reality benchmark within the broader Apodex Discovery framework. It turns a consequential goal into a problem manifest and a stateful, executable environment. The manifest is a static contract defining the objective, task decomposition, permitted data and tools, resource budgets, success criteria, and hidden verification path. The environment is the interactive substrate in which the solver acts. It returns new observations such as tool outputs, execution errors, retrieved evidence, simulations, measurements, and verifier feedback. A prompt supplies fixed context; an environment changes in response to actions. The public site currently presents four environment families: • AAV capsid design covers viability prediction, tissue targeting, three-dimensional structure reconstruction, and candidate-sequence design under limited experimental feedback. • Drug repurposing requires the solver to reconcile clinical trials, molecular and genetic evidence, safety signals, perturbation data, and conflicting literature. A plausible mechanism may still fail because of toxicity, exposure, patient heterogeneity, or trial design. • Clinical trials turn statistical analysis plans into executable code and auditable tables, listings, and figures. When a plan is under-specified, the solver must declare a defensible choice and preserve cell-level lineage. • LLM engineering uses real models, corpora, GPUs, budgets, and hidden checks. Training can appear stable for hours before entropy depletion and gradient instability cause failure. These environments expose domain-specific failure modes that a static prompt cannot reproduce. The project reports that ten STEM PhDs spent two months surveying 561 industries across 16 sectors. They assembled 423 high-value problems and selected 20 for deeper development. A candidate needed an inspectable deliverable, substantive intermediate tasks, a measurable rubric, runnable instances, external evidence, and a defined path for future confirmation or falsification. Open-problem rule: The answer may be unknown today, but the evidence that could establish it must be specified. A new drug may take years to validate, a material may not yet have been manufactured, and a hypothesis may depend on a future experiment. TRACES can still test whether the current investigation preserved the evidence, uncertainty, and boundary conditions required for later verification. Outcome and process verification TRACES separates evaluation into two channels. The outcome verifier evaluates final and load-bearing intermediate artifacts against hidden ground truth, held-out data, simulations, experimental measurements, or measurable proxies. It can also apply hard gates for contamination, leakage, and invalid submissions. The process verifier evaluates how the solver produced its result. It reads the externally visible trajectory—actions, observations, artifacts, claims, and revisions—while remaining blind to the verified outcome and solver identity. Private chain-of-thought is removed. HDS6 scores six process capabilities: • Tools — selecting, invoking, and interpreting appropriate tools; • Repair — locating the underlying failure, correcting it, and confirming the fix; • Alternatives — preserving competing hypotheses and updating them as evidence changes; • Coherence — maintaining state, constraints, and logic across a long run; • Evidence — grounding important claims in observations, data, experiments, or sources; and • Scope — stating where a conclusion holds and where it may fail. Their initials spell TRACES. Keeping outcome and process scores separate exposes four cases that a single score can hide: • Strong outcome · strong process — The result succeeded and the investigation is inspectable. • Strong outcome · weak process — The system may be lucky, contaminated, or correct for reasons that will not generalize. • Weak outcome · strong process — The investigation may be sound but affected by noise, limited power, or a difficult environment. • Weak outcome · weak process — Both the result and the procedure require repair. The clinical-table case demonstrates the value of this separation. Final-cell scoring treats the two runs as equivalent. Trajectory review distinguishes a reproducible report from correct numbers supported by incomplete evidence. Process feedback and repair TRACES can convert a process diagnosis into a compact repair note. The note identifies a faulty step, describes the behavior needed for a stronger score, and asks the solver to try again without revealing the hidden answer or outcome score. One paper example asks the solver to estimate the number of unique, high-value documents across several sources. The first run mentioned capture-recapture but never used the document fingerprints it had collected. It assumed that each source contained about 100 documents and submitted an estimate of 108. The arithmetic was valid; a critical input lacked evidence. The process critic identified that failure. In the repaired run, the solver used fingerprint recurrence to estimate unseen documents, submitted 2,471, and moved closer to the hidden truth of 1,505. Its estimation score rose from 0.05 to 0.30. The repaired answer still substantially overshot the truth. The feedback improved grounding and exposed a remaining calibration problem; it did not guarantee success. Across 434 deficient trajectories, repaired reruns improved environment-specific outcome scores by 0.155 on average. Nine of ten tested environments had a positive mean change: 204 runs improved, 144 were unchanged, and 86 declined. The result is promising but not conclusive. The study did not include a matched control in which the same trajectories were rerun without repair notes, so ordinary run-to-run variation may explain part of the gain. This limitation preserves an important alternative explanation. How TRACES differs from other benchmarks TRACES builds on prior work in reasoning, interactive agents, research engineering, and scientific evaluation. The paper's Related Work shows how their evaluation units differ: • Static reasoning (MMLU, GPQA, Humanity's Last Exam) — fixed prompt and answer; final-answer correctness. • Interactive agents (WebArena, OSWorld, GAIA) — bounded digital episode; task completion. • Research and scientific agents (SWE-bench, RE-Bench, PaperBench, DiscoveryBench) — patch, experiment, or scientific artifact; tests or structured rubrics. • TRACES — complete solver-environment episode; hidden outcomes, load-bearing artifacts, and blind HDS6 trajectory review. This comparison concerns evaluation units, not difficulty, and the categories overlap. TRACES's distinctive choice is to grade both the submission and the externally visible trajectory. Its fixed episode interface also holds the problem, environment, tools, budgets, and verification constant, enabling controlled comparisons of models, harnesses, agent loops, guidance, and feedback handling. What TRACES actually claims The paper's central argument concerns the unit of evaluation. A consequential AI system should maintain state across long investigations, distinguish observations from assumptions, preserve viable alternatives, ground key claims in evidence, state the limits of its conclusions, and repair identified failures. TRACES does not establish that AI can discover the unknown. It provides a concrete design for evaluating whether a complete AI system can conduct a reliable, evidence-grounded, and self-correcting investigation whose results and trajectory can be audited, challenged, reproduced, and extended. Primary sources • Apodex Discovery technical report, arXiv:2608.11341v1 • TRACES: How it works • TRACES problem registry • TRACES public environments • Hugging Face paper page

译TRACES 是 Apodex Discovery 框架下的现实基准，评估 AI 系统能否通过有证据支撑、可审计且可修复的调查研究得出正确结果。在临床试验复现案例中，修复后的过程评分从 0.83 升至 0.95，而数字未变。该基准区别于仅测最终答案的传统基准，将评估对象从答案扩展至完整调查链：目标→规划→行动→观察→验证→修复。

Rohan Paul@rohanpaul_ai · 2天前37

You can now RL-train an agent through the same complex harness it will actually run in, without needing access to the harness internals. ClawGym II shows that Claude Code or OpenClaw can be treated as a black box and still become part of the RL training loop. The framework runs OpenClaw or Claude Code unchanged inside sandboxes, intercepts model calls at the serving boundary, and rebuilds fragmented calls into prefix-tree trajectories that PPO or GRPO can optimize. That lets the model learn through the harness without the training stack reproducing its tool routing, retries, context management, or subagents. With Qwen3-30A3B, this raised ClawGym-Bench Pass@1 by 9.98 points through OpenClaw and 14.81 points through Claude Code. Mix-harness training also worked: a policy trained from OpenClaw and Claude Code matched or slightly beat the corresponding single-harness models under both execution systems. The paper also reports gains on JobBench and OfficeQA, so the setup extends beyond ClawGym-style tasks. – arxiv. org/abs/2608.16798 Title: "ClawGym II: Exploring Black-Box RL on Agent Harness"

译ClawGym II 框架将 OpenClaw 或 Claude Code 作为黑盒纳入 RL 训练循环，无需访问其内部机制。通过 Qwen3-30A3B，该方法在 ClawGym-Bench Pass@1 上经 OpenClaw 提升 9.98 分、经 Claude Code 提升 14.81 分，混合训练亦有效，并扩展至 JobBench 和 OfficeQA。

Rohan Paul@rohanpaul_ai · 2天前37

LLMs can reproduce the broad direction of human survey results, but not the actual human response distribution. LLMs are useful for cheap pilot surveys where you mainly want the direction of an effect, but this paper finds they are not reliable replacements for actual human survey respondents when you care about realistic distributions, effect sizes, correlations, or downstream analysis. Models trained on LLM-generated survey data averaged R² = -0.18 when predicting real humans, versus 0.28 when trained on human data. – arxiv. org/abs/2608.14606 Title: "Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents"

译一项针对 LLM 作为合成调查受访者的心理测量学审计发现，LLM 能复现人类调查结果的大致方向，但无法还原真实的人类回答分布。用 LLM 生成数据训练的模型在预测真实人类时平均 R2 为 -0.18，而用人类数据训练的模型为 0.28。论文认为 LLM 仅适用于低成本试点调查，不能替代真实受访者。

Rohan Paul@rohanpaul_ai · 2天前41

New Tsinghua and other Chinese Univ paper finds AI agents can optimize a training plan for hours, but rarely realize when the plan itself is wrong. Once training starts, an AI agent's first strategy tends to become the strategy it keeps optimizing. Across 1,338 post-training trajectories, only 74 of 3,557 adjacent training experiments, 2.1%, changed the high-level strategy. Most iterations stayed inside the same approach, tweaking data, hyperparameters, formatting, or bugs. The agents were not inactive. They could train, evaluate, diagnose failures, and improve checkpoints. The problem appeared when evidence suggested the whole approach should change. Adding an experiment journal, skill library, and evaluator lifted GSM8K by 12.6 points and HumanEval by 40.8 points over the Opus 4.6 baseline, but still did not trigger strategy changes. Human guidance could redirect the plan before training, yet the agent returned to local tuning once training began. Even 2–8× more inference tokens mostly bought more refinement, with almost no reliable gain on AIME 2025. For AI R&D agents, the missing mechanism may be an explicit trigger to reopen the strategy itself, not just keep optimizing it. – arxiv. org/abs/2608.19072 Title: "What is Missing from AI Post-Training AI: An Empirical Analysis"

译清华等高校论文发现，AI智能体可优化训练计划数小时，却很少意识到计划本身有误。在1，338条后训练轨迹中，3，557个相邻实验仅74个（2.1%）改变了高层策略，多数迭代停留在同一方法内调整数据、超参数等。加入实验日志、技能库和评估器虽使GSM8K提升12.6分、HumanEval提升40.8分，但仍未触发策略变更；2-8倍推理token也几乎未带来AIME 2025可靠增益。

Rohan Paul@rohanpaul_ai · 2天前42

New Harvard + Stanford paper says, don’t replace your embedding model with an LLM. Use embeddings for the cheap, fast first pass. And bring in an LLM only when the retrieval problem actually requires reasoning. An LLM costs up to 1,431X more than an embedding model of comparable quality. Across 10 LLMs and 26 embedding models on 37 tasks, Gemini 3.1 Pro scored 77.6 versus 77.2 for Octen-8B, a statistical tie. The cost was nowhere close: $154.14 for the LLM benchmark pass versus $0.11 for Octen-8B, or 1,431× more. The task split explains when that extra spend can make sense. LLMs led retrieval by 8.5 points, while embeddings led classification by 5.6; clustering, semantic similarity, and pair classification were effectively tied. Embeddings encode documents once and reuse vectors, while an LLM can read multiple documents together with the query and reason across them. So overall recommendation, embeddings for candidate retrieval, LLMs only where the shortlist actually needs reasoning. – arxiv. org/abs/2608.12875 Title: "The Embedder's Dilemma: LLMs Are Better, but at What Cost?"

译哈佛与斯坦福新论文建议，勿用LLM替代嵌入模型做首轮检索，仅在需要推理时引入LLM。同等质量下LLM成本最高达嵌入模型的1，431倍：Gemini 3.1 Pro在37项任务中得分77.6，与Octen-8B的77.2统计持平，但成本为$154.14对$0.11。LLM在检索任务领先8.5分，嵌入模型在分类任务领先5.6分，建议嵌入模型负责候选检索，LLM仅处理需推理的短名单。

Rohan Paul@rohanpaul_ai · 3天前36

An agent can improve without retraining the model. Once agents rewrite their own prompts, memory, and routing, every update becomes a behavioral change. As agents keep rewriting prompts, memories, skills, and routing rules, those updates become part of what the system has learned. Fixing today's failure can quietly break something that worked yesterday. The paper calls this harness-level forgetting: the model stays frozen, but the behavior around it keeps changing. Their Harness Continual Learning (HCL) framework puts every proposed harness update behind a gate. For evolving agents, prompt, memory, skill, and routing changes should be handled like code changes: regression-test them before they become persistent. – arxiv. org/abs/2608.19013 Title: "Harness Continual Learning: Continual Adaptation Beyond Model Parameters"

译一篇新论文提出"框架持续学习"（HCL）框架，指出智能体无需重训模型，仅通过重写自身提示词、记忆和路由即可实现行为更新。论文将模型冻结而周边行为持续变化的现象称为"框架级遗忘"，并主张对提示词、记忆、技能和路由的改动应像代码变更一样进行回归测试后再持久化。

elvis@omarsar0 · 3天前30

Impressive research from Google on building better environments for agents. Training environments for agents are hand-built and go stale. The agent improves, the environment does not, and it's not able to see the agent's weaknesses in the first place. EnvHarness wraps a static environment in a programmable plug-in layer that reshapes its behavior without touching the underlying logic. Every reshaped environment keeps its original verifier; this is what makes the reshaping safe to train on. EnvRigger treats the policy as a black box, reads its execution trajectories, synthesizes harness components aimed at the diagnosed flaws, then validates them with fresh rollouts. Across five benchmarks in four domains, up to 9.0 points better on held-out instances with 9.8% fewer execution steps. Paper: https://arxiv.org/abs/2608.19880 Track more trending AI papers in our academy: https://academy.dair.ai/

译Google 提出 EnvHarness 与 EnvRigger，解决智能体训练环境静态化、无法随智能体进化的问题。EnvHarness 通过可编程插件层重塑环境行为且保留原验证器，EnvRigger 将策略视为黑盒、诊断其轨迹缺陷并合成新组件。在四个领域五个基准上，held-out 实例最高提升 9.0 分，执行步骤减少 9.8%。

DAIR.AI@dair_ai · 4天前42

Finally a good paper testing whether memory-based self-improving agents actually improve. The re-evaluation adds two things prior work skipped, multiple runs to measure variance and randomly shuffled task orders. Both hurt. Agent evaluation is already noisy on multi-step tasks, and stacking a self-improvement loop amplifies that noise. Default task orderings impose an implicit curriculum that much of the reported gain was riding on. Adding detailed rubrics and environment feedback to memory construction recovers part of the drop, and a significant gap remains. Paper: https://arxiv.org/abs/2608.18066 Track more trending AI papers in our academy: https://academy.dair.ai/

译一项新论文重新评估了基于记忆的自我改进智能体，发现其增益可能源于评估噪声。研究补充了先前工作忽略的两点：多次运行测量方差和随机打乱任务顺序，两者均显著降低性能。默认任务顺序隐含的课程学习是报告增益的主要来源，添加详细评分标准与环境反馈可部分恢复性能，但差距仍明显。

elvis@omarsar0 · 4天前35

Banger paper on harness continual learning. (bookmark it) If you already are allowing your agents to rewrite their own prompts, skills, or memory files, this one is worth your time. (bookmark it) Continual learning has always tracked what changes in the weights. Modern agents accumulate experience in the harness instead, across prompts, memories, tools, skills, and routing rules. What this means is that if you update any harness component, previously reliable behavior can break with the model completely untouched. The paper names that harness-level forgetting and provides a way to measure it. Guarded harness evolution separates proposing an update from committing it. A Continual Optimizer drafts a candidate harness from post-execution feedback, and a Continual Evaluator commits only after checking current improvement, historical retention, and validity. Relative gains exceed 10% across textual reasoning, multimodal perception, and open-world interaction. Paper: https://arxiv.org/abs/2608.19013 Track more trending AI papers in our academy: https://academy.dair.ai/

译一篇关于智能体框架持续学习的新论文指出，现代智能体在提示词、记忆、工具等框架组件中积累经验，更新任一组件可能导致模型未变但行为失效，即"框架级遗忘"。论文提出"受保护的框架演化"机制，由持续优化器生成候选框架、持续评估器在确认改进、历史保留和有效性后才提交，在文本推理、多模态感知和开放世界交互中相对提升超10%。

Rohan Paul@rohanpaul_ai · 4天前32

5× context compression did surprisingly little to GPT-5.5’s final task result. The compressed agent remained statistically as successful as the full-context version. What changed was how it got there: it relied much more heavily on retrieval to reconstruct dropped information. – arxiv. org/abs/2608.16370 Title: "What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics"

译5倍上下文压缩对GPT-5.5的最终任务结果影响出奇地小。 压缩后的智能体在统计上与完整上下文版本同样成功。 变化的是达成方式：它更重度依赖检索来重建被丢弃的信息。 - arxiv.org/abs/2608.16370 标题："上下文压缩让智能体付出什么代价？任务完成指标未揭示的交互成本"

Rohan Paul@rohanpaul_ai · 4天前35

The race to build “AI Scientists” may be optimizing for the wrong unit: the agent alone. This position paper argues that scientific agents should be studied as human-agent systems, where the thing you evaluate is the scientist + agent pair. Most current systems still treat the human as a supervisor: set the goal, review a phase, approve the final artifact. Far fewer are built for continuous, fine-grained collaboration during the work itself. The problem is that agents do not naturally know when they need human input. In 10 science tasks, GPT-5-mini almost never asked for help. But that input matters: in the case studies, experts caught errors the agents missed, while the agents sped up execution. The gain came from collaboration, not autonomy. The paper’s proposed benchmark is therefore different: does the human-agent team produce better science than either member alone, without collaboration cost overwhelming the gain? – arxiv. org/abs/2608.14667 Title: "Position: AI Agents in Scientific Teams Should Be Studied as Human-Agent Systems"

译一篇立场论文主张，科学智能体应作为"科学家+智能体"的人机协作系统来评估，而非孤立优化智能体本身。在10项科学任务中，GPT-5-mini几乎从不主动请求人类帮助，但专家能发现智能体遗漏的错误，而智能体加速执行--增益来自协作而非自主性。论文提出以人机团队是否比任何单独一方产出更好科学为基准。

elvis@omarsar0 · 4天前37

Finally a good paper testing whether agents can really post-train other agents. (bookmark it) They analyzed a large corpus of publicly released post-training trajectories. Across tasks, the agent locks in its training strategy at the very first step and spends the entire remaining budget on local adjustments inside it. They then tried three escalating fixes. An experience-driven scaffold lifted execution broadly, worth 12.6 points on GSM8K and 40.8 on HumanEval, and the strategy stayed frozen. Human guidance redirected the opening choice, and the agent slid back into local loops once training began. Extra inference compute paid off on easy tasks and did almost nothing on the hardest one. What agents lack here is a way to reconsider strategy while execution is still running. Paper: https://arxiv.org/abs/2608.19072 Track more trending AI papers in our academy: https://academy.dair.ai/

译一篇新论文系统检验了智能体能否后训练其他智能体，分析大量公开后训练轨迹后发现：智能体在第一步就锁定训练策略，剩余预算全部用于局部调整。三种干预措施中，经验驱动脚手架在GSM8K提升12.6分、HumanEval提升40.8分，但策略仍不改变；人类引导仅重定向初始选择，额外推理算力对最难任务几乎无效。核心缺失是执行中无法重新考虑策略的能力。

Rohan Paul@rohanpaul_ai · 6天前46

New Stanford paper shows a leaderboard can tell you which agent scored highest on that benchmark, but it may not tell you which agent is actually better for your work. Across 3 enterprise agent benchmarks, this paper finds that less than 3% of score variation comes from the agent itself. A much larger 7–23% comes from the interaction between the agent and the specific task. In plain English: an agent that looks better overall may simply fit the benchmark's task mix better. The problem gets worse on hard tasks. On τ2-bench action checks, reliability falls from 0.752 overall to 0.000 on the hardest task quartile. And across 50 train/test splits, projected reliability correlated with held-out reliability at r = -0.90. So a leaderboard can look precise while being a weak guide for your actual workflow. So for choosing agents, the better question is not "Which agent ranks 1st?" but "Which agent works reliably on my task types, difficulty level, and cost constraints?" That also supports routing different task classes to different agents instead of forcing 1 model to win everything. – arxiv. org/abs/2608.11323 Title: "Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations"

译斯坦福新论文发现，企业智能体基准测试中，不到3%的分数差异源于智能体本身，7-23%来自智能体与具体任务的交互。在τ2-bench最难任务上，可靠性从0.752骤降至0.000；50组训练/测试分割中，预测可靠性与实际可靠性相关系数为r=-0.90。排行榜看似精确，却难指导实际工作流选择。

elvis@omarsar0 · 6天前35

Really interesting paper. I recommend it to anyone interested in training agents using existing harnesses. (bookmark it) ClawGym II runs RL through OpenClaw and Claude Code as opaque boxes. A serving proxy sits at the model boundary and captures every call the harness makes, then those calls get organized into prefix trees so PPO and GRPO can optimize over the recovered multi-turn structure. Qwen3-30A3B gains 9.98 points of Pass@1 through OpenClaw and 14.81 through Claude Code, stable across 200 to 400 optimization steps. Mix-harness training pushes further. One model gets optimized jointly by heterogeneous harnesses, which points at policies that generalize across execution systems instead of overfitting to a single one. Paper: https://arxiv.org/abs/2608.16798 Track more trending AI papers in our academy: https://academy.dair.ai/

译ClawGym II 将 RL 训练置于 OpenClaw 和 Claude Code 等现有框架之上，通过服务代理捕获框架的每次调用并组织成前缀树，使 PPO 和 GRPO 能优化恢复的多轮结构。

Rohan Paul@rohanpaul_ai · 6天前46

Researchers found our current approach to making AI smarter over time has a giant blind spot. AI is not actually understanding or applying high-level abstract lessons at all. Developers spend massive amounts of time building systems that condense past AI mistakes into neat little rules for the future. This paper proves that the AI essentially throws those rules in the trash and only looks at raw historical logs. Modern LLM systems try to get better over time by storing past tasks as either raw step-by-step histories or condensed summary rules. The study tested if these agents actually use their stored memories by secretly swapping the correct tips with random garbage text. • When the step-by-step histories were messed up, the AI failed hard, proving it heavily relies on copying exact past actions. • But when researchers completely corrupted the condensed summary rules, the AI kept acting normally and showed zero performance drop. If an AI cannot apply an abstract lesson to a new situation, it is not truly reasoning or learning. This raises the question if the entire AI industry need to rethink how memory works because right now these agents are just mimicking instead of understanding. arxiv. org/abs/2601.22436 "LLM Agents Are Not Always Faithful Self-Evolvers"

译新研究发现，LLM智能体并未真正理解或应用抽象经验规则，而是仅依赖原始历史日志进行模仿。实验通过篡改存储内容验证：破坏逐步历史记录导致AI性能大幅下降，而完全破坏总结规则却无任何性能损失。这表明当前AI系统更多是模仿而非真正推理学习，引发对行业记忆机制设计的重新思考。

Rohan Paul@rohanpaul_ai · 6天前43

Agent skills work for a very specific reason: they turn messy past experience into a clean procedure the agent can follow. The researchers gave agents the same past trajectories in 2 forms: Workflow Memory, which keeps more execution detail, and a distilled SKILL.md. The skill version performed 6.06 percentage points better than Workflow Memory. Because the agent was not getting more experience. It was getting the same experience packaged better. Their trajectory analysis makes the mechanism clearer: 65.7% of skill cases worked through procedural anchoring, while only 4.5% worked by supplying missing knowledge. So skills mainly help with execution: what to do first, which tools to use, what to verify, and which mistakes to avoid. This also explains the failure mode. A skill can still hurt when it is used in the wrong situation or followed too rigidly. Overall takeaway, self-improving agents need better distillation and application of experience, not just bigger memory libraries. – arxiv. org/abs/2608.14036 Title: "Demystifying Agent Skills: Why They Work-Until They Don't"

译一项研究揭示 Agent Skills 有效的核心原因：将杂乱的过往经验提炼为清晰流程，而非简单存储更多执行细节。实验对比中，使用蒸馏后的 SKILL.md 比保留完整执行细节的 Workflow Memory 性能高出 6.06 个百分点。轨迹分析显示，65.7% 的成功案例依赖程序性锚定，仅 4.5% 靠补充缺失知识，说明技能主要优化执行顺序与工具选择，但误用或僵化遵循反而有害。

Rohan Paul@rohanpaul_ai · 6天前36

New Harvard+Chicago study. A year of production data shows that efficient LLM serving depends less on a single clever scheduler than on understanding how traffic changes and repeats. The researchers studied 6.12B requests across 9,174 models for 1 year and found repeatable patterns that serving systems can use. Users often come back to the same model with the same growing context. 99% of the reuse they measured came from requests returning within 15 minutes. So the GPU may already have much of the previous prompt computed and can reuse it instead of doing that work again. A load balancer that only spreads traffic evenly can weaken this advantage by sending the next turn to another GPU. The traffic itself also changes over months: popular models turn over, outputs get shorter, and newer users tend to send larger inputs. That makes short or synthetic workload benchmarks a poor picture of long-running production. Overall, routing, caching, and capacity planning should use the history and structure of real traffic, rather than treating every request as an isolated job. – arxiv. org/abs/2608.13573 Title: "A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing"

译哈佛与芝加哥团队分析 9，174 个模型、6.12B 次请求的一年生产数据，发现高效 LLM 服务更依赖理解流量变化与重复模式，而非单一调度器。99% 的缓存复用来自 15 分钟内的重复请求，均匀负载均衡会削弱该优势。研究建议路由、缓存与容量规划应基于真实流量历史，而非短时或合成基准。

elvis@omarsar0 · 7天前37

Recommended resource. Largest dataset of agent skills I’ve come across. Great for mining cool ideas and patterns for your agents.

译GitSkills 数据集收录了 GitHub 上 282，200 个公共仓库中的约 380 万份 SKILL.md 文件，归并为 1，877，981 个不同技能内容，以单个 SQLite 文件提供。该数据集基于 Anthropic 发布技能格式九个月后的全量挖掘，可用于提取智能体设计模式与灵感。

Rohan Paul@rohanpaul_ai · 7天前47

A scary finding from new Pennsylvania Uni paper. AI agents can go off-script in a simple way: they can forget your rules while still remembering the job. If a rule must survive the whole session, normal chat history may be the wrong place to store it. Say you tell an agent, “Never send an email without asking me first.” Long sessions get compressed into a summary so the model can keep going. This paper finds that the summary often keeps the task but drops the rule. Across 3 long-context settings, current compactors kept only 17% of these session rules on average. So the agent may remember that it is managing your inbox, yet forget that it needs approval before sending anything. That is the scary part. The fix was simple: keep standing rules in a separate registry instead of trusting the context summary to remember them. A small 9B extractor tracked the rules as users stated them and restored them after compaction, pushing retention above 90% across all 3 settings. If you build agents that run for hours, use tools, or take actions, treat user constraints like persistent state, not disposable chat history. – arxiv. org/abs/2608.11242 Title: "Lost in Compaction: Evaluating Side-Constraint Loss under Context Compaction"

译宾夕法尼亚大学新论文发现，AI智能体在长会话压缩后常遗忘用户设定的规则，却仍记得任务本身。在3种长上下文设置中，现有压缩器平均仅保留17%的会话规则。简单修复方案是使用独立的9B参数提取器在压缩后恢复规则，使保留率提升至90%以上。