Rohan Paul@rohanpaul_ai · 4天前44This paper says the web needs new rules because AI agents now read websites for people.
The problem is that today’s web still assumes a human is looking at each page, seeing ads, clicking links, and reading visual layouts.
AI agents break that setup because they can collect and summarize content without sending people back to the original sites, which hurts publishers and makes websites block them.
The authors propose treating a helpful AI agent like a human’s proxy, so it should get similar access as that person, but with clear identity, purpose, limits, and payment rules.
They propose adding a new “agent metadata” layer to normal web requests, where an AI agent tells a website who it is, which human it represents, and why it wants the content.
The website then uses a new policy file called agents.txt to decide what to do: allow it, rate-limit it, charge tokens, inherit the user’s subscription, serve agent-friendly content, or block bad behavior.
They also want content to carry provenance tags, so agents can tell whether something was made by a human, AI, or both.
Without a new setup, the web may become harder for agents to access, worse for publishers to fund, and less reliable as AI-made content feeds more AI-made content.
----
Link – arxiv. org/abs/2606.19116
Title: "Towards an Agent-First Web: Redesigning the Web for AI Agents"
译一篇新论文指出,当前Web假设人类浏览页面、观看广告、点击链接,但AI智能体可收集并总结内容而不回访原站,损害出版商利益并导致网站封锁。作者提议将AI智能体视为人类代理,在Web请求中添加“agent metadata”,标明身份、所代表的人类、目的、限制和支付规则。网站通过新策略文件`agents.txt`决定允许、限速、收费、继承用户订阅、提供代理友好内容或屏蔽。内容还需附带provenance标签,让智能体识别来源是人类、AI还是两者。缺乏新机制将导致Web更难访问、出版商更难盈利、AI内容循环降低可靠性。