LUMOS:面向可访问性基底的AI智能体的语义操作系统层
阅读原文· arxiv.org现有操作系统接口针对人类用户设计,AI智能体依赖截图、OCR和视觉裁剪带来高token成本、视觉歧义和延迟。LUMOS在AI智能体与操作系统间构建语义交互层,将原生可访问性元数据和浏览器UI结构转换为带稳定标识符、角色、名称、值、边界和动作能力的机器可读语义蓝图,并通过操作系统自动化API查询光标附近UI元素实现实时语义指针定位。LLM通过基于可访问性的观察-行动循环使用受限可见UI原语操作。LUMOS不取代视觉智能体,而是减少对截图的依赖。
Current operating systems expose interfaces optimized for human users but not for AI agents. Humans benefit from pixels, icons, windows, visual grouping, mouse movement, and keyboard shortcuts; AI agents instead need compact semantic state, grounded actions, and reliable feedback. As a result, many computer-use agents are forced to interpret screenshots, OCR output, and visual crops, introducing high token costs, visual ambiguity, latency, and coordinate uncertainty. This paper introduces LUMOS (Language Model Unified Machine-Readable Operating-System Semantics), a semantic interaction layer between AI agents and operating systems. LUMOS converts native accessibility metadata and browser UI structures into machine readable semantic blueprints with stable identifiers, roles, names, values, bounds, and action affordances. It also supports live semantic pointer grounding by querying the UI element under or near the cursor through operating-system automation APIs. An LLM then acts through an accessibility grounded observe act loop using constrained visible-UI primitives rather than application-specific scripts. LUMOS does not claim to replace visual agents; instead, it reduces dependence on screenshots when operating systems already provide semantic structure. These results suggest a path toward AI-native operating systems and machine-readable interaction layers.