DroneCATS:评测多模态大模型作为无人机通用视觉-语言-动作智能体

HuggingFace Daily Papers(社区热门论文)·2026-09-01 08:00·2天前
AI 导读

论文提出 DroneCATS-Agent 架构和 DroneCATS 基准,将多模态大模型直接放入无人机控制回路,无需微调或函数调用,评测接近、跟踪、视外搜索和多机指挥四项能力。结果显示小型开源模型导航成功率常高于前沿模型,却因过早或从不宣布到达而失败;模型的视觉空间感知尚可,瓶颈在于维持动作协议并正确发出终止动作。

HuggingFace Daily Papers(社区热门论文)
45AI 编辑部评分,满分 100

DroneCATS:评测多模态大模型作为无人机通用视觉-语言-动作智能体

2026-09-01 08:00· 2天前
AI 导读

论文提出 DroneCATS-Agent 架构和 DroneCATS 基准,将多模态大模型直接放入无人机控制回路,无需微调或函数调用,评测接近、跟踪、视外搜索和多机指挥四项能力。结果显示小型开源模型导航成功率常高于前沿模型,却因过早或从不宣布到达而失败;模型的视觉空间感知尚可,瓶颈在于维持动作协议并正确发出终止动作。

Multimodal Large Language Models (MLLMs) are strong perceivers of images and video. We ask how far that reach extends into acting: dropping an MLLM directly into a drone's control loop, with its entire action space declared solely in the prompt. Recent systems approach this setting but increasingly narrow the model's decision-making. We widen it back. We introduce DroneCATS-Agent, an architecture where the MLLM is a swappable component, and DroneCATS, a benchmark treating the model as the independent variable. Beyond merely flying toward a pixel, our agent entrusts the model to yaw and search, deliberate when unsure, and self-declare arrival---all without fine-tuning or function-calling schemas. Evaluating frontier and open models across four core capabilities---approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet---reveals that even the simplest embodied settings are far from solved. Crucially, to identify what breaks first at the edge, our roster scales down to 2B parameters. The findings expose a stark paradox: it is not the flying that fails. Small open models often navigate into the success radius more reliably than frontier models, yet lose the episode by declaring arrival prematurely or not at all. Multi-drone commanding amplifies this divide, with small models failing by blindly copying a single coordinate across distinct views. Viewed as vision-language-action agents, the models' spatial perception holds up, but their action protocol does not. What separates a deployable edge model from a frontier model is not navigation, but the discipline to sustain a declared protocol and emit the correct terminating action. The open problem is closing this gap at onboard compute costs---yielding a fast model that plans persistently and knows exactly when it is done---and DroneCATS is built to measure that distance.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org