1]Meta 2]Nanyang Technological University 3]University of Washington 4]Brown University 5]Northwestern University \contribution[*]Equal contribution \contribution[‡]Intern at Meta \contribution[#]Equal contribution \contribution[†]Corresponding authors

HumanCLAW: Can Vision-Language Models Act Through a Body?

Li Siyao

Jiawei Gu

Shuai Liu

Kairui Hu

Zekun Li

Chengcheng Tang

Po-Chen Wu

Ivan Shugurov

Lingni Ma

Michael Zollhoefer

Sizhe An

Abhay Mittal

Amy Zhao

Ranjay Krishna

Manling Li

Chuan Guo

Abstract

Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM’s decision with motor control. When a task fails, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, e.g., losing balance and falling. In this work, we introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command, and the command is translated into a sub-second chunk of continuous full-body motion with real physical consequences, including gravity and collisions. The body can therefore act freely in the physical world, while execution-side disturbances, balance and motor errors, are factored out. What remains measurable is the model’s action intelligence: its moment-to-moment choice of what the body should execute next. Based on this framework, we build HumanCLAW-Bench: long-horizon, egocentric find-navigate-interact episodes across indoor scenes. We test nine state-of-the-art VLMs and find that none solves the benchmark; the best model reaches only a success rate. Recognizing the target is not the bottleneck. What current VLMs lack is embodied self-awareness: they lose track of their own body, failing to tell where it is, whether it has reached the goal, or whether it has hit an obstacle.

1 Introduction

Recent vision-language models (VLMs) achieve strong performance on tasks involving visual grounding, spatial relations, and spatial reasoning (Chen et al., 2024; Yang et al., 2024). If such a model is given a physical body, an egocentric observation, a history of its actions, and a goal, can it determine an executable action appropriate for the current moment? A positive answer would be tantalizing: the decision would come from general-purpose reasoning rather than from a policy fitted to a specific embodiment trajectory; an agent built on this reasoning could act in scenarios it was never trained on, without collecting massive action demonstrations for the decision maker. We call this capacity action intelligence: the ability to decide, moment by moment, what a physical body should do next inside the execution loop, where each decision is executed and its physical outcome informs the next. We view it as the operational component of spatial intelligence, converting spatial understanding into closed-loop embodied decisions by selecting, parameterizing, and sequencing actions as their physical consequences unfold.

In this paper, we explore whether recent VLMs have action intelligence. Yet turning a VLM’s decision into executable motion is not trivial. For a simple embodiment, the path is short: a -DoF end-effector action can be directly written out as text (Yang et al., 2025). A complex embodiment like the human body offers no shortcut. With well over a hundred pose dimensions changing many times per second, full-body motion is far too much to write out as text at every control step. Existing evaluations largely fall on either side of this gap between text command and executable motion. On one side, agent benchmarks abstract away the motion: to keep evaluation tractable, commands are executed through simulator-defined action abstractions, scripts, or pre-animated routines (Puig et al., 2018; Shridhar et al., 2020; Liu et al., 2024; Chang et al., 2025), so the model never has to realize full-body behavior whose physical consequences emerge from the motion itself. On the other, end-to-end VLAs do learn to generate executable motion (Zitkovich et al., 2023; Kim et al., 2025; Black et al., 2024; Lee et al., 2025), but fitting on embodiment-specific trajectories entangles the decision with the learned control policy, making it difficult to isolate how far the generalist’s reasoning itself extrapolates into physical action. Moreover, for a humanoid, motor execution can further confound evaluation: the body may, for example, lose balance while climbing stairs, producing a failure unrelated to the action decision. Therefore, measuring action intelligence requires what neither side provides: a setting where the decision is decoupled from execution, yet the body still acts freely in the physical world (Figure˜1).

To this end, we first endow an off-the-shelf VLM with the ability to act freely in the physical world. HumanCLAW closes this gap without training the decision maker. At each sub-second step, the harnessed VLM takes as input its egocentric view and text history, reasons explicitly about what it sees, where its body stands, and what to do next, and outputs one atomic whole-body skill with parameters, checked by a skill-specific verifier. Atomic is the operative word: a skill is a minimal, unambiguous unit of bodily action—walk at a given stride, turn by a given angle, sit at a given height—never a compositional instruction like “sit on the sofa,” which silently bundles finding, approaching, orienting, and committing into one fuzzy command. Keeping the vocabulary atomic pushes this composition into the VLM’s own step-by-step reasoning, with no task-specific action training for the decision maker. A skill-conditioned motion generator supplies what the VLM cannot emit: it realizes each accepted skill as continuous, hundred-DoF human motion. A half-physics simulator then applies its physical consequences through contact, collision, gravity, and object displacement, while factoring out balance and motor-tracking failures. The resulting egocentric view returns for the next decision, closing the loop (Figure˜2). In effect, the VLM is playing a first-person game (Tan et al., 2024; Zhang et al., 2025a), but through a richer, physically grounded abstraction: the action space consists of parameterized whole-body skills rather than key presses, and outcomes are determined by physical interaction rather than by script. Each step thus pairs an explicit decision with its physical outcome, allowing failures to be attributed at the skill-decision level.

Based on HumanCLAW, we then propose HumanCLAW-Bench, which asks whether current VLMs can carry an abstract, long-horizon task through to completion. It instantiates one concrete slice of action intelligence: closed-loop whole-body decision-making. The benchmark contains egocentric whole-body find-navigate-interact episodes across indoor houses: find a target object, bring the body to it, and sit on it. Each goal specifies what to achieve but leaves open what the body should do at every moment, so success depends on a sequence of sub-second action decisions. The decision maker remains frozen throughout; the benchmark measures how far internet-learned reasoning extrapolates when each choice becomes physical motion.

Across nine frontier VLMs, none reliably solves the benchmark: the strongest model completes the full find–navigate–interact progression in only of episodes. The results reveal a consistent gap between seeing and acting through a body. Once a target is genuinely rendered in the egocentric view, the strongest model’s reported target-seeing rate comes within percentage points of the objectively measured visibility rate. The failures concentrate in everything after seeing: the agent cannot reliably determine whether it has arrived, where its body is placed relative to the scene, or when its motion has collided with the world. We call this missing capacity embodied self-awareness: the ability to maintain an online estimate of one’s own body state, its relation to the surrounding scene, and the physical consequences of its recent actions. Current VLMs can describe the world, but they do not reliably track the body they now control. With motor execution factored out by construction, these failures belong to the decision maker.

• •

  HumanCLAW, a controlled setting in which an off-the-shelf VLM acts freely in the physical world: the decision stays decoupled from motor execution, each atomic skill is realized as continuous whole-body motion with physical consequences, and every failure reads at the decision level—with the decision maker never trained.

• •

  HumanCLAW-Bench, egocentric whole-body find-navigate-interact episodes across indoor houses with staged, progressive metrics.

• •

  An extensive evaluation and failure analysis of nine frontier VLMs, which locates their bottleneck not in perception but in embodied self-awareness: knowing where the body is, whether it has arrived, and what its motion has just caused.

2 HumanCLAW

2.1 Problem Setting

CLAW-like agents make language goals actionable through symbolic interfaces. HumanCLAW studies the same agentic loop when the interface is a full body in physical space. The agent still observes, decides, acts, and receives feedback, but an action no longer ends as a command: it is realized as body motion and constrained by the physical world.

Concretely, HumanCLAW pursues a long-horizon task specified by a single high-level instruction (e.g., “find a sofa and sit on it”) that fixes the goal but leaves every execution detail open. As shown in Figure 2, it closes this loop at a sub-second rate. At each timestep , the agent takes a 2D egocentric RGB observation and a compact textual history of past decisions and feedback, and a skill-decision module built around a harnessed VLM maps them to a single parameterized skill call

(1)

where is an atomic skill from a fixed skill set and its continuous parameter. A decoupled motion generator then realizes the call as a sub-second, zero-shot-reliable future motion chunk

(2)

conditioned on the current body state . Finally, a locomotion-decoupled physical simulator executes this chunk, updates the physical state of the world, and returns the next observation .

Observe–think–act thus forms a complete closed loop: rather than committing to a fixed plan, the agent dynamically decides what to do next and continually adjusts to the spatial relation between its body and the scene that its own actions reshape.

2.2 Egocentric Action Decision

As shown in Fig. 3, HumanCLAW employs a contextual reasoning harness that enables an open-ended VLM to infer the low-level motion skill a humanoid should execute over the next s from egocentric visual observations. Rather than prompting the VLM to predict motion directly, the harness progressively structures its decision-making through guided reasoning. It grounds the model in the agent’s embodiment and task context, elicits an explicit ego-centric spatial state, guides reasoning from high-level intent to mid-level decisions and low-level skills, and verifies the proposed skill before execution.

Contextual Prompting. At each step, the VLM receives a prompt that specifies the agent identity, the high-level instruction, the available motion skills, and the rules for using each skill. For example, the model is told that it controls a blue humanoid robot with an egocentric camera, and that its high-level goal is to find a target object and navigate to or interact with it, such as finding a toilet and sitting on it. The prompt also includes a compact textual history of recent steps, including previous visual states, reasoning traces, selected actions. This history is a key cue for action coherence: it informs the VLM what it has been trying to do, how the environment responded, rather than treating each egocentric image as an independent captioning problem.

Ego-Perception and Spatial Understanding. Before choosing a motion, the VLM is required to output a formatted visual state from the current egocentric observation. This state describes visible objects from left to right, their approximate distance to the body, whether the target object is visible, and whether the immediate surrounding area contains obstacles or relevant affordances such as stairs, open space, or a sitting surface. This step externalizes the model’s spatial interpretation, making the subsequent decision depend on an explicit ego-view understanding rather than an implicit image-to-action guess.

High-to-Mid-to-Low Reasoning. The core of the harness is a progressive high-to-mid-to-low reasoning scaffold. The VLM first interprets the long-horizon instruction under the current visual state, e.g., whether the agent should search for the target, approach it, align with it, or complete an interaction. It then derives a mid-level objective that may remain valid for several seconds, such as moving toward a visible object, entering an open area, aligning with stairs, or turning before sitting. Finally, the VLM converts this mid-level objective into the low-level motion skill to execute at the current step. This differs from treating the mid-level objective as an open-loop plan. At every step, the VLM explicitly decide whether to inherit the previous mid-level objective or revise it according to the new egocentric observation and feedback. The harness therefore preserves the stability of hierarchical planning while keeping the policy closed-loop and reactive.

Low-Level Motion Skill Choice. After spatial understanding and mid-level reasoning, the VLM selects a motion skill from a predefined skill pool and predicts its associated parameters. Specifically, it infers both the skill category (e.g., turn or sit) and its continuous parameters (e.g., <degree> or <height>). The resulting action is emitted in a structured JSON format, providing the motion generator with a deterministic interface for instantiating the selected skill.

Skill-Specific Spatial Response Verifier. Before motion generation, HumanCLAW applies a short-context verifier specialized to the proposed skill. The verifier checks whether executing the proposed motion in the current spatial state would cause an unintended consequence: for example, walking into an obstacle, stopping before the target is reached, climbing without a visible stair, or sitting from an invalid pose. This step is motivated by an observation that VLM spatial reasoning degrade as rollout context grows longer; the planner may hallucinate progress, such as claiming it has reached an object that remains far away. The verifier uses compact, skill-specific prompts to decide whether to accept the proposal or replace it with a corrected skill from the same action pool; the per-skill checklist is given in Table˜6.

2.3 Skill-Conditioned Motion

Real human actions are fundamentally different from hard-coded actions in a symbolic agent. A code agent can turn left and immediately turn right, or switch from walking forward to walking backward without any transition. Human motion cannot: a person needs to decelerate, shift weight, recover balance, and produce short transient motions between intentions. These unavoidable “imperfections” are part of embodied action intelligence. Action decision should therefore be executed and evaluated under the physical consequences of natural human motion, not under an idealized instantaneous controller.

Meanwhile, however, the decoupled design requires a reliable zero-shot interface between the VLM skill selector and the motion actor. If the planner issues walk forward, the generated motion should correspond to a commonsense human walk: it should not moonwalk, stagger, or produce unrelated upper-body motion. Otherwise, failures can no longer be attributed to the planner’s action intelligence, because the motion interface itself is misaligned with the semantic meaning of the skill.

Atomic Skills with Deterministic Text Interface. We address this tension by defining a finite set of atomic, semantically unambiguous motion skills. The VLM agent is responsible for composing these skills over time, while the motion generator is responsible only for realizing the selected primitive. For example, sitting on a sofa is not exposed as a single command such as sit(sofa at (x,z)). Instead, it is the VLM agent that decides whether to walk toward the sofa, turn, step backward, or sit based on the current egocentric state. This keeps long-horizon composition in the action intelligence module while keeping each motion primitive locally reliable.

Here specifies local walking displacement and heading change, is the signed turning angle, denotes stair height and depth, and denotes the target sitting height. Each skill has a deterministic text and JSON interface for the VLM, but its execution is generated as continuous human motion rather than retrieved as a fixed animation.

This design gives HumanCLAW both sides of the decoupling. From the VLM’s perspective, the action space is stable, interpretable, and commonsense-aligned: a skill name has a predictable local outcome. From the simulator’s perspective, the executed trajectory remains a realistic full-body motion with natural transitions, contact-relevant body pose, and variation across contexts. The motion generator below implements this interface by using a general motion DiT as a human motion prior and attaching skill-specific ControlNet adapters for parameterized control.

This interface requires motion generation to be both realistic at the trajectory level and reliable at the skill level: motions should contain natural transitions and variation, while each skill name consistently produces its intended local outcome.

Motion Base DiT. Unlike most motion generation models that synthesize a complete sequence, our backbone follows Zhang et al. (2025c) and is trained for receding-horizon continuation. At step , it observes the last five clean body states and generates the next fifteen frames at fps, corresponding to the s motion chunk executed by the simulator. The base model is intentionally skill-agnostic: it inherits the current motion dynamics, but not the skill name or task intent. It therefore serves as a general prior over locally natural human motion, while semantic control is added later through skill ControlNet.

A key design choice is how the past motion state conditions generation. Instead of compressing history into a single condition vector (Zhang et al., 2025c), we concatenate it directly with the noisy future as motion tokens:

(3)

Each frame state is linearly projected to a M-parameter, -layer DiT with -head self-attention processes the full -frame sequence. The flow time is embedded by a sinusoidal MLP and injected through adaLN-Zero in every DiT block; the base model uses no history encoder and no semantic condition. The network predicts a velocity field for all tokens, and we keep only the future-token outputs:

(4)

This history-as-token conditioning makes kinematic continuity explicit: future tokens can directly attend to the exact recent pose and velocity states, rather than relying on a compressed history representation that may lose boundary information.

We train the backbone on AMASS motions preprocessed into -frame chunks at Hz. Before training, each segment is calibrated into a chunk-local frame whose origin and heading are defined by the initial body state. This removes global translation and yaw, so the base DiT models local human dynamics rather than absolute world motion. The first five frames are clean history, and the last fifteen frames form the clean future chunk . Flow matching samples Gaussian noise and a time , constructs

(5)

and minimizes on the future frames. At inference, we start from noise and solve the learned ODE with a -step midpoint solver. The resulting base DiT provides a skill-agnostic motion prior: it produces motion that is kinematically continuous with the past, while the next stage aligns this prior with the text-level skill name and its continuous parameters.

Plug-and-Play Skills as ControlNet. Once the base DiT provides a kinematically continuous motion prior, the next problem is to align this prior with the zero-shot skill interface exposed to the VLM. As shown in Figure 4, we do this with one ControlNet-style adapter per atomic skill, while keeping the base DiT frozen. Thus, adding a new skill only involves training a new ControlNet and does not change any previously trained skill. At rollout time, the VLM-selected skill name simply chooses which adapter to attach, and the skill parameters provide the adapter condition.

For a skill with continuous parameter , we copy the DiT blocks of the base model into a trainable control branch and add a skill-condition encoder . The encoder is chosen according to the parameter type: scalar conditions such as side-step distance or sitting height use an MLP, while spatial conditions such as walking displacement and yaw use Fourier features before projection to the DiT hidden dimension. At layer , the frozen base branch and trainable control branch are combined as

(6)

where is the frozen base block, is the trainable control block, is the skill-condition encoder, and is initialized to zero. The final projection layer remains the one from the base DiT; the adapter only modifies the hidden token stream.

The skill parameterization is deliberately tied to physical outcomes that can be read from the motion chunk itself. For walking, the condition is the chunk-local final pelvis displacement and yaw change. For side stepping and stepping backward, it is the corresponding lateral or backward displacement. For turning, it is the final body yaw. For stairs and sitting, the parameters are heuristic geometric quantities such as step height/depth or target sitting height. Therefore, training does not require manually labeled action parameters: after we curate or filter AMASS clips for a skill, the continuous condition is extracted from the calibrated chunk geometry. This does not mean that the training data is used indiscriminately. We manually review and aggressively filter the clips for each ControlNet to keep the skill distribution single-purpose and commonsense-aligned. Clips with abnormal style or mixed intent, such as unstable walking, walking while performing unrelated upper-body actions, or stair motions with excessive body twisting, are removed. This curation is important for zero-shot alignment: when the VLM calls walk, turn, or climb_upstairs, the corresponding adapter should realize the ordinary meaning of that skill rather than an idiosyncratic AMASS motion.

Each adapter is trained with the same flow-matching objective as the base DiT. The base network is frozen, and only the condition encoder, copied control blocks, and zero-initialized residual projections are optimized. Given a clean history, a noisy future, and the extracted skill condition, the adapter predicts the future-frame velocity field and is supervised with . We train skill adapters with AdamW using learning rate , batch size , and to optimization steps depending on the skill. We do not use classifier guidance or classifier-free guidance; skill control is provided directly by the selected adapter and its deterministic parameter interface.

Figure 5 shows these properties qualitatively. On its own the base DiT produces diverse but unconstrained trajectories (a); attaching the skill ControlNet converges them to the zero-shot-reliable behavior the skill name denotes (b). Because all skills share the same motion backbone, transitions between consecutive skills stay smooth, with inertia consistent with human dynamics (c). Moreover, as the continuous condition is read directly from motion geometry, this unsupervised parameterization affords fine-grained control over the skill outcome, e.g., sitting to different target heights (d).

This design makes the skill set extensible. Adding a new primitive only requires defining its text/JSON schema, collecting or filtering motion clips with the relevant outcome parameter, and training a new adapter on top of the frozen base. Conversely, improving one skill adapter does not affect the VLM prompt format, the simulator, the base prior, or the other skills.

2.4 Locomotion-Decoupled Physical Simulation

Another challenge in studying embodied action intelligence is how to decouple action reasoning from low-level locomotion motor control. In a fully dynamic humanoid simulation, long-horizon task failure can arise from the motor system. For example, in cluttered environments or on stairs, a humanoid may lose balance and fall, prematurely terminating the episode and preventing a meaningful evaluation of its action intelligence.

At the same time, action intelligence must be evaluated through physical outcomes. Existing code agents often behave more like ghosts: they follow predefined kinematic animation trajectories, bypassing the physical effects afforded by a humanoid’s full action space. For a humanoid, however, whether an action is appropriate depends on its effect in the world: the humanoid may safely pass through a narrow space, knock over a movable object, be blocked by a wall, or stop before reaching the target. A purely kinematic simulator removes these consequences, while a fully dynamic torque controller entangles them with balance and motor-skill failures. HumanCLAW therefore adopts an intermediate half-physics (Siyao et al., 2025) simulation mechanism that preserves physical interaction with the scene while abstracting away low-level motor control.

In HumanCLAW simulation, the world follows rigid-body physics, including collision, friction, gravity, articulated objects, and movable objects. The human body, however, is not actuated by simulated joint torques. Instead, it is driven by equivalent kinematic velocities derived from the generated motion sequence. This means the agent does not fail because of balance loss or imperfect motor tracking, e.g., while stepping on stairs, but still physically interacts with the environment: a wall can block the body, contacts can push movable objects, and unsupported vertical motion remains constrained by gravity and contact.

Concretely, we build our environment on AI Habitat (Savva et al., 2019) with the Bullet (Coumans, 2015) physics engine. Given a generated s pose sequence from Sec. 2, we convert consecutive poses into equivalent root and joint velocities. For two adjacent poses and with simulation step size , the linear joint velocity is computed as

with rotational components computed from relative rotations in the corresponding tangent space. These equivalent velocities are applied as the only actuation signal for the humanoid. Following prior passive-body simulation practice (Siyao et al., 2025), we run the simulator at Hz and use passive joint stiffness to preserve compliant contact response.

This half-physics design lets HumanCLAW evaluate action intelligence under physical consequences without making the benchmark dominated by locomotion controller failures. The generated motion specifies the intended body movement, while the simulator determines how that movement interacts with the physical world. As a result, failures such as walking into a wall, disturbing objects, or stopping too early are attributable to action-level decisions, while failures from low-level balance control are largely factored out.

3 HumanCLAW-Bench

3.1 Task Design and 3D Scene Setup

We design a find-navigate-interact task to probe long-horizon action intelligence—controlling a body, acting decisively at the  s timescale, and staying aware of how one’s own actions reshape the scene. At the start of each episode the agent receives a single instruction, “find a <obj>, navigate to it with zero distance, and finally sit on it.” The task is progressive: each stage can be attempted only after its prerequisite is met, so the agent is left to explore and decide its actions autonomously. We currently instantiate the interaction as sit (e.g., sitting on a couch or a bed).

We build our 3D scenes on HSSD (Khanna et al., 2024), a large and diverse set of human-designed indoor houses that span multiple floor levels and a wide variety of furniture and small objects. Our benchmark uses the validation houses, which provide episodes over six target categories (chair, bed, couch, potted_plant, toilet, tv); each episode specifies an agent starting pose and a target object category, and is solved by reaching any instance of that category. The six categories split into two subsets: bed, couch, and toilet form the -episode sit subset, whose episodes carry the full find–navigate–interact progression, while chair, potted_plant, and tv form the -episode navigation subset, which terminates at navigation. Find and navigation metrics are therefore computed over all episodes, and interaction metrics over the sit episodes. To expose the agent to physical consequence, we further convert plausibly movable objects (e.g. table-top items, small furniture) from static to dynamic, so that contact displaces them; large fixtures (beds, sofas, toilets, walls) remain static.

Difficulty Analysis.

To enable difficulty-stratified evaluation, we score every episode along three interpretable geometric dimensions of the route to the target: distance, the dataset geodesic distance to the nearest same-category goal; choice, the navigational branchiness, given by the number of turns on the Douglas–Peucker-simplified path plus the number of rooms it traverses; and obstacle, the clutter, given by the average number of obstacles within m of the route. Since HSSD provides only a scalar geodesic distance and no path geometry, we first reconstruct the shortest navigable route by running A^⋆ on a per-floor cm 2D occupancy grid of walls and static furniture; its length matches the ground-truth geodesic to within a couple percent (median ratio ), confirming the routes are faithful. Each dimension is split into easy/medium/hard by fixed physical thresholds (distance /m; choice /; obstacle /) that spread the episodes across tiers (roughly //) rather than forcing equal thirds, and the overall difficulty averages the three levels. Fig. 7 reports the resulting distributions with one worked example per tier, letting us analyze performance and each geometric factor separately across easy, medium, and hard episodes.

3.2 Evaluation Metrics.

We evaluate VLMs along four dimensions: (1) high-level task success, (2) low-level action quality, (3) body awareness and physical disturbance, and (4) computational cost.

Progressive Success Rate. The find-nav-interact benchmark is a progressive task: the agent can proceed to the next subtask only after completing the previous one. We therefore define staged success as both objectively achieved and subjectively acknowledged. That is, the agent needs to satisfy the geometric success criterion and also recognize that the subtask has been completed. The objective criteria are defined as follows. FindSR is achieved when the target semantic ID appears in the ego-rendered image and occupies at least pixels in the view. NavSR is achieved when the minimum distance between the human agent and the target object’s axis-aligned bounding box (AABB) is less than 20 cm. InteractSR is achieved when the agent sits down and the pelvis link, corresponding to the hip region, makes contact with the target. Subjective completion is determined from the agent’s step-wise visual state and its active decision to stop after completing navigation or interaction; the headline metrics in our tables require both the objective criterion and this subjective completion, while Geo-prefixed variants report the objective criterion alone.

Action Quality. A VLM agent should compose motion skills into smooth and coherent behavior. Poor skill orchestration can lead to unstable motions, such as frequent alternating left and right turns. We measure action smoothness using motion jerk, i.e., the third-order derivative of the agent’s kinematic motion.

Body Awareness and Physical Disturbance. Body awareness is a key component of embodied action intelligence. Lack of body awareness can cause unexpected interactions with the environment, such as scraping against a wall while passing through a corridor or knocking over a vase. We therefore report the average number of steps per episode in which a collision occurs, denoted as Colls.. We also measure environmental disturbance by counting the average number of movable objects that are directly or indirectly displaced by the human agent within an episode, denoted as #Dtb, and the average displacement distance of each disturbed object, denoted as dDtb.

Cost. Finally, we report the average number of tokens consumed by the agent per episode, including both input and output tokens, as a measure of computational cost.

4 Experimental Result

4.1 Zero-Shot Motion Fidelity

Low-level skills are zero-shot reliable.

Our controllable motion prior—a shared base diffusion transformer with a per-skill ControlNet, trained once on AMASS/BABEL and queried zero-shot at deployment—faithfully realizes the commanded direction and magnitude of every skill without any per-task or per-scene tuning. In Table 1, commanding a skill with a target magnitude yields an achievement ratio close to unity with small variance: walking tracks the commanded step distance at , turning the commanded angle at , side-stepping at , stepping back at , and sitting reaches the commanded pelvis height at , while stair climbing realizes – of the commanded riser as a stable, calibratable gain. The per-command standard deviation is tiny (), so the controller executes the same command the same way every time—the near-constant ratios reflect a fixed, predictable gain rather than noise. We contrast this with MoMask (Guo et al., 2024), a strong text-to-motion model, prompted with the identical commands rendered in natural language with the magnitude spelled out (e.g. “walks forward at meters per second”). Even with the number in the prompt, MoMask renders a generic motion for each verb instead of the requested magnitude—it walks at a habitual speed and sits to a habitual chair height regardless of the command—so its fidelity ratios depart from unity and scatter with an order of magnitude larger variance (standard deviations of – versus – for ours). Because our motor layer is this reliable out of the box, the benchmark largely isolates high-level embodied reasoning—where to go, what to perceive, and how to sequence actions—rather than low-level motor competence, and the failures we analyze are attributable to planning, perception, and body-awareness rather than to execution of the underlying skills.

媒体内容 · 前往原文查看

Method	walk	side_step	step_back	turn_in_place	climb_upstairs	walk_downstairs	sit_in_place
Ours
MoMask (Guo et al., 2024)

{NiceTabular}

l ccc ccc c ccc High-Level Success Rate Body Misaware & Scene Disturb Action Quality Cost
VLM Backbone FindSR NavSR InteractSR Coll. #Dtb/ep dDtb(m) Motion Jerk avg steps in tok/step out tok/step
GPT-5.5 55.1% 13.9% 3.4% 43.4% 1.53 \cellcolorranktwo1.56 \cellcolorrankone4.2 82.4 \cellcolorrankone4360 354
Gemini-3.1 \cellcolorrankone64.9% \cellcolorrankone42.4% \cellcolorrankone16.8% 39.5% 1.50 \cellcolorrankone1.22 5.7 59.1 5890 \cellcolorranktwo311
Gemini-2.5 \cellcolorranktwo58.5% 21.6% 3.5% 37.7% \cellcolorranktwo1.35 3.03 8.7 71.3 6412 401
Claude-4.8 32.6% 8.6% 1.5% 44.2% 1.73 2.76 5.3 81.0 7047 625
Gemma-4-31B 58.1% \cellcolorranktwo28.7% \cellcolorranktwo11.1% 40.1% 1.66 1.86 \cellcolorranktwo4.8 78.5 4632 322
Qwen3.6-27B 51.0% 20.9% 0.2% 43.0% 1.95 2.16 6.5 81.6 4862 482
Qwen3.6-35B-A3B 44.6% 5.8% 0.0% \cellcolorranktwo34.6% 1.45 2.38 7.4 74.0 4766 477
Qwen3.5-27B 37.8% 13.5% 0.0% \cellcolorrankone34.5% \cellcolorrankone0.93 2.38 6.8 37.6 4641 473
InternVL3.5-38B 46.8% 0.8% 0.0% 51.2% 1.90 2.56 7.2 93.0 \cellcolorranktwo4459 \cellcolorrankone309

4.2 Benchmark

We evaluate nine off-the-shelf VLMs as the frozen decision maker and report the full validation results in Table 4.1.

The best model, Gemini-3.1, finds and sits on the target in only of episodes, and four of the nine sit on it in at most of episodes. Behind this low ceiling is a sharp stage-by-stage drop that every model shares. FindSR ranges from to , NavSR from to , and InteractSR from to , so most of the task is lost twice, first between finding a target and reaching it, and again between reaching it and interacting with it.

Seeing a target rarely becomes reaching it.

We summarize the first drop with a landing rate, the ratio of NavSR to FindSR, which asks how much of what a model finds it actually reaches. Even Gemini-3.1 lands only of what it finds, Gemma-4-31B , and InternVL3.5-38B . Recognizing a target is common across models. Turning that recognition into a body that stops next to the target is what separates them, and it is where the benchmark leaves the most room to improve.

Body awareness and cost should be read together with success.

Low disturbance can signal inaction rather than control. Qwen3.5-27B records the lowest object displacement ( per episode) and the fewest steps (), yet it barely moves and interacts in of episodes. At the other extreme, InternVL3.5-38B and Claude-4.8 pair the highest collision fractions ( and ) with a collapsed NavSR, the profile of an agent that moves without control. Spending more tokens does not help either. Claude-4.8 uses the most input and output tokens per step ( and ) but trails on every success metric, while Gemini-3.1 leads at a lower cost.

Action quality reflects the coherence of a model’s choreography.

Task success says nothing about how an agent moves, so we report a Motion Jerk column—the denoised third time-derivative of the pelvis trajectory at the decision timescale—measuring the coherence of a backbone’s choreography rather than the fixed gait of the shared generator. By construction it penalizes incoherent sequencing, not the amount of motion: a single skill scores – (forward walk , turn ) and a purposeful walk–turn–walk trajectory , whereas an aimlessly sequenced policy reaches ; fixing the skill content and varying only order confirms this—five forward-walk chunks then five lateral steps score , but interleaving the same ten chunks yields ( higher) purely from ordering. Under this measure (lower is smoother), GPT-5.5 is the most coherent (), followed by Gemma-4-31B (), and quality is decoupled from success: Gemini-2.5, despite a competitive success rate, is the worst (), oscillating side to side with the highest turn-reversal rate (about one step in five), while Qwen3.6-35B-A3B is second worst (), adding bouts of in-place spinning.

The strongest open model is close to the frontier.

Gemma-4-31B (//) matches or beats every proprietary model except Gemini-3.1, and it beats GPT-5.5 and Claude-4.8 on all three success rates. Progress on HumanCLAW-Bench is therefore open to the community rather than gated behind a single closed model.

{NiceTabular}

l ccc ccc c ccc Setting High-Level Success Rate Body Misaware & Scene Disturb Action Quality Cost
FindSR NavSR InteractSR Coll.% #Dtb dDtb(m) Motion Jerk avg steps in tok/step out tok/step
Baseline (hist 10, img 1) 58.0% 27.0% 18.9% 41.4% 1.35 9.38 4.9 78.5 4676 322
Text-history length
   hist 0 65.0% 11.0% 0.0% 38.6% 1.27 6.46 5.1 88.9 2582 292
   hist 20 59.0% 27.0% 7.5% 42.9% 1.18 1.56 4.8 80.3 6692 326
   hist 50 60.0% 28.0% 7.5% 40.4% 1.24 6.16 4.8 77.6 10538 320
   hist 100 56.0% 26.0% 11.3% 37.6% 1.08 7.23 4.7 79.7 12904 318
Visual-history frames
   img 2 57.0% 31.0% 9.4% 43.0% 1.19 2.55 4.7 78.8 5234 321
   img 5 55.0% 28.0% 7.5% 41.1% 1.30 8.86 5.1 82.7 6335 324
   img 10 53.0% 13.0% 3.8% 34.9% 0.95 3.19 5.2 92.3 7685 303
Design components
   w/o. verifier 51.0% 2.0% 0.0% 33.2% 0.71 10.40 4.8 47.5 4095 247
   w/o. mid-level 62.0% 30.0% 0.0% 35.4% 1.22 2.08 5.0 70.3 3769 264

4.3 Ablation Study

We ablate the harness on a -episode mini-val split, varying the text-history length, the number of visual-history frames, and the design components around the baseline (hist , img ) in Table 4.2.

The verifier is the decisive component.

Removing the verifier collapses NavSR from to and InteractSR from to , while FindSR barely changes ( to ). Its effect is on acting, not on seeing. Without the verifier episodes also end far earlier ( steps versus ), which shows an agent that stops or drifts before the loop closes. The verifier is what turns a stream of skill proposals into a closed loop that reaches and commits.

Text history is necessary but saturates quickly.

With no history the agent cannot sustain a plan, and NavSR and InteractSR fall to and . A short structured memory recovers most of the gap, but lengthening it does not help. NavSR stays near across histories of , , , and steps, and InteractSR is highest at the shortest setting, while input tokens per step climb from about K to K. A compact history captures nearly all of the benefit at a fraction of the cost.

More image frames can hurt.

Adding a few visual-history frames is roughly neutral, but too many degrade the agent. At img , NavSR drops from to and InteractSR from to while input tokens rise. The decision model is bottlenecked by reasoning over the current view, so flooding it with past frames dilutes the signal rather than improving coverage.

Mid-level reasoning carries the long-horizon interaction.

Removing the mid-level objective barely touches finding or navigating—FindSR () and NavSR () even edge above the baseline—yet InteractSR collapses from to . The reason is that sitting is not a single reactive step but a short compositional routine: the agent has to reach the seat, turn to align its back to it, and only then sit—a several-second sequence that a purely step-by-step policy cannot hold together. Without a mid-level objective to bind these steps, the agent approaches the target but never closes the terminal interaction, and episodes end earlier ( vs steps). Mid-level reasoning is thus what sustains the medium-horizon commitments that turn navigation into interaction.

4.4 Error Analysis and Key Findings

Having seen that no current VLM solves the benchmark, we now dissect where and why episodes fail, using four complementary views of the same rollouts. A capability funnel tracks how episodes drain across the findnavigateinteract stages, and a per-stage breakdown attributes each stage’s failures to a single root cause (Fig. 8); all root-cause labels are assigned automatically by fixed rules over the logged rollouts (Section˜8). Table 4.4 reports success-rate variants that separate objective (geometry-only) from model-acknowledged success at each stage, isolating perception from decision; Table 5 breaks collisions down by body part; and Fig. 9 shows representative body-unawareness rollouts. We organize the analysis into four findings, one per stage of the embodied loop.

{NiceTabular}

l cc ccc cc Find Nav Interact
VLM Backbone FindSR GeoFindSR NavSR@20cm GeoNavSR@20cm NavSR@m InteractSR GeoInteractSR
GPT-5.5 55.1% 63.2% 13.9% 27.7% 15.5% 3.4% 9.7%
Gemini-3.1 \cellcolorrankone64.9% \cellcolorrankone69.9% \cellcolorrankone42.4% \cellcolorrankone44.7% \cellcolorrankone52.5% \cellcolorrankone16.8% \cellcolorrankone21.4%
Gemini-2.5 \cellcolorranktwo58.5% 66.4% 21.6% 23.6% 31.0% 3.5% 4.5%
Claude-4.8 32.6% 55.7% 8.6% 13.5% 11.4% 1.5% 2.2%
Gemma-4-31B 58.1% \cellcolorranktwo67.7% \cellcolorranktwo28.7% \cellcolorranktwo35.6% \cellcolorranktwo33.4% \cellcolorranktwo11.1% \cellcolorranktwo17.9%
Qwen3.6-27B 51.0% 62.8% 20.9% 34.8% 24.2% 0.2% 0.3%
Qwen3.6-35B-A3B 44.6% 59.0% 5.8% 14.0% 11.4% 0.0% 0.0%
Qwen3.5-27B 37.8% 48.9% 13.5% 17.7% 19.2% 0.0% 0.0%
InternVL3.5-38B 46.8% 59.1% 0.8% 12.6% 1.6% 0.0% 0.0%

Table 4.4 separates two notions of finding: GeoFindSR counts an episode whenever the target is objectively rendered in the ego view (at least pixels in the image), whether or not the model says so, while FindSR additionally requires the model to acknowledge it. For the strongest models the two differ by only – points (Gemini-3.1 vs. , GPT-5.5 vs. , Gemma-4-31B vs. ), so once a target genuinely enters the ego view it is almost always recognized—recognition itself is largely solved (Fig. 8a). Where finding breaks is upstream, in bringing the target into view at all: about half of all episodes never do. Replaying every failed episode and attributing it to a single root cause (Fig. 8b), the dominant deficit is mid-level reasoning: chiefly ineffective exploration (), where the agent searches the scene incoherently and never renders the target, together with related search failures—approaching without ever turning to view () and giving up mid-route (). Genuine perception lapses—the target is rendered but goes unacknowledged—account for a secondary , and the rest are ineffective actions (), being unknowingly jammed against geometry (), and abandoning a too-far goal (). Finding is therefore gated by where and how the agent chooses to explore, not by whether it can recognize a target once it appears.

Among the 5,473 episodes across all nine VLMs in which the agent actively finds the target—the object is rendered with 100 semantic pixels in the ego view and is acknowledged in the model’s visible-state description—3,706 (68%) still fail to navigate to it (reach within  m and stop). We replay every such trajectory offline and, using the objective last-frame rendering together with the per-step body displacement and action stream, attribute each failure to a single root cause (Fig. 8). Nearly two-thirds of the failures are errors of egocentric self-spatial awareness rather than of perception or low-level control, and they surface as a termination decision that fails in both directions: the agent either keeps navigating after it has already arrived or stops while still far away, because it cannot tell where its body is relative to the target. The largest group is body awareness (34%): the agent is physically at or against the target yet misjudges its own state—in 20% it has arrived within  m but never signals arrival and keeps navigating, and in 14% it is jammed against an obstacle for a stretch of steps yet keeps issuing forward commands, unaware it is blocked. A further 30% is an ego-spatial distance hallucination: the agent stops and declares arrival while still more than  m away, believing a distant target is within reach. The remaining failures are comparatively minor: mid-level reasoning (17%; the agent turns away—e.g. to avoid an obstacle—loses the target from view and cannot re-acquire it, 16%, or finds it too late to physically reach, under 1%), visual perception (9%; the target has left the ego view but the model still claims to see it), and pure action/locomotion (10%; the body moves freely but keeps turning and issuing ineffective actions, never converging on the visible target). Discovery timing is negligible (under 1%): with a distance-aware criterion, only 0.5% of failures lack enough remaining steps to cover the distance after first sighting. In short, the humanoid agent’s core weakness is not whether it can see the target or whether it can move, but knowing where it is relative to—and whether it has reached—the target it already sees.

媒体内容 · 前往原文查看

{NiceTabular}

l ccccc VLM Backbone Coll.% Arm/Hand Torso Leg/Foot Head
GPT-5.5 43.37 23.24 13.78 38.83 2.99
Gemini-3.1 39.48 \cellcolorranktwo20.97 11.72 33.29 \cellcolorrankone2.62
Gemini-2.5 37.69 22.70 13.63 30.87 2.94
Claude-4.8 44.18 28.97 16.90 37.94 6.93
Gemma-4-31B 40.09 21.21 \cellcolorranktwo11.38 34.72 \cellcolorranktwo2.66
Qwen3.6-27B 42.97 25.48 12.70 36.86 3.54
Qwen3.6-35B-A3B \cellcolorranktwo34.58 21.20 11.40 \cellcolorrankone27.99 3.71
Qwen3.5-27B \cellcolorrankone34.53 \cellcolorrankone20.59 \cellcolorrankone10.93 \cellcolorranktwo29.58 2.87
InternVL3.5-38B 51.17 34.68 23.12 44.75 6.10

媒体内容 · 前往原文查看

Among the 726 episodes across the same nine VLMs in which the agent successfully navigates to the seat—it actively stops within  m of the target and thus already satisfies navigation—513 (71%) still fail to sit. We replay every such trajectory and, from the pelvis–object mesh contacts logged along it (exact contact, not bounding-box overlap) together with the action stream, attribute each failure to a single root cause (Fig. 8). The failures are overwhelmingly errors of body awareness—the agent commits to sitting but puts its body in the wrong place (81%)—rather than errors of decision (19%). The dominant mode is sitting into thin air (58%): the agent issues a sit, yet at no step does its pelvis touch any surface—it simply lowers its body where it is standing, with no seat beneath it. Another 14%sit on the wrong thing: the pelvis lands on a different object or on the floor rather than the target seat (most often when the goal is a couch, bed, or toilet, whose neighbours invite a mistaken landing). A final 9% briefly do reach the seat—the pelvis makes genuine mesh contact during a sit—but the agent then keeps acting (predominantly turning, with occasional stands and re-sits) instead of stopping, and shifts its body back off the seat before the last frame. The remaining 19% are decision failures: the agent reaches and stops standing in front of the seat but never issues a sit at all. In short, once the humanoid has walked up to the target, sitting is gated not by whether it decides to sit but by where it puts its body: four in five failures drop the pelvis onto empty space, the wrong object, or off the seat—a finer-grained instance of the same egocentric spatial deficit that limits navigation. What interaction ultimately demands is thus the finest form of embodied spatial reasoning: placing the body precisely with respect to both itself and the surrounding scene, and anticipating the spatial relation each action will produce—where the pelvis will land relative to the seat once the sit unfolds—rather than merely deciding to act.

Underlying failures at every stage is the agent’s obliviousness to its own body. Collisions concentrate in the parts it does not watch: across all nine models the legs and feet collide most (– of steps), the arms and hands next (–), while the head—seldom near obstacles—almost never does (under , Table 5). Two rollouts make the failure concrete (Fig. 9): in (a) the agent’s arm knocks over a chair that was plainly in view a few frames earlier, yet it never registers the contact; in (b) an obstacle stays continuously visible straight ahead, yet the agent keeps walking its legs into it. In both, the scene is seen but the body is not. A VLM trained purely by visual question answering behaves like a ghost: it holds no instinctive model of which pixels are its own limbs, and it never tries to infer where they are. Part of this blindness is inherited from the interface itself: the agent receives only egocentric RGB and text history, with no proprioceptive body state or contact signal, so any sense of its own body can only be reconstructed from vision alone (Section˜6). This is not a peripheral nuisance—body awareness underlies of navigation failures and of interaction failures (Findings 4–5)—but a core capability that embodied action has yet to acquire: feeling the body, not merely seeing the world.

5 Related Work

5.1 Embodied Tasks and Interactive Environments

Embodied environments differ not only in the tasks they contain, but also in how an agent’s actions are grounded and executed in the simulation. VirtualHome (Puig et al., 2018) models household activities as programs over structured scene states and renders atomic instructions with predefined avatar animations, prioritizing activity semantics over closed-loop physical control. ALFRED (Shridhar et al., 2020) introduces egocentric language following, but exposes a discrete set of navigation and interaction actions whose execution is resolved by the simulator. Habitat 2.0 (Szot et al., 2021) and BEHAVIOR-1K (Li et al., 2023) move toward physics-enabled rearrangement and mobile manipulation, supporting articulated objects and persistent physical state under robot-specific control. Habitat 3.0 further introduces humanoid avatars and human-in-the-loop interaction, primarily to study robots acting around or collaborating with human partners (Puig et al., 2024).

Existing embodied environments are often mismatched to general-purpose MLLMs. Program-driven environments resolve actions through predefined transitions, leaving little need to reason about body motion or physical consequence. Fully physics-based simulators, by contrast, require high-frequency continuous control and entangle spatial decisions with motor execution. HumanCLAW strikes a middle ground: a frozen MLLM selects parameterized whole-body motion skills based on egocentric observations, utilizing motion generation model autoregressively generate continuous long horizon behavior. Semi-physics execution preserves collision, contact, and object disturbance while abstracting low-level control, enabling the basic embodied interaction.

Closest to our find-and-navigate setting are object-goal navigation and vision-and-language navigation, where an embodied agent locates and moves to a target from egocentric observations (Savva et al., 2019; Khanna et al., 2024; Anderson et al., 2018). These tasks, however, typically abstract the agent as a moving camera or a velocity-controlled point and score only whether a location is reached, leaving out how a full body arrives, orients, and interacts under physical consequence—exactly the middle layer HumanCLAW targets.

5.2 Multi-modal Foundation as Embodied Decision Makers

Recent surveys distinguish hierarchical agentic systems from end-to-end vision–language–action (VLA) policies (Ma et al., 2026; Liang et al., 2025). Hierarchical agents use multimodal large language models (MLLMs) for task decomposition, spatial grounding, skill selection, feedback-driven replanning (Ichter et al., 2023; Liang et al., 2023; Huang et al., 2023b; Song et al., 2023; Driess et al., 2023; Huang et al., 2023a; Duan et al., 2024; Li et al., 2026b), and open-world memory with reusable skills (Wang et al., 2023, 2024). Frozen VLMs have also been driven in closed loop through first-person games (Tan et al., 2024; Zhang et al., 2025a), where actions remain key presses whose outcomes are resolved by the game script. Although suited to long-horizon reasoning, they often reduce execution to symbolic actions or skill outcomes, obscuring body-level embodied interaction with environments. End-to-end VLA policies instead learn direct motor grounding from embodied data (Zitkovich et al., 2023; Ghosh et al., 2024; Kim et al., 2025; Black et al., 2024; NVIDIA et al., 2025), but couple task reasoning to robot-specific data and low-level control. Action-reasoning models such as MolmoAct (Lee et al., 2025) externalize an editable mid-level spatial trace before emitting actions, yet the decision maker is still trained on action data and evaluated with execution in the loop, rather than queried as a frozen generalist.

Existing benchmarks assess goal interpretation, action sequencing, and long-horizon planning (Li et al., 2024; Choi et al., 2024; Chang et al., 2025; Yang et al., 2025; Hong et al., 2026), yet primarily score plans, simulator actions, or robot trajectories. HumanCLAW instead evaluates how frozen off-the-shelf VLMs sustain egocentric spatial action over long-horizon tasks. Their skill-level decisions are realized as continuous whole-body motion in a physical world, which preserves collision, contact, gravity, and object dynamics while largely abstracting away balance and motor-tracking failures. This setup evaluates progressive task success together with action coherence, body awareness, and environmental disturbance.

A growing line of work probes the spatial intelligence of VLMs directly (Chen et al., 2024; Yang et al., 2024, 2025; Li et al., 2025), reporting that current models struggle with metric distance, orientation, and multi-step spatial simulation. These evaluations, however, mostly pose spatial understanding as question answering over images, videos, or cross-view pairs, in open-loop settings without environmental feedback. Action intelligence, as we use the term, is the operational component of spatial intelligence that such settings leave unmeasured: converting spatial understanding into closed-loop embodied decisions—selecting, parameterizing, and sequencing actions as their physical consequences unfold—with the model’s own body state part of the problem. It is likewise narrower than a planner in the generic sense (given an observation and a goal, output what the agent should do next): a planner is typically scored open-loop on the plausibility of its proposals, whereas action intelligence is measured through execution, where every decision is realized as motion and its consequence returns in the next egocentric view. Our error analysis is consistent with the spatial-reasoning findings: even when perception succeeds, the dominant failures are egocentric-spatial—self-localization, reachability, and knowing when the body has arrived—so HumanCLAW serves as a physically grounded, closed-loop stress test of exactly this bottleneck.

5.3 Controllable Human Motion Generation as Agentic Tools

Text-to-motion models learn strong priors for plausible full-body movement from large motion-language corpora (Guo et al., 2022; Tevet et al., 2023; Guo et al., 2024; Huang et al., 2024). Beyond just using text, following controllable motion generation methods further condition motion on spatial constraints, character identity, human interaction, ego perception, physics, and music (Hassan et al., 2021; Zhang and Tang, 2022; Zhao et al., 2023; Xie et al., 2024; Yuan et al., 2023; Siyao et al., 2025; Wu et al., 2025; Jia et al., 2026; Cong et al., 2026; Zhang et al., 2025b; Li et al., 2026a; Zhang et al., 2025b; Diomataris et al., 2025; Zhang et al., 2025c; Siyao et al., 2022, 2023, 2024). Although additional control modalities improve motion fidelity, their input protocols beyond the output format of existing MLLMs agent, i.e., text-only output. An agentic executable tool set requires a compact and predictable interface: each tool-calling should follow its semantic intent and continuous parameters while producing a reliable local outcome. Therefore, HumanCLAW proposes a shared receding-horizon motion prior and a plug-and-play adapter for each atomic skill based on text templates with parameters, leaving long-horizon skill selection, parameterization, and composition to VLMs.

6 Discussion

HumanCLAW isolates a middle layer of physical action—continuous whole-body motion with real scene consequences, but without balance or motor-tracking failures—and uses it to ask how well today’s VLMs act through a body. Across nine state-of-the-art models the answer is consistent: none solves the benchmark, and the failures do not lie where a vision–language agent is usually judged. Perception is largely intact—once a target is rendered it is almost always recognized (Finding 3)—yet competence collapses in everything that follows recognition. What is missing is embodied self-awareness: knowing where the body is and when it has arrived (Finding 4), placing it precisely enough to interact (Finding 5), and sensing when it collides with the world (Finding 6). The common thread is that current VLMs reason about the scene but not about the body they now inhabit.

We attribute this to how these models are built. A VLM trained by visual question answering learns to describe what it sees, not to feel what it does; it treats its own limbs as just more pixels and never forms the proprioceptive, consequence-predicting model that acting through a body requires. The result behaves like a ghost—fluent about the world, oblivious to itself. Closing this gap will likely need more than sharper recognition or a longer context window: it calls for persistent spatial memory, calibrated termination, and an internal model of the body together with the spatial relations each action produces.

Beneath the framework lies a philosophical commitment: we hold that generalizable action intelligence arises from reasoning, not from fitting action data. A policy fitted on trajectories generalizes within the support of its data; a reasoner generalizes as far as its knowledge extrapolates. HumanCLAW is built on this bet—the decision maker stays frozen and general, the motor layer is a fixed, reusable prior, and all task-level composition happens zero-shot in reasoning. On this view, a general action agent improves not by collecting ever more trajectories but by becoming a better reasoner: new skills register without retraining, new tasks require only new rules, and every advance in foundation models transfers to the body for free. Our results leave the bet open but sharpened: today’s reasoning does not yet reach the body, yet the failures are decision-level—exactly the layer where better reasoning, rather than more data, could be brought to bear. Absent a comparison against fitted action policies, this last step remains our working hypothesis: what the experiments establish is where the failures sit, not which route will close them.

Our study has limits. The interaction vocabulary is small; richer manipulation would broaden the interact stage and stress finer body placement. Decision-level attribution is likewise relative to the skill vocabulary: with motor execution factored out, what reads as a decision failure still depends on the granularity of the atomic skills, and a finer or coarser vocabulary would draw the boundary elsewhere. Half-physics deliberately abstracts away balance and motor tracking, so HumanCLAW measures action decisions rather than low-level control, and transferring a competent decision maker onto a physical humanoid remains future work. The interface likewise simulates no tactile channel: collisions displace the world but are never felt, so the agent can achieve proprioception only through its egocentric view—which may be intrinsically difficult, and a body-state or contact signal could be the missing input rather than a missing faculty. Finally, we evaluate frozen off-the-shelf VLMs; whether targeted training for body awareness can lift these ceilings is an open—and, we believe, tractable—question.

HumanCLAW-Bench leaves ample headroom: the strongest model completes the full task on only of episodes, and even navigation alone peaks at —headroom concentrated exactly where progress in embodied self-awareness will compound. We offer it as a clean testbed for the next generation of agents—agents that not only see and plan, but act.

References

• Anderson et al. (2018) Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
• Black et al. (2024) Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. : A vision-language-action flow model for general robot control, 2024. https://arxiv.org/abs/2410.24164.
• Chang et al. (2025) Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, and Tsung-Yen Yang. PARTNR: A benchmark for planning and reasoning in embodied multi-agent tasks. In The Thirteenth International Conference on Learning Representations, 2025. https://openreview.net/forum?id=T5QLRRHyL1.
• Chen et al. (2024) Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialVLM: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455–14465, 2024.
• Choi et al. (2024) Jae-Woo Choi, Youngwoo Yoon, Hyobin Ong, Jaehong Kim, and Minsu Jang. LoTa-Bench: Benchmarking language-oriented task planners for embodied agents. In The Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=ADSxCpCu9s.
• Cong et al. (2026) Xiaoyan Cong, Zekun Li, Zhiyang Dou, Hongyu Li, Omid Taheri, Chuan Guo, Abhay Mittal, Sizhe An, Taku Komura, Wojciech Matusik, et al. Umo: Unified in-context learning unlocks motion foundation model priors. arXiv preprint arXiv:2603.15975, 2026.
• Coumans (2015) Erwin Coumans. Bullet physics simulation. In ACM SIGGRAPH 2015 Courses, page 7:1. ACM, 2015. 10.1145/2776880.2792704.
• Diomataris et al. (2025) Markos Diomataris, Berat Mert Albaba, Giorgio Becherini, Partha Ghosh, Omid Taheri, and Michael J Black. Moving by looking: Towards vision-driven avatar motion generation. arXiv preprint arXiv:2509.19259, 2025.
• Driess et al. (2023) Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. PaLM-E: An embodied multimodal language model. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 8469–8488. PMLR, 2023. https://proceedings.mlr.press/v202/driess23a.html.
• Duan et al. (2024) Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate-anything: Automating real-world robots using vision-language models. In Conference on Robot Learning, 2024.
• Ghosh et al. (2024) Dibya Ghosh, Homer Rich Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, Jianlan Luo, You Liang Tan, Lawrence Yunliang Chen, Quan Vuong, Ted Xiao, Pannag R. Sanketi, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024. 10.15607/RSS.2024.XX.090.
• Guo et al. (2022) Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152–5161, 2022. 10.1109/CVPR52688.2022.00509.
• Guo et al. (2024) Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. MoMask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900–1910, 2024. https://openaccess.thecvf.com/content/CVPR2024/html/Guo_MoMask_Generative_Masked_Modeling_of_3D_Human_Motions_CVPR_2024_paper.html.
• Hassan et al. (2021) Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J. Black. Stochastic scene-aware motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11354–11364, 2021. 10.1109/ICCV48922.2021.01118.
• Hong et al. (2026) Yining Hong, Jiageng Liu, Han Yin, Manling Li, Leonidas Guibas, Li Fei-Fei, Jiajun Wu, and Yejin Choi. ESI-Bench: Towards embodied spatial intelligence that closes the perception-action loop. arXiv preprint arXiv:2605.18746, 2026.
• Huang et al. (2023a) Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. VoxPoser: Composable 3d value maps for robotic manipulation with language models. In Proceedings of the 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 540–562. PMLR, 2023a. https://proceedings.mlr.press/v229/huang23b.html.
• Huang et al. (2023b) Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Proceedings of the 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1769–1782. PMLR, 2023b. https://proceedings.mlr.press/v205/huang23c.html.
• Huang et al. (2024) Yiming Huang, Weilin Wan, Yue Yang, Chris Callison-Burch, Mark Yatskar, and Lingjie Liu. Como: Controllable motion generation through language guided pose code editing. In European Conference on Computer Vision, pages 180–196. Springer, 2024.
• Ichter et al. (2023) Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Proceedings of the 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 287–318. PMLR, 2023. https://proceedings.mlr.press/v205/ichter23a.html.
• Jia et al. (2026) Wenqi Jia, Zekun Li, Abhay Mittal, Chengcheng Tang, Chuan Guo, Lezi Wang, James Matthew Rehg, Lingling Tao, and Size An. Iam: Identity-aware human motion and shape joint generation. arXiv preprint arXiv:2604.25164, 2026.
• Khanna et al. (2024) Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat synthetic scenes dataset (HSSD-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16384–16393, 2024. 10.1109/CVPR52733.2024.01550.
• Kim et al. (2025) Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P. Foster, Pannag R. Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. OpenVLA: An open-source vision-language-action model. In Proceedings of the 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 2679–2713. PMLR, 2025. https://proceedings.mlr.press/v270/kim25c.html.
• Lee et al. (2025) Jason Lee, Jiafei Duan, Haoquan Fang, Yuquan Deng, Shuo Liu, Boyang Han, Mohammadreza Salehi, Jae Sung Hwang, et al. MolmoAct: Action reasoning models that can reason in space. arXiv preprint arXiv:2508.07917, 2025.
• Li et al. (2023) Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. BEHAVIOR-1K: A benchmark for embodied AI with 1,000 everyday activities and realistic simulation. In Proceedings of the 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 80–93. PMLR, 2023. https://proceedings.mlr.press/v205/li23a.html.
• Li et al. (2025) Linjie Li, Mahtab Bigverdi, Jiawei Gu, Zixian Ma, Yinuo Yang, Ziang Li, Yejin Choi, and Ranjay Krishna. Unfolding spatial cognition: Evaluating multimodal models on visual simulations. arXiv preprint arXiv:2506.04633, 2025.
• Li et al. (2024) Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, and Jiajun Wu. Embodied agent interface: Benchmarking LLMs for embodied decision making. In Advances in Neural Information Processing Systems, volume 37, 2024. 10.52202/079017-3188. https://papers.nips.cc/paper_files/paper/2024/hash/b631da756d1573c24c9ba9c702fde5a9-Abstract-Datasets_and_Benchmarks_Track.html.
• Li et al. (2026a) Zekun Li, Sizhe An, Chengcheng Tang, Chuan Guo, Ivan Shugurov, Linguang Zhang, Amy Zhao, Srinath Sridhar, Lingling Tao, and Abhay Mittal. Llamo: Scaling pretrained language models for unified motion understanding and generation with continuous autoregressive tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2209–2220, June 2026a.
• Li et al. (2026b) Zekun Li, Rui Zhou, Rahul Sajnani, Xiaoyan Cong, Daniel Ritchie, and Srinath Sridhar. Genhsi: Controllable generation of human-scene interaction videos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 138–149, 2026b.
• Liang et al. (2023) Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. In IEEE International Conference on Robotics and Automation, 2023.
• Liang et al. (2025) Wenlong Liang, Rui Zhou, Yang Ma, Bing Zhang, Songlin Li, Yijia Liao, and Ping Kuang. Large model empowered embodied AI: A survey on decision-making and embodied learning, 2025. https://arxiv.org/abs/2508.10399.
• Liu et al. (2024) Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, et al. VisualAgentBench: Towards large multimodal models as visual foundation agents. arXiv preprint arXiv:2408.06327, 2024.
• Ma et al. (2026) Yueen Ma, Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. A survey on vision-language-action models for embodied AI. IEEE Transactions on Neural Networks and Learning Systems, 2026. 10.1109/TNNLS.2025.3650584.
• NVIDIA et al. (2025) NVIDIA, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, et al. GR00T N1: An open foundation model for generalist humanoid robots, 2025. https://arxiv.org/abs/2503.14734.
• Puig et al. (2018) Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. VirtualHome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494–8502, 2018. https://openaccess.thecvf.com/content_cvpr_2018/html/Puig_VirtualHome_Simulating_Household_CVPR_2018_paper.html.
• Puig et al. (2024) Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, Vladimir Vondrus, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars, and robots. In The Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=4znwzG92CE.
• Savva et al. (2019) Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A platform for embodied AI research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9339–9347, 2019. 10.1109/ICCV.2019.00943.
• Shridhar et al. (2020) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10740–10749, 2020. https://openaccess.thecvf.com/content_CVPR_2020/html/Shridhar_ALFRED_A_Benchmark_for_Interpreting_Grounded_Instructions_for_Everyday_Tasks_CVPR_2020_paper.html.
• Siyao et al. (2022) Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3D dance generation by actor-critic GPT with choreographic memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
• Siyao et al. (2023) Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando++: 3D dance GPT with choreographic memory. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45(12):14192–14207, 2023.
• Siyao et al. (2024) Li Siyao, Tianpei Gu, Zhitao Yang, Zhengyu Lin, Ziwei Liu, Henghui Ding, Lei Yang, and Chen Change Loy. Duolando: Follower GPT with off-policy reinforcement learning for dance accompaniment. In International Conference on Learning Representations (ICLR), 2024.
• Siyao et al. (2025) Li Siyao, Yao Feng, Omid Taheri, Chen Change Loy, and Michael J Black. Half-physics: Enabling kinematic 3d human model with physical interactions. arXiv preprint arXiv:2507.23778, 2025.
• Song et al. (2023) Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LLM-Planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998–3009, 2023. https://openaccess.thecvf.com/content/ICCV2023/html/Song_LLM-Planner_Few-Shot_Grounded_Planning_for_Embodied_Agents_with_Large_Language_ICCV_2023_paper.html.
• Szot et al. (2021) Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel X. Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems, volume 34, pages 251–266, 2021. https://proceedings.neurips.cc/paper/2021/hash/021bbc7ee20b71134d53e20206bd6feb-Abstract.html.
• Tan et al. (2024) Weihao Tan, Wentao Zhang, Xinrun Xu, Haochong Xia, Ziluo Ding, Boyu Li, Bohan Zhou, Junpeng Yue, Jiechuan Jiang, Yewen Li, et al. Cradle: Empowering foundation agents towards general computer control. arXiv preprint arXiv:2403.03186, 2024.
• Tevet et al. (2023) Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H. Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Representations, 2023. https://openreview.net/forum?id=SJ1kSyO2jwu.
• Wang et al. (2024) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. https://openreview.net/forum?id=ehfRiF0R3a.
• Wang et al. (2023) Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with LLMs enables open-world multi-task agents. In Advances in Neural Information Processing Systems, volume 36, 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/6b8dfb8c0c12e6fafc6c256cb08a5ca7-Abstract-Conference.html.
• Wu et al. (2025) Qingxuan Wu, Zhiyang Dou, Chuan Guo, Yiming Huang, Qiao Feng, Bing Zhou, Jian Wang, and Lingjie Liu. Text2interact: High-fidelity and diverse text-to-two-person interaction generation. arXiv preprint arXiv:2510.06504, 2025.
• Xie et al. (2024) Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. OmniControl: Control any joint at any time for human motion generation. In The Twelfth International Conference on Learning Representations, 2024. https://openreview.net/forum?id=gd0lAEtWso.
• Yang et al. (2024) Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces. arXiv preprint arXiv:2412.14171, 2024.
• Yang et al. (2025) Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. EmbodiedBench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 70576–70631. PMLR, 2025. https://proceedings.mlr.press/v267/yang25f.html.
• Yuan et al. (2023) Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. PhysDiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16010–16021, 2023. 10.1109/ICCV51070.2023.01467.
• Zhang et al. (2025a) Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, and Ofir Press. VideoGameBench: Can vision-language models complete popular video games? arXiv preprint arXiv:2505.18134, 2025a.
• Zhang et al. (2025b) Libo Zhang, Zekun Li, Tianyu Li, Zeyu Cao, Rui Xu, Xiaoxiao Long, Wenjia Wang, Jingbo Wang, Yuan Liu, Wenping Wang, et al. Egoreact: Egocentric video-driven 3d human reaction generation. arXiv preprint arXiv:2512.22808, 2025b.
• Zhang and Tang (2022) Yan Zhang and Siyu Tang. The wanderings of odysseus in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20481–20491, 2022. https://openaccess.thecvf.com/content/CVPR2022/html/Zhang_The_Wanderings_of_Odysseus_in_3D_Scenes_CVPR_2022_paper.html.
• Zhang et al. (2025c) Yan Zhang, Yao Feng, Alpár Cseke, Nitin Saini, Nathan Bajandas, Nicolas Heron, and Michael J. Black. PRIMAL: Physically reactive and interactive motor model for avatar learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12725–12736, 2025c. https://openaccess.thecvf.com/content/ICCV2025/html/Zhang_PRIMAL_Physically_Reactive_and_Interactive_Motor_Model_for_Avatar_Learning_ICCV_2025_paper.html.
• Zhao et al. (2023) Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14738–14749, 2023. https://openaccess.thecvf.com/content/ICCV2023/html/Zhao_Synthesizing_Diverse_Human_Motions_in_3D_Indoor_Scenes_ICCV_2023_paper.html.
• Zitkovich et al. (2023) Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Proceedings of the 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 2165–2183. PMLR, 2023. https://proceedings.mlr.press/v229/zitkovich23a.html.

\beginappendix

7 Skill-Specific Verifier Checks

The verifier applies a fixed, skill-triggered checklist. Its purpose is to counter long-context degradation: as the rollout history grows, the planner’s estimates of spatial distance and relation become unreliable, so each check is re-answered from the current egocentric view alone, through a compact, skill-specific prompt. The verifier runs only when one of the skills in Table˜6 is proposed; if a check fails, the proposal is replaced with a corrected skill from the same action pool. Skills whose precondition cannot be assessed from the egocentric view—e.g. stepping back, where the region behind the body is unobservable—carry no check.

媒体内容 · 前往原文查看

Trigger skill	Checks
walk forward	Is the path ahead clear; would the motion collide with a wall or furniture; does moving forward advance the current goal.
stop	Is the high-level goal actually complete; has the body arrived at, or made contact with, the target.
climb up	Is a stair step directly underfoot; is the body facing the staircase.
climb down	Same checks as climb up: a step directly underfoot and the body facing the staircase.
turn (before sitting)	Is the body already flush against the seat (zero distance); after the turn, would the seat lie directly behind the body.

8 Automated Root-Cause Attribution

All root-cause labels in Figure˜8(b–d) are assigned automatically—no human annotation is involved. A deterministic classifier replays each failed episode’s logs and evaluates, per decision step, the geodesic distance from the pelvis to the nearest target instance on the scene’s navigation mesh, pelvis–object mesh contacts, the target’s rendered semantic pixel count in the egocentric view, the action stream, and the model’s own stated visible state. Within each stage, the rules in Table˜7 are evaluated top to bottom and the first matching rule assigns the label, so every failed episode receives exactly one error type and the labeling is exactly reproducible from the logs.

媒体内容 · 前往原文查看

Error type	Rule (evaluated top to bottom; first match assigns the label)
Find failures — no step both renders the target with semantic pixels in the egocentric view and has the model acknowledge it
target in view but not seen	The target renders pixels at some step, but the model never acknowledges it in its visible-state output.
near but not in view	The pelvis comes within  m of a target instance, yet the target never renders pixels: the agent reaches the object without ever bringing it into view.
too far	The geodesic distance from the starting pose to the nearest target instance exceeds  m.
unaware jammed	The agent issues locomotion commands for consecutive steps while per-step body displacement stays below  m.
ineffective actions	A window of consecutive steps in which the agent keeps issuing movement or turning commands while the body remains confined within a  m box.
midpoint give up	The agent first cuts its geodesic distance to the target to below half the initial value, but then retreats until the episode ends at more than the initial distance.
ineffective exploration	All remaining cases in which the agent never brings the target into view.
Nav failures — the target is actively found, but the agent never both actively stops and brings the pelvis within  m of the target
unaware arrived/touch	The pelvis comes within  m of the target at some step, but the agent never actively stops: it keeps issuing navigation actions after arrival.
stop while far	The agent actively stops and declares arrival, but the pelvis has never come within  m of the target.
find too late	The target is first rendered only after step of the -step budget, leaving too few steps to reach it.
unaware jammed	The target is still in view ( pixels) at the final frame, and the trajectory contains consecutive locomotion steps with body displacement below  m: the agent is lodged against geometry yet keeps issuing movement commands.
ineffective actions	The target is in view at the final frame and the body moves freely (no jammed run), yet the agent’s actions never bring it within  m.
target hallucination	The target renders pixels at the final frame, but the model’s final visible-state still claims to see it.
lost target	The target is out of view at the final frame and the model no longer claims to see it: the agent turned away and never re-acquired it.
Interact failures — navigation succeeds (active stop within  m), but the sit fails
stand after sit	Some sit step achieves pelvis–seat mesh contact with the target, yet the agent continues acting and its pelvis is off the seat at the final frame.
sit on air	A sit is issued, but the pelvis never makes mesh contact with any surface throughout the episode.
sit wrong	A sit is issued and the pelvis ends resting on a different object, the floor, or a wall rather than the target seat.
stop but never sit	The agent reaches the seat and stops, but never issues a sit.