大语言模型的最新进展已使其能够从纯文本推理过渡到多模态系统。最初是通过视觉-语言模型（VLM）整合视觉感知能力，而最近则是在视觉-语言-动作（VLA）模型中生成机器人动作。然而，由于嵌入式机器人平台在算力、内存和功耗方面存在严格限制，并且需要满足实时控制要求，在这些平台上部署此类模型仍然是一项挑战。
在同步控制流水线中，当 VLA 模型进行推理时，机械臂处于空闲状态等待指令，这会导致振荡行为和延迟的修正。为了解决这个问题，异步推理可以通过将生成过程与执行过程分离，从而实现平滑且连续的运动。然而，要使其有效，端到端的推理延迟必须始终短于动作执行时长。因此，这一时间约束为模型的吞吐量设定了上限。
将 VLA 模型引入嵌入式平台并非模型压缩问题，而是一个复杂的系统工程难题，需要架构分解、延迟感知调度以及与硬件对齐的执行。应对这些挑战对于将多模态基础模型的最新进展转化为可实际部署的嵌入式机器人系统至关重要。
本指南介绍了恩智浦（NXP）在记录可靠的机器人数据集、微调 VLA 策略（ACT 和 SmolVLA）方面的实操最佳实践，并重点展示了 NXP i.MX 95 SoC 在优化后实现的实时性能。
🎥 数据集录制：真正重要的因素
高质量、一致的数据胜过“更多但杂乱”的数据。本节将宝贵的经验教训转化为具体的检查清单和模式。
在我们的案例中，我们为“将茶包放入马克杯”这一任务录制了一个数据集。
1) 一致性优先
- 固定摄像头：使用刚性支架以避免位姿漂移。如果在录制或评估过程中，由于机器人的振动或操作员重置环境导致一个或多个摄像头发生位移，您会观察到严重的精度损失。
- 受控照明：搭建环境时，尽可能对照明进行最大程度的控制（使用固定光源，并远离一天中光线会变化的阳光）。
- 强对比度：除非你的部署场景就是如此，否则应避免使用“白上加白”的训练数据。要最大化机械臂、操作对象与环境之间的对比度。
- 固定标定：确保对机器人和遥控操作员的标定参数进行备份，这样即使代码崩溃，也无需重新录制之前的数据片段。
- 不要作弊：不要使用模型在推理时无法获取的信息。在数据录制过程中，操作员很容易依赖对场景的直接视觉观察。然而，这会引入数据集中不存在的信息。数据集采集必须仅限于策略在运行时可以使用的相同摄像头输入。
2) 使用夹爪摄像头（强烈推荐）
从仅场景视角转向混合视角可以提高整体精度，但摄像头数量越多，延迟受到的影响就越大。因此，你必须找到合适的平衡点。在我们的案例中，使用3个摄像头达到了平衡：
| 顶部 | 夹爪 | 左侧 |
|---|---|---|
 |  |  |
| 整个场景的全局视图。 | 用于精确抓取和对齐的最近距离视图。 | 补充顶部视角，用于判断高度和深度。 |
我们强烈建议使用安装在夹爪上的摄像头。通过提供近距离、与任务相关的视角，它能持续提高精细操作任务的成功率。重要的是，它也是最能有效强制正确数据采集实践的摄像头，使操作员能够完全依赖机器人的感知，而不是直接观察场景。
安装夹爪摄像头时，我们建议使用魔术贴或应力释放导向装置固定线缆，以防止其遮挡视野或在运动过程中断开连接。
3) 改进抓取能力
简单的硬件调整，例如在夹爪上套热缩管，可以增加摩擦力、减少粗糙度、减少操作过程中的滑脱，并提高任务成功率（减少“接近成功”的片段），从而提升策略学习的稳定性。
4) 多样性与数据划分
在录制数据集时，你应该：
- 变化片段分布：将工作空间划分为起始位置聚类，每个聚类至少记录 10 个片段。通过改变物体位置和旋转角度来增加多样性。
例如，我们将机器人手臂的可达工作空间划分为 11 个聚类，每个聚类尺寸为 10 × 10 厘米。
- 区分训练集与验证集：策略很容易在训练集上过拟合，因此务必确保验证集是模型未曾见过的数据。
例如，我们从训练集中移除了第 6 个聚类。
- 尽可能多地记录动作：小型 VLA 模型在未见过的运动上泛化能力有限。因此，请记录覆盖更广泛自由度范围的片段。
例如，我们以水平或垂直方向抓取茶包。
- 预判失败情况：有时策略无法在第一次就接触到物体，需要“返回重试”。我们发现，让约 20% 的片段对应返回物体的情形，有助于模型提升整体成功率。
例如，我们训练集中约 20% 的片段是恢复片段。
这反映了 VLA 论文和社区指南中的最佳实践。以下是同一聚类内数据多样性的 3 个示例：
| 起始位置 1 | 起始位置 2 | 恢复片段 |
|---|---|---|
 |  |  |
起始位置 1 和 2 对应同一聚类内的不同位置。相比之下，在恢复片段中，机器人并非从“起始模式”开始；而是已经靠近杯子，并应直接从该位置取出茶包。
🎛️ 微调 VLA
我们的实际操作：
任务：“抓取茶包并将其放入杯子中。”
数据集：
- 120 个片段：10 个聚类 ×（10 个不同的茶包起始位置 + 2 个恢复片段）
- 3 个摄像头（640x480 像素，30 帧/秒）：顶部、夹爪、左侧
- 第 6 号聚类被移除用于验证
批次大小：8
训练：选择在 20 万步后验证损失最低的模型检查点
在训练集和验证集上，兼顾准确率、泛化能力和动作平滑度的最佳权衡范围，对于 ACT（每块 100 个动作）而言，出现在 10 万到 16 万训练步之间。对于 SMolVLA 训练（每块 50 个动作），这种权衡出现在更多训练步之后。我们发现，在模型开始过拟合的时间点之后继续训练一小段时间，往往能提升整体准确率。
经验法则：通过评估在训练集和验证集上的成功率来选择最终检查点，而不是根据训练损失。
⚡ 针对恩智浦 i.MX 95 应用处理器进行优化
i.MX 95 系统级芯片集成了 6 个 Arm Cortex‑A55、一个 Cortex‑M7 和一个 Cortex M33 微控制器、一个 Mali GPU、一个全新的恩智浦 ISP，以及 eIQ® Neutron NPU，旨在支持多摄像头和强大的 I/O 能力，实现高效、安全的边缘推理。[nxp.com]
1) 分而治之
我们不是将模型作为一个整体图来运行，而是将 VLA 图分解为多个逻辑阶段：编码器、解码器和动作专家。这使得每个组件都可以独立地进行优化、调度和部署。
在实践中，SmolVLA 被划分为以下几个子模块：
- 视觉模块：处理 RGB 摄像头帧并生成视觉嵌入向量。
- 大语言模型主干：根据视觉和文本嵌入向量生成动作 token。
- 动作专家：应用流匹配来迭代地对动作样本进行去噪，并输出最终的控制指令。
这种分离方式允许对每个模块进行单独优化。可以测量每个模块量化的影响，以选择延迟和准确率之间的最佳权衡。此外，将动作专家与 VLM 隔离，是使其以较低频率运行的理想方案。
2) 量化
为了优化 i.MX 95 系统级芯片上的推理性能，我们在不同模块上探索了几种量化技术。我们发现，对视觉编码器和 LLM 预填充阶段进行量化对准确率的影响有限，而对动作专家中的去噪流进行量化则会显著降低性能。
这种行为是符合预期的，因为量化误差会在迭代去噪步骤中累积。
这就是为什么我们决定将该模块保持较高精度以维持稳定性，而在其他模块上，我们根据层类型探索了从 8 位混合精度到 4 位量化的各种量化配置。
此外，我们对不同模块应用了内部优化。结果如下表所示，标记为“优化模型”。
3) 异步推理：基于控制感知的调度
在同步控制循环中，流水线按以下方式运行：
- 采集观测数据
- 运行完整模型推理
- 执行生成的动作
在第 (2) 步期间，机器人处于空闲状态。如果推理延迟不可忽略，则会产生以下问题：
- 运动中的空闲间隙
- 因过时观测数据导致的振荡性修正
- 有效控制频率降低
- 恢复行为表现不佳
采用异步推理后，动作生成与执行并行运行：
- 机器人执行当前的动作片段
- 同时计算下一个动作片段
这提高了有效控制频率，减少了观测数据的陈旧性，并改善了恢复行为。
在 i.MX 95 SoC 等嵌入式平台上，异步推理至关重要——但只有在推理延迟保持在动作时间预算之内时才有效：推理时间 < 执行时间
| 同步推理 | 异步推理 | |
|---|---|---|
| 每个片段的动作数 | 100 | 100 |
| FPS | 60 | 60 |
| 片段大小阈值 | 不适用 | 0.2 |
| 聚合函数 | 不适用 | 加权平均 |
| 动作队列演变 |  |  |
| 结果 |
📊 我们在 i.MX 95 应用处理器上取得的成果
设置
- 任务：“拿起茶包并将其放入马克杯中。”
- 测试集（20 个回合）：每个簇有 2 个随机位置。
- 验证集（10 个回合）：簇 n°6 中的所有 10 个位置
| 平台（CPU） | 策略 | 格式 | 推理延迟 | 准确率 测试集（20） | 准确率 验证集（10） | 全局准确率（30） |
|---|---|---|---|---|---|---|
| i.MX 95 | ACT | ONNX FP32 | 2.86 秒 | 1.00 | 0.90 | 0.96 |
| i.MX 95 | ACT | 优化版 | 0.32 秒 | 1.00 | 0.60 | 0.89 |
| i.MX 95 | SmolVLA | ONNX FP32 | 29.1 秒 | 0.50 | 0.40 | 0.47 |
⏩ 后续步骤
我们的近期目标是提高 SmolVLA（ONNX FP32）的任务准确率。我们已经建立了基线，并测得优化后的板载推理延迟为 6.15 秒。
下一阶段将专注于在我们的 NPU 上进行更深层次的优化。与此同时，我们计划从单任务设置转向更长周期、更复杂的场景。为此，我们将引入：
- 用于可扩展数据生成和基准测试的仿真环境
- 用于策略优化的强化学习（RL）
- 用于弥合领域差距并提升真实世界性能的仿真到现实迁移
目标是从单一经过验证的操作任务，转向一种可复现的方法论，用于在嵌入式机器人系统上部署 VLA 策略。
✅ 可供复用的检查清单
录制
- 固定支架已确认
- 相机对焦与照明良好
- 夹爪抓取良好
- 校准文件备份已保存
- 对比度已确认
训练
- 每 20k 步保存/评估检查点
- 同时保存你的训练参数，以便在需要时能够恢复训练
- 提前准备好你的验证集以及用于精度和延迟的追踪方法
在 i.MX 95 SoC 上部署
- 你对精度感到满意
- 联系我们以优化你的模型
📚 资源与灵感
- ACT 文档与论文（核心思想、动作分块、低演示样本成功率）。[huggingface.co]，[arxiv.org]
- SmolVLM/SmolVLA 系列与代码仓库（紧凑多模态 + VLA 设计）。[huggingface.co]，[github.com]，[smolvla.net]
- Sherry Chen 在 Hugging Face 博客上关于在 SO-101 上训练 ACT 的文章（实践经验、常见陷阱、修复方法）。[huggingface.co]
本文提及的模型
Recent advances in Large Language Models have enabled the transition from text-only reasoning to multimodal systems. First, with the integration of visual perception in Vision–Language Models (VLMs), and more recently with the generation of robot actions in Vision–Language–Action (VLA) models. Deploying these models on embedded robotic platforms remains a challenge due to tight constraints in terms of compute, memory, and power, as well as real-time control requirements.
In synchronous control pipelines, while the VLA is running inference, the arm is idle awaiting commands leading to oscillatory behavior and delayed corrections. To tackle that, asynchronous Inference can enable smooth and continuous motion by dissociating generation from execution. However, to be effective, the end-to-end inference latency must remain shorter than the action execution duration. This temporal constraint therefore sets an upper limit on the model's throughput.
Bringing VLA models to embedded platforms is not a matter of model compression, but a complex systems engineering problem requiring architectural decomposition, latency-aware scheduling, and hardware-aligned execution. Addressing these challenges is essential to translate recent advances in multimodal foundation models into practical and deployable embedded robotic systems.
This guide presents NXP’s hands‑on best practices for recording reliable robotic datasets, fine‑tuning VLA policies (ACT and SmolVLA), and hightlights the real-time performance that NXP i.MX 95 SoC achieves after optimization.
🎥 Dataset Recording: What Actually Matters
High‑quality, consistent data beats “more but messy” data. This section turns hard‑earned lessons into concrete checklists and schemas.
In our case, we recorded a dataset for the task: "Put the tea bag in the mug."
1) Consistency First
- Fixed cameras: Use rigid mounts to avoid pose drift. If during recording or evaluation one or more cameras shift because of the robot's vibrations or the operator resetting the environment, you can observe a severe accuracy loss.
- Controlled lighting: Set up your environment where you can have as much control as possible on lighting (Fixed light source(s) and far from sunlight that vary during the day).
- Strong contrast: Avoid training with “white on white” unless that’s your deployment domain. Maximize contrast between the arm, the object and the environment.
- Fixed calibration: Make sure to have backups of your robot and teleoperator calibrations so you don't have to re-record your previous episodes if the code crashes.
- Do not cheat: Do not use information the model will not have access to at inference time. During data recording, it is tempting for the operator to rely on direct visual observation of the scene. However, this introduces information that is absent from the dataset. Dataset collection must be restricted to the same camera inputs that will be available to the policy at runtime.
2) Use a Gripper Camera (Highly Recommended)
Moving from scene‑only views to mixed viewpoints increases the global accuracy, but the more cameras you have the more the latency is impacted. Therefore, you must choose right compromise. In our case that balance was reached with 3 cameras:
| Top | Gripper | Left |
|---|---|---|
| | |
| The global view of the whole scene. | The closest view for precise grasps and alignment. | Complement the top view for height and depth. |
We strongly recommend using a gripper-mounted camera. It consistently improves success rates on fine manipulation tasks by providing a close, task-relevant viewpoint. Importantly, it is also the camera that most effectively enforces correct data collection practices, allowing the operator to rely exclusively on the robot’s perception rather than observing the scene directly.
When installing a gripper camera, we recommend securing the cable with Velcro or a strain-relief guide to prevent it from obstructing the field of view or becoming disconnected during motion.
3) Improve Prehension
Simple hardware tweaks like heat‑shrink tubing over gripper claws increase friction, reduce roughness, reduce slippage during episodes, and increase task success rate (fewer “almost success” episodes), improving policy learning stability.
4) Diversity & Splits
When recording a dataset, you should:
- Vary episodes distribution: Divide your workspace into starting-position clusters, and record at least 10 episodes per cluster. Add diversity by changing the object position and rotation.
e.g. we partitioned the robot arm’s reachable workspace into 11 clusters, each measuring 10 × 10 cm.
- Differentiate training & validation sets: Policies can easily overfit on the training set, so make sure that the validation set is unseen by the model.
e.g. we removed cluster 6 from the training set.
- Record the most movements you can: Small VLA models exhibit limited generalization on unseen motion. Therefore, record episodes that cover the wider ranges of degrees of freedom.
e.g. we grasped the tea bag either in horizontal or vertical position.
- Anticipate failure: Sometimes the policy will not reach the object the first time and will have to "go back to it". We noticed that having 20% of all episodes that corresponds to the case of going back to the object help the model improve overall success rate.
e.g. around 20% of our training set corresponds to recovery episodes.
This mirrors best practices across VLA papers and community guides. Here are 3 examples of data diversity within the same cluster:
| Starting position 1 | Starting position 2 | Recovery episode |
|---|---|---|
| | |
Starting positions 1 and 2 correspond to different positions within the same cluster. In contrast, during the recovery episode, the robot does not begin in "starting mode"; but is instead already near the mug and should proceed directly to retrieve the tea bag from that location.
🎛️ Fine‑Tuning VLAs
What we did in practice:
Tasks: "Grab the tea bag and place it in the mug."
Dataset:
- 120 episodes: 10 clusters x (10 different tea bag starting positions + 2 recovery episodes)
- 3 cameras (640x480px, 30fps): Top, Gripper, Left
- Cluster n°6 was removed for validation
Batch size: 8
Training: Model checkpoint with the lowest validation loss after 200k steps was chosen
The range providing the best trade-off between accuracy, generalization, and motion smoothness across both the training and validation sets was found for ACT (100 actions per chunk) within a 100k-160k training steps. For SMolVLA training (50 actions per chunk), the trade‑off appears after many more training steps. We found that continuing training slightly past the point where the model begins to overfit tends to improve overall accuracy.
Rule of thumb: choose final checkpoint by evaluating success on both training and validation set, not by training loss.
⚡ Optimizing for the NXP i.MX 95 Applications processor
The i.MX 95 SoC integrates 6× Arm Cortex‑A55, a Cortex‑M7 and a Cortex M33 MCU, a Mali GPU, a new NXP ISP, and the eIQ® Neutron NPU, targeting efficient, secure edge inference with multi‑camera support and strong I/O. [nxp.com]
1) Divide And Conquer
Instead of running the models as one monolithic graph, we decompose the VLA graph into logical stages: encoders, decoders, and action experts. Therefore, allowing each component to be optimized, scheduled, and deployed independently.
In practice, SmolVLA is partitioned into the following sub-blocks:
- Vision: processes RGB camera frames and produces visual embeddings.
- LLM backbone: generates actions tokens from visual and textual embeddings.
- Action expert: applies flow matching to iteratively denoise action samples and outputs final control commands.
This separation allows per-block optimizations. The impact of each block quantization can be measured to choose the best tradeoff between latency and accuracy. Also, isolating the action expert from the VLM was ideal to run it at lower frequency.
2) Quantization
In order to optimize the inference for i.MX 95 SoC, we explored several quantization techniques on different blocks. We found that quantizing the vision encoder and LLM prefill had limited impact on accuracy, whereas quantization of the denoising flow in the action expert significantly degrades performance.
This behaviour is expected, as quantization errors are accumulating across iterative denoising steps.
That is why we decided to keep this block at higher precision to preserve stability, while on the other blocks, we explored various quantization configurations, from 8-bit mixed precision to 4-bit quantization, depending on the layers.
In addition, we applied in-house optimization on the different blocks. Results are shown in the below table, referred as optimized models.
3) Asynchronous Inference: Control-Aware Scheduling
In a synchronous control loop, the pipeline operates as:
- Capture observation
- Run full model inference
- Execute generated action
During step (2), the robot remains idle. If inference latency is non-negligible, this produces:
- Idle gaps in motion
- Oscillatory corrections due to stale observations
- Reduced effective control frequency
- Poor recovery behavior
With Asynchronous Inference, action generation runs in parallel with execution:
- The robot executes the current action chunk
- The next chunk is computed simultaneously
This increases effective control frequency, reduces observation staleness, and improves recovery behavior.
On embedded platforms such as the i.MX 95 SoC, asynchronous inference is essential — but only effective if inference latency is kept under the action horizon budget: inference time < execution time
| Synchronous inference | Asynchronous inference | |
|---|---|---|
| Actions per chunk | 100 | 100 |
| FPS | 60 | 60 |
| Chunk size threshold | N/A | 0.2 |
| Aggregate function | N/A | weighted_average |
| Action queue evolution | | |
| Results |
📊 What We Achieve on i.MX 95 Applications Processor
Setup
- Tasks: "Grab the tea bag and place it in the mug."
- Test set (20 episodes): 2 random positions for each cluster.
- Validation set (10 episodes): all 10 positions in cluster n°6
| Platform (CPU) | Policy | Format | Inference Latency | Accuracy Test Set (20) | Accuracy Validation Set (10) | Global Accuracy (30) |
|---|---|---|---|---|---|---|
| i.MX 95 | ACT | ONNX FP32 | 2.86 s | 1.00 | 0.90 | 0.96 |
| i.MX 95 | ACT | Optimized | 0.32 s | 1.00 | 0.60 | 0.89 |
| i.MX 95 | SmolVLA | ONNX FP32 | 29.1 s | 0.50 | 0.40 | 0.47 |
⏩ Next Steps
Our immediate objective is to improve task accuracy with SmolVLA (ONNX FP32). We have already established a baseline and measured an optimized on-board inference latency of 6.15 s.
The next phase will focus on deeper optimizations on our NPUs. In parallel, we aim to move from single-task setup toward longer-horizon and more complex scenarios. To do that, we will introduce:
- Simulation environments for scalable data generation and benchmarking
- Reinforcement Learning (RL) for policy refinement
- Sim-to-Real transfer to bridge domain gaps and improve real-world performance
The goal is to move from a single validated manipulation task toward a reproducible methodology for deploying VLA policies on embedded robotic systems.
✅ Checklists You Can Reuse
Recording
- Fixed mounts verified
- Good cameras focus and illumination
- Good gripper claws prehension
- Calibration files backups saved
- Contrast validated
Training
- Save/eval checkpoints every 20k steps
- Save also your training parameters to be able to resume training if needed
- Prepare in advance your validation set and your tracking method for accuracy and latency
Deployment on i.MX 95 SoC
- You are satisfied with your accuracy
- Contact us to have your model optimized
📚 Resources & Inspiration
- ACT documentation & paper (core idea, action chunking, low‑demo success). [huggingface.co], [arxiv.org]
- SmolVLM/SmolVLA family & repos (compact multimodal + VLA design). [huggingface.co], [github.com], [smolvla.net]
- Sherry Chen’s HF blog on training ACT on SO‑101 (practical lessons, pitfalls, fixes). [huggingface.co]