摘要——我们将 RLVE 框架从单轮推理谜题扩展至多轮、工具增强的电商对话。EcomRLVE-GYM 提供了 8 个可验证环境——商品发现、替代推荐、购物车构建、退货、订单追踪、政策问答、套餐规划以及多意图流程——每个环境都包含程序化问题生成、12 轴难度课程体系以及算法可验证的奖励机制。我们使用 DAPO 算法对 Qwen 3 8B 模型进行了 300 步训练，并展示了初步结果，表明环境规模扩展和自适应难度能够迁移至智能体化的真实世界任务完成场景。
本项目源自 PyTorch OpenEnv 黑客松，目前仍在持续演进中，欢迎关注我们获取最新动态 🔥
为什么购物智能体需要强化学习？
大语言模型能够进行流畅对话，但将其部署为购物助手时，一个长期存在的差距便暴露无遗：流畅 ≠ 任务完成。当顾客提出"帮我找一款 25 美元以下、两天内发货的 USB-C 充电器"时，智能体需要调用正确的商品目录搜索，对三个硬性约束进行过滤，避免幻觉出从未检索到的产品 ID，并在首选结果缺货时妥善处理后续跟进。
监督微调可以通过演示数据教会智能体表层工具使用，但无法扩展到真实电商所需的组合级约束配置空间、信息不完整对话以及多步骤交易工作流。
基于可验证奖励的强化学习（RLVR）提供了一种替代方案：智能体以结果为导向进行优化——商品是否满足约束条件？购物车是否正确？退货是否针对正确的订单行发起？挑战在于构建既具备可验证性（无需 LLM 作为评判者的主观性）又具备自适应性（难度随策略能力增长而提升）的奖励函数。
从 RLVE-Gym 到 EcomRLVE-GYM
RLVE-Gym 提供了 400 个环境，涵盖排序、乘法、数独及其他算法推理任务；然而，这些均为单轮、文本输入/文本输出的谜题——将其扩展至智能体领域被留作未来工作。
EcomRLVE-GYM 填补了这一空白：我们保持在可验证的范式内（电商结果可以通过算法检查），同时扩展到多轮、工具增强的智能体对话——在这些环境中，智能体必须执行操作（调用工具、修改世界状态），而不仅仅是进行推理（生成文本答案），从而弥补了搜索系统的不足。
EcomRLVE-GYM 将客服结果转化为结构上可验证的形式：
上述每一个信号都可以通过一个能够访问隐藏真实目标的程序进行评估。无需人工标注或大语言模型作为裁判。
一次训练回合的样貌
在解释框架之前，先来看一个难度 d = 4 的 EcomRLVE 回合示例。环境会生成一个隐藏目标，一个模拟用户开启对话，智能体必须使用工具来满足请求。每一步操作都通过算法验证——无需大语言模型裁判。
奖励完全由代码计算得出：基于（产品、变体、数量）元组的 F1 分数、在更少轮次内完成任务的效率奖励，以及一项幻觉检查——确保每个推荐的产品 ID 确实被检索过。如果智能体选择了 Lightning 接口变体而非 USB-C，模拟用户会在对话中途纠正——而 F1 分数也会随之下降。
八个环境
每个环境都涵盖一个不同的真实购物场景。智能体必须使用工具（目录搜索、购物车操作、订单查询、政策咨询）完成任务，并由程序——而非人类或其他大语言模型——进行评分。
| 环境 | 智能体必须完成的任务 |
|---|---|
| 产品发现 | 找到满足用户所有约束条件的产品 |
| 替代方案 | 某件商品缺货——寻找相似、兼容的替代品 |
| 购物车构建 | 添加用户要求的确切产品、变体和数量 |
| 退货与换货 | 识别正确的订单行，发起退货，建议换货方案 |
| 订单追踪 | 确定用户指的是哪个订单，并报告其当前状态 |
| 政策问答 | 回答关于商店政策的确定性提问（退货期限、配送规则等） |
| 套餐规划 | 在预算范围内推荐一个完整的项目购物清单 |
| 多意图旅程 | 处理一个按顺序串联 2–5 个上述任务的对话。 |
每个环境都使用相同的三部分奖励信号：
- 任务奖励——智能体是否实际完成了目标？（例如，是否推荐了正确的产品、购物车是否正确、是否追踪了正确的订单？）
- 效率奖励——智能体是否在没有浪费轮次的情况下完成了任务？用户引起的轮次（提出后续问题、确认操作）不计入智能体的消耗——只有智能体错误引起的轮次才会计入。
- 幻觉惩罚——智能体是否只推荐了它在当前会话中实际检索到的产品？推荐从未被查询过的产品 ID 会受到惩罚，因此智能体不能凭记忆凭空捏造结果。
无效输出（格式错误的 JSON、非法工具调用）会立即触发失败评分，从而从一开始就强烈激励智能体生成格式正确的响应。
自适应难度课程
一个单一的难度数值 d 同时控制任务的 12 个独立方面。这一点很重要，因为电商对话在多个不同维度上同时存在困难——而不仅仅是在一个维度上。
以下是四个有代表性的难度轴：
| 变化内容 | 简单（d = 0） | 中等（d = 6） | 困难（d = 12） |
|---|---|---|---|
| 用户有多少个约束条件 | 2 | 5 | 8 |
| 用户遗漏约束条件的频率 | 5% | 70% | ~80% |
| 搜索结果中干扰项的比例 | 0% | 12% | 24% |
| 对话中途缺货的商品 | 0% | 30% | 50% |
其他八个轴涵盖轮次预算、输入噪声（拼写错误、俚语）、上下文切换、检索深度、订单历史大小、策略复杂度和工具预算。完整细分见技术报告。
自适应调度。每个环境独立追踪智能体的成功率，并且仅在智能体可靠地通过当前难度级别后，才推进到更困难的问题。这使得每个环境都在智能体的能力边界上进行训练——既避免了“太简单而学不到东西”，也避免了“太难而无法取得进展”。
深度解析：购物车构建（E_CART）
购物车构建是一个很好的示例，因为它需要完整的搜索→检查→澄清→操作循环，具有二元真实答案，并且引入了一个大多数推荐基准测试所没有的挑战：变体选择。
要成功完成任务，智能体必须掌握五项不同的技能：
| 技能 | 实际含义 |
|---|---|
| 商品发现 | 通过构造良好的查询语句搜索商品目录，找到正确的商品 |
| 规格选择 | 识别正确的颜色、尺寸或接口类型——而不仅仅是找到正确的商品 |
| 购物车管理 | 按照用户要求的确切规格和数量将商品加入购物车 |
| 澄清对话 | 当用户请求不明确时（例如缺少尺码），向用户提出有针对性的追问 |
| 多商品订单 | 在单次对话中处理包含多种不同商品的购物清单 |
智能体使用六种工具来完成这一任务：
| 工具 | 功能说明 |
|---|---|
| catalog_search | 使用自然语言查询搜索商品目录 |
| catalog_get_variants | 返回某商品可用的规格选项（颜色、尺寸、接口等） |
| cart_add | 将指定规格和数量的商品加入购物车 |
| cart_view | 读取当前购物车内容，以便智能体验证其是否与请求一致 |
| user_get_visit_history | 获取用户最近浏览过的商品 |
| ask_user | 当缺少细节时向客户发送澄清问题 |
问题
生成器会采样 1–5 个目标商品（难度随 d 值递增），每个商品可能要求特定的规格（USB-C 与 Lightning 对比，哑光与亮面对比）以及大于 1 的数量。智能体必须：
- 搜索商品目录以找到每个商品
- 调用 catalog.get_variants 查看可用选项
- 将正确的 (product_id, variant_id, qty) 元组加入购物车
为什么规格很重要
真实商品目录的规格数据是稀疏的——许多商品没有规格，而有规格的商品通常也只按颜色或尺寸变化。为了创建更丰富的区分任务，我们在每个回合初始化时合成规格数据：
- 按类别优先级列表选取最自然的属性进行变化（电子产品 → 接口类型；服装 → 尺码；厨房用品 → 材质）。
- 对于每个目标商品，我们生成 3 种规格：1 个目标规格 + 2 个合理的干扰项。例如，"Anker 65W USB-C 充电器"会生成 {USB-C, Lightning, HDMI}。
- 验证器会检查复合键 (product_id, variant_id)——商品正确但规格错误意味着该单元不匹配。
难度缩放
| 维度 | d = 0 | d = 3 | d = 6 | d = 9 |
|---|---|---|---|---|
| 不同商品数量 | 1 | 2 | 3 | 4 |
| 需要指定规格 | 21% | 66% | 93% | 99% |
| 多数量 | 0% | 30% | 50% | 50% |
在 d=0 时，智能体添加单个无变体复杂度的商品——学习基本的 catalog.search → cart.add 工作流程。在 d=6 时，它需要处理 3 件商品，几乎每件都需要特定变体，其中一半数量大于 1。
评分
购物车必须完全正确——正确的商品、正确的变体、正确的数量。部分正确的购物车可获得部分分数，但满分要求每件商品都匹配。如果智能体添加了错误的变体，模拟用户会在对话中途纠正（"那是 Lightning 版本，但我需要 USB-C"），让智能体有机会在回合结束前自行修正。
轨迹：简单 vs. 困难
来自 Qwen 3 8B 智能体的两个真实 E_CART 回合。相同环境，相同智能体——仅凭难度就改变了局面。
| d = 1——1 件商品，无变体 | d = 8——3 件商品，含变体 + 拼写错误 | |
|---|---|---|
| 用户消息 | "购物清单：SonicWave Keypad。请全部添加。" | "请将以下商品添加到我的购物车：Eco-Frienly HrenLefa Charger by GreenLeaf、High-Performance AqaPure Air Fryer、Lighrweiht PixelVault Tank Top (3)。"——来自噪声轴的拼写错误 |
| 第 1 轮 | user.get_visit_history() → 4 条结果。立即锁定目标。 | user.get_visit_history() → 9 条结果。必须在充电器、空气炸锅、上衣之间进行消歧。 |
| 第 2 轮 | cart.add("syn_000162", qty=1) → 购物车：1 件商品，$2.74。"我找到了 SonicWave Keypad。" 用户："太好了，就是这个。" | 无工具调用——展示 3 张商品卡片。用户："请添加 100W 输出的充电器、带活性炭过滤器的空气炸锅、3 件 XS 码的上衣。" |
| 第 3 轮 | answer: { "done": true }——3 轮完成。 | catalog.get_variants × 3 → 充电器：6 种变体（100W、120W、30W……），空气炸锅：6 种（竹炭、永久、活性炭……），上衣：7 种（XS、S、XL……）。用户："我需要 3 件 XS 码的上衣。" |
| 第 4 轮 | cart.add(charger, "100W", qty=1) ✓ cart.add(fryer, "Bamboo", qty=1) ✗ 应为 Charcoal。cart.add(tank_top, "size_XL", qty=3) ✗ 应为 XS。用户："空气炸锅是 Bamboo 不是 Charcoal，上衣是 XL 不是 XS。" | |
| 第 5 轮 | 重新获取上衣变体，添加 2 件 XS——未修正空气炸锅。用户："空气炸锅仍显示 Bamboo。我需要 3 件 XS，不是 2 件。" | |
| 第 6 轮 | 再次重新获取上衣变体——仍然忽略空气炸锅。用户："请更改空气炸锅的变体。" | |
| 第 7 轮 | “我找不到木炭滤网款。”——其实有；智能体在第3轮看到了，但从未选择它。用户：“那把空气炸锅去掉吧。” | |
| 第8轮 | 答案：{ “done”: true } ——购物车仍然错误：充电器 ✓，空气炸锅（竹制）✗，背心（3×XL + 2×XS）✗ | |
| 奖励 | r_task = +1.00，r_eff = +0.33，r_hall = 0.00，r_total = +0.80 ✓ | r_task ≈ 0.00，r_eff = −0.43，r_hall = 0.00，r_total = −0.06 ✗ |
| 结果 | 购物车与目标匹配。3轮，2次有效操作。 | 变体错误、数量错误，用户放弃。8轮，6次有效操作。 |
在d=1时，智能体用3轮干净利落地完成任务。在d=8时，它开始失控——选了竹制而非木炭款，选了XL而非XS，用户两次纠正后仍未修复空气炸锅，然后幻觉说该变体不存在。这正是难度课程所暴露的那种多步错误级联，而自适应训练应该教会智能体从中恢复。
用户模拟
一个可验证的环境需要一个行为逼真的用户模拟器。我们使用Qwen3.5（9.7B）来生成自然、多样化的用户消息，而非固定模板——涵盖从充满拼写错误的请求到对话中途切换话题的各种情况。
两个设计选择对训练质量至关重要：
偏好与已声明的约束相匹配。每个模拟用户都有一组隐藏的偏好（价格敏感度、品牌忠诚度、配送速度等）。这些偏好有意偏向用户所传达的任何约束——因此，如果用户说“25美元以下”，奖励函数实际上会关注价格。如果没有这一点，智能体可能会因为正确遵循用户指令而受到惩罚。
策略性省略。大语言模型故意在开场消息中省略一些约束，以迫使智能体提出澄清性问题。系统精确追踪哪些信息被提及、哪些未被提及，因此智能体永远不会因从未获得的信息而受到惩罚。
环境扩展
遵循RLVE的方法论，我们定义了嵌套的环境集合：
C1 ⊂ C2 ⊂ C4 ⊂ C8
| 集合 | 环境 | 训练的技能 |
|---|---|---|
| C1 | 购物车 | 搜索查询构建、购物车操作 |
| C2 | + 替换 | 约束条件下的相似性推理 |
| C4 | + 产品发现、退货 | 事务性工作流（检索 + 推荐、退货发起） |
| C8 | + 状态、策略、捆绑包、旅程 | 知识检索、规划、组合性 |
我们假设——与 RLVE 的发现一致——C8 智能体在单一环境专精模型自己的任务上，表现也优于后者。
早期结果
我们使用 DAPO 算法对 Qwen 3 8B 在 C1（购物车构建）上训练了 300 步，作为初步可行性研究。
| 配置 | |
|---|---|
| 基础模型 | Qwen 3 8B |
| 算法 | DAPO（G = 8 次 rollout/提示词） |
| 学习率 | 1e-5 |
| 商品目录 | 200 万件商品，使用 Alibaba-NLP/gte-modernbert-base（768 维）的 FAISS 索引 |
| 用户模拟 | Qwen3.5 9.7B |
我们观察到所达到的难度逐步增长，这证实了自适应调度能够产生稳定的学习信号，而非 RLVE 论文所预测的饱和（静态低难度）或饥饿（静态高难度）模式。
亲自尝试
使用下方嵌入的演示，直接在浏览器中运行一个实时回合。以下是入门方法：
- 从下拉菜单中选择一个环境（例如，E_CART 用于购物车构建，E_PD 用于产品发现）。
- 设置难度——0 是简单的单约束任务；6+ 会引入信息缺失、噪声检索和变体选择。
- 点击“重置回合”——模拟用户将带着购物请求开始。
- 你现在就是智能体：进行工具调用、分析输出，并提交最终的产品 ID 列表。
- 在两次运行之间点击“重置回合”，以开始一个新的场景。
资源
环境、验证器和训练配置均已开源：
git clone https://github.com/owlgebra-ai/EcomRLVE-Gym
cd EcomRLVE-Gym
pip install -e .
200 万件商品的目录已在 Hub 上：
from datasets import load_dataset
catalog = load_dataset("owlgebra-ai/Amazebay-catalog-2M", split="train")
print(f"{len(catalog)} products loaded")
参考文献
Zeng, Z., Ivison, H., Wang, Y., 等. (2025). RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments. ICML 2025. arXiv:2511.07317
Yu, Q., Zhang, Z., Zhu, R., 等. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476
Shao, Z., Wang, P., Zhu, Q., 等. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300
DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing Reasoning in LLMs through Reinforcement Learning. Nature.
Meta AI. (2024). Llama 3.1: A Foundation Model for General Intelligence. llama.meta.com
Qwen Team. (2025). Qwen3 Technical Report. arXiv:2505.09388
TL;DR— We extend the RLVE framework from single-turn reasoning puzzles to multi-turn, tool-augmented e-commerce conversations. EcomRLVE-GYM provides 8 verifiable environments — product discovery, substitution, cart building, returns, order tracking, policy QA, bundle planning, and multi-intent journeys — each with procedural problem generation, a 12-axis difficulty curriculum, and algorithmically verifiable rewards. We train a Qwen 3 8B model with DAPO over 300 steps and present early results demonstrating that environment scaling and adaptive difficulty transfer to agentic, real-world task completion.
This project originated in the Pytorch OpenEnv Hackathon and is still evolving, follow us for updates 🔥
Why RL for shopping agents?
Large language models can hold fluent conversations, yet deploying them as shopping assistants reveals a persistent gap: fluency ≠ task completion. A customer who asks "find me a USB-C charger under $25 that ships in two days" needs an agent that invokes the right catalog search, filters on three hard constraints, avoids hallucinating product IDs it never retrieved, and handles follow-ups when the top result goes out of stock.
Supervised fine-tuning can teach surface-level tool use from demonstrations, but it cannot scale to the combinatorial space of constraint configurations, partial-information dialogues, and multi-step transactional workflows that real e-commerce demands.
Reinforcement learning with verifiable rewards (RLVR) offers an alternative: the agent optimises for outcomes — did the products satisfy the constraints? Was the cart correct? Was the return initiated for the right order line? The challenge is constructing reward functions that are both verifiable (no LLM-as-a-judge subjectivity) and adaptive (difficulty that grows with the policy's capability).
From RLVE-Gym to EcomRLVE-GYM
RLVE-Gym provides 400 environments for sorting, multiplication, Sudoku, and other algorithmic-reasoning tasks; however, those are all single-turn, text-in / text-out puzzles — extending to agentic domains was left as future work.
EcomRLVE-GYM fills that gap: we stay in the verifiable regime (e-commerce outcomes can be checked algorithmically) while extending to multi-turn, tool-augmented, agentic conversations — environments where the agent must act (call tools, modify world state) rather than merely reason (produce a text answer) and compensates for the deficiency of the search system.
EcomRLVE-GYM transforms customer-service outcomes structurally verifiable:
Every signal above can be evaluated by a program with access to the hidden ground-truth goal. No human annotation or LLM-as-a-judge is needed.
What a training episode looks like
Before we explain the framework, here is what a single EcomRLVE episode looks like at difficulty d = 4. The environment generates a hidden goal, a simulated user opens the chat, and the agent must use tools to satisfy the request. Every action is verified algorithmically — no LLM judge required.
The reward is fully computed by code: F1 over (product, variant, qty) tuples, an efficiency bonus for finishing in fewer turns, and a hallucination check that every recommended product ID was actually retrieved. If the agent had picked the Lightning variant instead of USB-C, the simulated user would have corrected it mid-dialogue — and the F1 would have dropped.
The eight environments
Each environment covers a distinct real-world shopping scenario. The agent must complete the task using tools (catalog search, cart operations, order lookups, policy queries) and is scored by a program — not a human or another LLM.
| Environment | What the agent must do |
|---|---|
| Product Discovery | Find products that satisfy all the user's constraints |
| Substitution | An item is out of stock — find a similar, compatible alternative |
| Cart Building | Add the exact products, variants, and quantities the user asked for |
| Return + Replacement | Identify the right order line, initiate a return, suggest a replacement |
| Order Tracking | Resolve which order the user means and report its current status |
| Policy QA | Answer a deterministic question about store policy (return window, shipping rules, etc.) |
| Bundle Planning | Recommend a complete shopping list for a project within a budget |
| Multi-Intent Journey | Handle a conversation that chains 2–5 of the above tasks in sequence |
Every environment uses the same three-part reward signal:
- Task reward — did the agent actually complete the goal? (e.g., were the right products recommended, was the cart correct, was the right order tracked?)
- Efficiency reward — did the agent complete it without wasting turns? Turns the user caused (asking a follow-up, confirming an action) don't count against the agent — only turns caused by agent mistakes do.
- Hallucination penalty — did the agent only recommend products it actually retrieved during the session? Recommending product IDs that were never looked up is penalised, so the agent cannot invent results from memory.
Invalid outputs (malformed JSON, illegal tool calls) trigger an immediate failure score, creating a strong incentive for well-formed responses from step one.
Adaptive difficulty curriculum
A single difficulty number d controls 12 independent aspects of a task simultaneously. This is important because e-commerce conversations are hard in many different ways at once — not just along one dimension.
Here are four representative difficulty axes:
| What changes | Easy (d = 0) | Medium (d = 6) | Hard (d = 12) |
|---|---|---|---|
| How many constraints the user has | 2 | 5 | 8 |
| How often the user omits a constraint | 5% | 70% | ~80% |
| Fraction of search results that are distractors | 0% | 12% | 24% |
| Items that go out of stock mid-conversation | 0% | 30% | 50% |
The other eight axes cover turn budget, input noise (typos, slang), context switches, retrieval depth, order-history size, policy complexity, and tool budget. The full breakdown is in the technical report.
Adaptive scheduling. Each environment tracks the agent's success rate independently and only advances to harder problems once the agent is passing the current level reliably. This keeps every environment training at the agent's capability frontier — avoiding both "too easy to learn from" and "too hard to make progress on".
Deep dive: Cart Building (E_CART)
Cart building is a good showcase because it requires the full search → inspect → clarify → act loop, has a binary ground truth, and introduces a challenge absent from most recommendation benchmarks: variant selection.
To succeed, the agent must develop five distinct skills:
| Skill | What it means in practice |
|---|---|
| Product Discovery | Search the catalog with well-formed queries to find the right items |
| Variant Selection | Identify the correct color, size, or connector type — not just the right product |
| Cart Management | Add items with the exact variant and quantity the user asked for |
| Clarification Dialogue | Ask the user a focused follow-up when a request is ambiguous (e.g., missing size) |
| Multi-Item Orders | Handle shopping lists with several different products in a single conversation |
The agent uses six tools to accomplish this:
| Tool | What it does |
|---|---|
catalog_search | Searches the product catalog with a natural-language query |
catalog_get_variants | Returns available variants (color, size, connector, etc.) for a product |
cart_add | Adds a product to the cart with a specific variant and quantity |
cart_view | Reads the current cart so the agent can verify it matches the request |
user_get_visit_history | Fetches recently viewed products by user |
ask_user | Sends a clarification question to the customer when a detail is missing |
The problem
The generator samples 1–5 target products (scaling in difficulty with d), each potentially requiring a specific variant (USB-C vs Lightning, Matte vs Glossy) and a quantity > 1. The agent must:
- Search the catalog to find each product
- Call
catalog.get_variantsto see available options - Add the correct
(product_id, variant_id, qty)tuples to the cart
Why variants matter
Real product catalogs have sparse variant data — many products have none, and those that do typically vary only by colour or size. To create a richer discrimination task, we synthesize variants at episode initialization:
- A per-category priority list picks the most natural attribute to vary (electronics →
connector_type; clothing →size; kitchen →material). - For each target product, we generate 3 variants: 1 target + 2 plausible distractors. An "Anker 65W USB-C Charger" produces
{USB-C, Lightning, HDMI}. - The verifier checks composite keys
(product_id, variant_id)— correct product but wrong variant means the unit is unmatched.
Difficulty scaling
| Axis | d = 0 | d = 3 | d = 6 | d = 9 |
|---|---|---|---|---|
| Distinct items | 1 | 2 | 3 | 4 |
| Variant required | 21% | 66% | 93% | 99% |
| Multi-quantity | 0% | 30% | 50% | 50% |
At d = 0 the agent adds a single product with no variant complexity — learning the basic catalog.search → cart.add workflow. At d = 6 it juggles 3 items, nearly all requiring a specific variant, with half needing qty > 1.
Scoring
The cart must be exactly right — correct product, correct variant, correct quantity. Partial credit is given for partially correct carts, but a perfect score requires every item to match. If the agent adds the wrong variant, the simulated user corrects it mid-dialogue ("that's the Lightning version, but I need USB-C"), giving the agent a chance to self-correct before the episode ends.
Trajectories: easy vs. hard
Two real E_CART episodes from a Qwen 3 8B agent. Same environment, same agent — difficulty alone changes the game.
| d = 1 — 1 item, no variants | d = 8 — 3 items, variants + typos | |
|---|---|---|
| User message | "Shopping list: SonicWave Keypad. Please add everything." | "pls add the following to my aart: Eco-Frienly HrenLefa Charger by GreenLeaf, High-Performance AqaPure Air Fryer , Lighrweiht PixelVault Tank Top (3)." — typos from noise axis |
| Turn 1 | user.get_visit_history() → 4 results. Spots the target immediately. | user.get_visit_history() → 9 results. Must disambiguate across chargers, fryers, tops. |
| Turn 2 | cart.add("syn_000162", qty=1) → Cart: 1 item, $2.74. "I found the SonicWave Keypad." User: "Great, that's the exact one." | No tool calls — presents 3 product cards. User: "Add the Charger with 100W output, Air Fryer with Charcoal filter, 3 Tank Tops in size XS." |
| Turn 3 | answer: { "done": true } — Done in 3 turns. | catalog.get_variants × 3 → charger: 6 variants (100W, 120W, 30W…), air fryer: 6 (Bamboo, Permanent, Charcoal…), tank top: 7 (XS, S, XL…). User: "I need three Tank Tops in XS." |
| Turn 4 | cart.add(charger, "100W", qty=1) ✓ cart.add(fryer, "Bamboo", qty=1) ✗ should be Charcoal. cart.add(tank_top, "size_XL", qty=3) ✗ should be XS. User: "Air Fryer has Bamboo not Charcoal, Tank Top is XL not XS." | |
| Turn 5 | Re-fetches tank top variants, adds 2× XS — doesn't fix the air fryer. User: "Air Fryer still shows Bamboo. I need three XS, not two." | |
| Turn 6 | Re-fetches tank top variants again — still ignores the air fryer. User: "Please change the Air Fryer variant." | |
| Turn 7 | "I cannot find a Charcoal filter variant." — It exists; the agent saw it in turn 3 but never selected it. User: "Skip the Air Fryer then." | |
| Turn 8 | answer: { "done": true } — Cart still wrong: Charger ✓, Air Fryer (Bamboo) ✗, Tank Top (3×XL + 2×XS) ✗ | |
| Reward | r_task = +1.00, r_eff = +0.33, r_hall = 0.00, r_total = +0.80 ✓ | r_task ≈ 0.00, r_eff = −0.43, r_hall = 0.00, r_total = −0.06 ✗ |
| Outcome | Cart matches goal. 3 turns, 2 effective. | Wrong variants, wrong quantities, user gave up. 8 turns, 6 effective. |
At d=1 the agent solves the task in 3 clean turns. At d=8 it spirals — picking Bamboo instead of Charcoal, XL instead of XS, never fixing the air fryer despite two user corrections, then hallucinating that the variant doesn't exist. This is exactly the kind of multi-step error cascade that the difficulty curriculum surfaces, and that adaptive training should teach the agent to recover from.
User simulation
A verifiable environment needs a user simulator that behaves realistically. We use Qwen3.5 (9.7B) to generate natural, varied user messages rather than canned templates — covering everything from typo-filled requests to mid-conversation topic switches.
Two design choices matter for training quality:
Preferences match stated constraints. Each simulated user has a hidden set of preferences (price sensitivity, brand loyalty, shipping speed, etc.). These are deliberately biased toward whatever constraints the user communicated — so if the user said "under $25", the reward function actually cares about price. Without this, an agent could be penalised for correctly following the user's instructions.
Strategic omission. The LLM deliberately withholds some constraints from the opening message to force the agent to ask clarifying questions. The system tracks exactly what was and wasn't mentioned, so the agent is never penalised for information it was never given.
Environment scaling
Following RLVE's methodology, we define nested environment collections:
C1 ⊂ C2 ⊂ C4 ⊂ C8
| Collection | Environments | Skills trained |
|---|---|---|
| C1 | Cart | Serarch Query Formulation, Cart Manipulation |
| C2 | + Substitution | Similarity reasoning under constraints |
| C4 | + Product Discovery, Returns | Transactional workflows (Retrieval + recommendation, return initiation) |
| C8 | + Status, Policy, Bundle, Journey | Knowledge retrieval, planning, compositionality |
We hypothesise — consistent with RLVE's findings — that C8 agents outperform single-environment specialists, even on the specialist's own task.
Early results
We trained Qwen 3 8B with DAPO on C1 (Cart Building) for 300 steps as an initial viability study.
| Config | |
|---|---|
| Base model | Qwen 3 8B |
| Algorithm | DAPO (G = 8 rollouts/prompt) |
| LR | 1e-5 |
| Catalog | 2M products, FAISS index with Alibaba-NLP/gte-modernbert-base (768-dim) |
| User sim | Qwen3.5 9.7B |
We saw progressive growth in difficulty reached, confirming that adaptive scheduling produces a steady learning signal rather than the saturation (static-low) or starvation (static-high) patterns predicted by the RLVE paper.
Try it yourself
Run a live episode directly in your browser using the embedded demo below. Here is how to get started:
- Pick an environment from the dropdown (e.g.,
E_CARTfor cart building orE_PDfor product discovery). - Set a difficulty —
0is a simple single-constraint task;6+introduces missing information, noisy retrieval, and variant selection. - Click "Reset Episode" — the simulated user will open with a shopping request.
- You are the agentnow: Make tool calls, analyse outputs and submit the final list of product ids.
- Click "Reset Episode" between runs to start a fresh scenario.
Resources
The environments, verifiers, and training configs are all open-source:
git clone https://github.com/owlgebra-ai/EcomRLVE-Gym
cd EcomRLVE-Gym
pip install -e .
The 2M-product catalog is on the Hub:
from datasets import load_dataset
catalog = load_dataset("owlgebra-ai/Amazebay-catalog-2M", split="train")
print(f"{len(catalog)} products loaded")
References
Zeng, Z., Ivison, H., Wang, Y., et al. (2025). RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments. ICML 2025. arXiv:2511.07317
Yu, Q., Zhang, Z., Zhu, R., et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale.arXiv:2503.14476
Shao, Z., Wang, P., Zhu, Q., et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.arXiv:2402.03300
DeepSeek-AI. (2025). DeepSeek-R1: Incentivizing Reasoning in LLMs through Reinforcement Learning. Nature.
Meta AI. (2024). Llama 3.1: A Foundation Model for General Intelligence.llama.meta.com
Qwen Team. (2025). Qwen3 Technical Report.arXiv:2505.09388