JamSet与JamBench:首个项目级游戏代码框架数据集与基准
JamSet和JamBench是基于Godot引擎的首个项目级游戏代码框架数据集与基准。从24万仓库筛选出8133个验证项目,其中300个手动验证为JamBench。定义主题驱动生成与代码补全任务,用编译通过率、结构完整性(SCS)、行为对齐(BAS)评估。9个前沿模型评测显示项目规模增大导致运行时通过率从80.4%(小项目)降至5.7%(大项目)。代码智能体提升编译率但未改善运行时行为,瓶颈在架构设计。实验验证JamSet作为训练数据有效。所有数据与代码已公开。
Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine. Our key insight is that Game Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose. Building on the Godot engine's text-based format and headless execution mode, we design a deterministic verification pipeline from file integrity to runtime behavior collection, distilling 8,133 verified projects from over 240,000 repositories. Of these, 300 manually verified projects form JamBench; the rest constitute JamSet. JamBench defines theme-driven generation and code completion tasks, evaluated through a pipeline combining compilation pass rates, Structural Completeness Score (SCS), and Behavioral Alignment Score (BAS). Evaluation of 9 frontier models reveals a capability cliff as project scale increases, with runtime pass rates dropping from 80.4% on small projects to 5.7% on large ones (Task2a). Code Agents improve compilation rates yet yield no gains in runtime behavioral quality, indicating that the bottleneck lies in architectural design rather than syntactic correctness. Experiments validate JamSet as effective training data. All data and code are publicly available.