Abstract

LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder. Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances. Existing frameworks conflate documentation reading, behavioral exploration, and code synthesis into a single pass, causing agents to probe insufficiently, lose behavioral intent as context drifts, and propagate early misinterpretations into the final implementation. Inspired by classical requirements engineering, we argue that behavioral specification elicitation should be a first-class phase that precedes implementation. We present SpecFirst, a two-stage framework that forces the specification elicitation before code synthesis. A dedicated spec agent first probes the binary and combines observations with documentation into a structured specification. Next, a code synthesis agent then uses this specification to drive implementation. This decomposition resolves documentation ambiguities before coding begins and provides a stable behavioral reference throughout synthesis. We evaluate SpecFirst on all 200 ProgramBench instances across four models spanning two families and an order of magnitude of capability. SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%–21.3% and binary exploration coverage by 9.4%–18.5%, all statistically significant. Behavioral analysis on code synthesis further shows that a prior specification enables earlier and more sustained code construction. Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.

I Introduction

LLM-based agents have demonstrated strong performance on software engineering tasks such as bug fixing [21], feature implementation [40], and code completion [8, 16]. Those tasks share a common structure: an existing codebase anchors the agent’s work. Building a program entirely from scratch is fundamentally harder: the agent must decide everything (e.g., architecture and programming language) and reconstruct the full behavioral envelope of a target program from limited information alone. ProgramBench [41] makes this concrete: given only natural-language documentation and an execute-only binary as a behavioral oracle, an agent must produce a faithful re-implementation from scratch. Even the strongest frontier models (e.g., GPT-5.5) can only fully resolve fewer than 1% of instances, confirming that from-scratch program construction remains an open challenge.

Existing agent frameworks such as SWE-agent [40] and OpenHands [37] approach this task by providing the agent with natural-language documentation as the primary specification and an execute-only binary as a behavioral oracle [41]. While this setup permits behavioral discovery through probing, these frameworks mix specification elicitation and code synthesis into a single undifferentiated loop—the agent switches freely between planning, reading documentation, probing the binary, and writing code within the same turn budget, with no mechanism forcing comprehensive elicitation before implementation begins. This entanglement leads to three systematic limitations.

Ineffective exploratory probing. Without a dedicated phase for behavioral discovery, the agent rarely probes the binary thoroughly enough to construct a complete behavioral model before committing to implementation. Edge cases, error paths, and flag interactions that are absent from the documentation but present in the binary are frequently missed, because the agent shifts to coding before its exploration is complete.

Specification loss over long horizon. Without explicit upfront elicitation, behavioral intent must be maintained implicitly across turns and is susceptible to drift as context grows [11, 23]. Compression mechanisms such as reasoning token stripping (i.e., removing the reasoning content between agent turns) and context summarization exacerbate this by selectively discarding tokens that encode the original specification [24, 26], so later-turn outputs increasingly reflect a degraded approximation of the target behavior.

Error Propagation without anchoring. Without an elicited specification as a stable reference, errors introduced early in the run accumulate and propagate across subsequent turns without correction. When the agent misinterprets the documentation or makes an incorrect behavioral assumption, it continues building on that assumption turn after turn—each dependent decision compounding the original error with no artifact to anchor against.

In traditional software engineering, these limitations are addressed by treating specification elicitation as a first-class phase that precedes implementation [28, 34]: requirements engineers exercise prototypes and construct scenarios to surface behavioral detail that written artifacts never capture, before a single line of code is written [38, 44]. We argue the same decomposition should apply to agentic program construction. Spec-driven frameworks such as OpenSpec [14] and spec-kit [17] adopt a similar philosophy, but rely solely on prompting human stakeholders for specification input, and do not support automated behavioral elicitation through binary probing.

We present SpecFirst, a two-stage pipeline that explicitly separates behavioral specification elicitation from code synthesis. In the first stage, a dedicated spec agent receives the documentation and execute-only binary and focuses exclusively on understanding the target program’s behavior through systematic black-box probing, producing a structured behavioral specification artifact (i.e., SPEC.md). In the second stage, an existing code synthesis agent receives the documentation, the binary, and SPEC.md, and focuses exclusively on implementation, now equipped with a behavioral model that goes beyond what the documentation alone provides. This decomposition directly addresses each limitation above: the spec agent explores freely without competing with coding, documentation ambiguities are resolved through probing before implementation begins, and the behavioral gap between documentation and binary is explicitly addressed rather than left to chance.

We evaluate SpecFirst on the full 200-instance ProgramBench suite [41] across four models spanning two families and an order of magnitude of capability (Qwen3.5-397B-A17B, Qwen3.6-35B-A3B, GPT-5.5-high, and GPT-5.4-mini). Across all settings, SpecFirst consistently outperforms the baseline (i.e., mini-SWE-agent), with test pass rate improvement of 6.9%–21.3% and binary exploration coverage gains of 9.4%–18.5%, all statistically significant. Behavioral analysis further confirms that the decomposition restructures how the code synthesis agent allocates its budget: with a prior specification, the agent commits to writing code earlier and produces larger, more complete implementations.

1. 1.

   SpecFirst framework. We propose the first agentic pipeline that explicitly decouples behavioral specification elicitation from code synthesis for from-scratch program construction, directly instantiating the requirements-engineering phase.

2. 2.

   Empirical evaluation. We evaluate on all 200 ProgramBench instances across four models, demonstrating consistent improvements in test pass rate (6.9%–21.3%) and binary exploration coverage (9.4%–18.5%), all statistically significant.

3. 3.

   Behavioral analysis. We show that having a prior specification changes how the code synthesis agent spends its turn budget: it starts writing code earlier and sustains implementation longer, rather than alternating between probing and coding throughout the run.

The remainder of this paper is organized as follows. Section II presents the background and limitations of existing approaches. Section III introduces the details of SpecFirst. Section IV presents experimental settings. Section V presents the evaluation and results. Section VI surveys the related work and discusses the differences between our work and previous studies. Section VII discusses failure cases, formats, costs, and threats to validity. Section VIII concludes our study.

II Background and Motivation

II-A Problem Formulation

Let denote a target program that is a deterministic command-line program. The agent observes through two artifacts: (i) documentation (e.g., a readme file) that describes in natural language, and (ii) an execute-only binary that instantiates ’s observable behavior but whose source code and internal structure are inaccessible. The agent task is to produce an implementation containing source code and necessary files to build an executable file, that reconstructs the behavior of . The implementation may be written in any programming language and architecture chosen by the agent. Next, it builds a candidate executable from , and evaluates against a given hidden test suite , where each test consists of a command-line argument string , a standard-input stream , and the ground-truth output produced by on that input. The objective is to produce that maximizes test pass rate:

(1)

which measures the fraction of test cases where reproduces the observable behavior of exactly. A submission achieves if and only if is behaviorally equivalent to on . Figure 1 (blue box) presents the general pipeline of existing code synthesis agents [40, 37, 41]. The agent operates in an iterative probe-then-build loop: it invokes on selected inputs and reads to build an understanding of the program’s behavior, then progressively writes, tests, and refines source code until it judges the implementation sufficiently complete. The final output is a repository containing source code and a build file (e.g., compile.sh), which the evaluation harness compiles into a candidate executable and scores against hidden tests.

II-B Limitations of Existing Code Synthesis Agent

媒体内容 · 前往原文查看

⬇

gomplateisatemplaterendererthatsupportsmultipledatasources(JSON,YAML,HTTPendpoints,environmentvariables,andmore).

###Usageexamples:

Basicusagewithenvironmentvariables:echo’Hello,{{.Env.USER}}’|gomplate

Inlinetemplateevaluation:gomplate-i’theansweris:{{mul67}}’

Usingadatasourcefile:gomplate-dconfig=./config.yaml-i’thevaluewewantis:{{(ds"config").foo.bar.baz}}’

Seetheofficialdocumentationforfulldetailsandexamples:https://docs.gomplate.ca

License:seetheLICENSEfileinthisrepository.

In practice, provides only a surface-level description of —primary features, typical usage, and representative examples—while remaining ambiguous or silent on edge cases, output formats, and error behavior [5, 13]. Figure 1 shows the README for gomplate¹¹1https://github.com/hairyhenderson/gomplate, a command-line template renderer in ProgramBench. The README describes the tool’s purpose and lists common flags, but gomplate extends Go’s text/template engine with a large library of built-in functions spanning namespaces such as coll, math, net, crypto, and strings. No README can enumerate the precise semantics, argument types, error conditions, and output formats of every function; faithful re-implementation therefore requires extensive behavioral exploration of the binary. We use this instance to illustrate the limitations of the existing single-loop framework.

Ineffective exploratory probing. Without a dedicated exploration phase, the baseline agent interleaves probing with coding, allocating only a fraction of its turns to behavioral discovery before committing to an implementation. For gomplate, this is particularly damaging: the agent must understand not only the top-level CLI interface but also the semantics of functions across ten namespaces, each with its own argument conventions and edge-case behavior. In practice, we observe that the general code synthesis agent (i.e., mini-SWE-agent using Qwen3.6-35B-A3B) confines its probing to the functions explicitly mentioned in the README—primarily the env and data namespaces—while leaving the behavior of coll, math, net, and random largely unverified.

Specification loss over long horizon. Even when the collected specification is read correctly, its content can be lost across a long run if nothing anchors it. In gomplate, the general code synthesis agent (i.e., mini-SWE-agent using GPT-5.5-high) collected the full namespace index at turn 4, receiving function signatures for every namespace including aws, gcp, sockaddr, semver, sprig, and cidr. Yet across the remaining 25 turns, none of these six namespace names appears again in the agent’s reasoning or tool calls. The information was observed once and never re-referenced: the final submission’s function map registers 17 namespaces and silently omits all six, causing 126 test cases to fail before a single line of code is written for them.

Error propagation without anchoring. Without a persistent specification, a single early misinterpretation can silently corrupt every subsequent implementation decision. In the gomplate instance under Qwen3.6-35B-A3B, the agent correctly reads the typed signature Seq(start, end, step int64) []int64 from documentation at turns 4–5. Yet when Seq is first implemented at turn 57, it emits a degraded form—func Seq(args ...interface) []interface— discarding both typed arguments and return value. Over the next 140 turns the math namespace is refactored five times, each time re-emitting the wrong signature without ever consulting the original documentation. The incorrect signature ships in the final submission, causing 25 of 43 Seq-related tests to fail with type errors.

These limitations motivate SpecFirst’s core design: a dedicated spec agent that systematically explores the specification before implementation begins, externalizing its findings into a persistent specification file (Spec.md) that eliminates the coverage gap from ineffective probing, prevents early misinterpretations from propagating uncorrected, and ensures that Behavioral knowledge discovered during exploration is never lost across turns.

III Approach of SpecFirst

Our approach directly addresses all three limitations through a dedicated specification elicitation phase prior to implementation. We implement an autonomous spec agent that probes systematically through black-box interaction, uncovering behaviors that documentation leaves implicit or omits entirely — directly countering by ensuring exploration is complete before coding begins. The elicited specification is persisted as an explicit artifact (i.e., SPEC.md in our case), which serves as a stable behavioral contract that anchors all subsequent implementation decisions, resolving by providing an anchor that prevents early errors from compounding. Critically, because is stored externally rather than maintained implicitly in the model’s context, it remains intact regardless of context growth or compression, mitigating . The overall framework of SpecFirst is presented in Figure 1. The code synthesis agent can be any existing agent (e.g., OpenHands [37] or SWE-agent [40]). If the agent encounters an ambiguity or gap in during implementation, it may still probe binary to resolve it. We elaborate on the design and operation of the spec agent in the following sections.

III-A Eliciting Behavior Specification through Probing of the Executable File

At each turn, the spec agent selects a probe, i.e., an invocation of with arguments, flags, or stdin chosen from what it has inferred so far, and observes the run time results from triple channels . Free-form bash was chosen over a structured probe API because it mirrors how a human engineer would interact with an unfamiliar tool, and because it allows the agent to chain commands (e.g., constructing input files, running the binary, and inspecting its output in one step). For interactive or TUI programs, pre-installed tmux/libtmux provides a virtual terminal through which the agent can send keystrokes and capture screen state.

The spec agent conducts an iterative probing loop in which each observation of binary informs the next probe. Starting from the behavioral skeleton provided by , the agent progressively deepens its exploration of by targeting the classes of behavior that documentation characteristically under-specifies. Analyzing of the reveals four recurring probing patterns that go beyond what already provides:

• •

  Boundary probing: testing inputs at the limits of accepted ranges (e.g., empty input, maximum-length strings, special characters) to determine exact boundary semantics that documentation leaves implicit.

• •

  Error-path elicitation: systematically triggering error conditions (malformed input, missing required arguments, conflicting flags) to record the precise stderr message and exit code each condition produces.

• •

  Combinatorial flag testing: exercising flags in combination to surface interaction effects absent from per-flag documentation.

• •

  Output-format refinement: presenting adjacent inputs and comparing their outputs to resolve ambiguities in field ordering, delimiter choice, and whitespace handling that prose descriptions inevitably leave underspecified.

III-B Constraints and Shortcut Prevention

We observe that LLM agents can shortcut the intended task in ways that inflate apparent performance without genuine behavioral understanding. Two shortcuts are particularly dangerous in this setting: (a) source recovery locates the original source code via package registries, GitHub, or web search, which would trivialize re-implementation; (b) binary introspection uses disassemblers, tracers, or decompilers to bypass black-box interaction entirely. Neither shortcut represents the specification-elicitation behavior we aim to study, and both would confound the results.

Therefore, to address this, when designing prompts, we explicitly prohibit LLM from doing both classes of actions. All interactions with must go through its normal user interface (CLI flags, stdin/stdout). An automated judge inspects the agent’s command history for prohibited patterns (repository clones, registry installs, source tarball downloads, and disassembler calls); any detected violation disqualifies the run and records a score of zero. Wrapping or shimming the provided binary is similarly prohibited and detected. This enforcement ensures that all behavioral knowledge encoded in the generated spec file derives from black-box observation, making the specification-elicitation loop the sole source of information beyond .

III-C Specification Deliverable and Format

Even a thorough elicitation loop is only useful if its findings are communicated in a form the downstream exec agent can act on. The format of specification is a non-trivial design variable. A free-form transcript of raw observations is hard to act on, while an over-prescribed template may suppress behavioral detail that does not fit its categories. We adopt the sections format: a light scaffold of six named headings: Overview; Flags; Input & stdin; Output format; Error patterns; Edge cases. They are chosen as the minimal structure that covers the behavioral dimensions a re-implementer needs. The headings prescribe what to document, not how to discover or phrase it, ensuring the isolation between specification elicitation and code synthesis. More comparison of different formats (i.e., freeform and OpenSpec [14]) can be found in Section VII-B.

III-D Termination

There is no semantic oracle for specification completeness: no external judge can assess whether SPEC.md captures all of ’s observable behavior. An agent that terminates too early produces an incomplete spec; one that never terminates produces no spec at all. A run terminates under one of three conditions, applied in priority order.

• •

  Self-declared completion (primary): if the spec agent judges the specification complete, it finalizes the process. Completeness is the agent’s own judgment, which is itself a research variable we measure.

• •

  Step limit: if 1,000 agent–environment turns is reached, it terminated. The generous step limit (1,000 turns) is set to permit high-coverage exploration; in practice, most runs self-terminate well before this cap, confirming that the limit is a safety net rather than a binding constraint.

• •

  Wall-clock limit: We use 6 hours as a final safeguard. The spec agent is terminated after running 6 hours.

Finally, a deliverable gate enforces artifact presence. If the specification file is absent at submission, the run is rejected and the agent is asked to write it (capped at 8 rejections, after which the run is recorded as a failure).

III-E Principles of Agent Design

We introduce the principles of the agent design involved in SpecFirst in this section. General agent architecture. All our agents follow the ReAct agent design by following previous work [42, 43]. Except for that, we force all agents to emit reasoning output after each tool calling, including observation, considering alternative actions, and reasoning about next actions. This reasoning output forces explicit processing rather than blind tool call chaining and creates interpretable reasoning traces for debugging. Context Management. We compress information passing through the two stages (i.e., spec agent and code synthesis agent), to prevent information overload. More specifically, we condition the code synthesis agent with the structured behavioral specification artifact (i.e., SPEC.md), while discarding verbose reasoning and tool calling outputs from the spec agent. This preserves essential findings while removing redundant content that causes attention dilution when context windows become too large. We summarize the context information between stages rather than complete conversation histories [40, 42].

IV Experimental Design

In this section, we present our research questions (RQs), studied dataset, retrievers, used prompt templates, implementation details, and our approach to RQs.

IV-A Research Question

We aim to answer the following research question:

• •

  RQ1: What is the effectiveness of SpecFirst?

• •

  RQ2: How effective is SpecFirst across task difficulty levels?

• •

  RQ3: Does the Spec Agent improve behavioral exploration coverage?

• •

  RQ4: How does SpecFirst affect the behavior of the code synthesis agent?

In RQ1, we evaluate the effectiveness of SpecFirst in synthesizing programs from scratch and compare it with selected baseline. In RQ2, we evaluate the effectiveness of SpecFirst across instances with different difficulty levels. In RQ3, we evaluate the specification exploration ability of SpecFirst and compare it with the baseline. In RQ4, we investigate how the spec agent affects the behavior of the code synthesis agent by comparing pipeline configurations with and without spec elicitation.

IV-B Benchmark

We evaluate on the full ProgramBench suite [41], which comprises 200 command-line program instances drawn from real-world open-source tools, covering widely used software and complex applications, such as FFmpeg, SQLite, and the PHP interpreter. Those 200 task instances in ProgramBench are built from repositories written primarily in compiled languages: Rust (107), Go (46 repositories), C/C++ (45 repositories), 1 (Java), and 1 (Haskell). Each instance provides a natural-language documentation (a README and man page) describing the program, and an execute-only binary that serves as the behavioral oracle during spec elicitation and implementation. Each instance is labelled by difficulty—easy (28), medium (143), and hard (29)—and annotated with the implementation language of the reference binary. To evaluate the correctness and coverage of the implemented program, the benchmark contains 248,853 test functions across its 200 tasks, averaging a median of 770 tests per task. These suites achieve an average of 79.7% line coverage, and are hidden from the code synthesis agent.

IV-C Evaluation Metric

Average test pass rate as the primary metric. For each instance , let denote the number of hidden test cases passed by the candidate executable and the total number of test cases for that instance. The instance-level pass rate is , and the benchmark-level score is the mean over all instances:

(2)

This metric awards partial credit for implementations that reproduce a subset of the target’s behavior, making it sensitive to incremental improvements that a binary resolved/unresolved rate would mask. Note that even the strongest baseline, GPT-5.5-high without spec elicitation, fully resolves only 1 of 200 instances (resolved rate 0.5%) [41], yet achieves a non-trivial reflecting partial behavioral coverage across many instances. Average test pass rate therefore provides a meaningful signal for comparing pipeline configurations in a regime where full resolution is rare.

Probing Coverage. To quantitatively evaluate the specification exploration ability of our approach against the baseline, we introduce probing coverage as the primary metric. Given a target binary executable , let denote the set of all executable lines of code in . During the probing phase, the agent interacts with by issuing a sequence of probes; for each probe, we instrument to record the set of lines executed. We use a language-specific instrumentation tool for each language. For instance, we use go build -cover for Go binaries, gcc --coverage for C/C++ binaries, and cargo llvm-cov for Rust binaries to capture line-level execution traces for every probe issued during the session. Upon completion of the probing phase, we aggregate the unique recorded lines across all probes to obtain the covered set . Probing coverage is then defined as:

(3)

A higher probing coverage indicates that the agent has exercised a larger portion of the program’s behavior during specification, reflecting a more thorough and complete understanding of the binary’s functionality.

IV-D Models

We evaluate four models spanning two families and an order of magnitude of capability, selected to disentangle the effect of model scale from the effect of model family. Table I summarizes their key properties. All four models are evaluated with reasoning enabled, reflecting the multi-step inference the task demands. The Qwen models expose no effort-level control and are evaluated with thinking on by default. We do not tune any other decoding parameters and use provider defaults throughout, to avoid conflating model capability with hyper-parameter optimization.

媒体内容 · 前往原文查看

Model	Family	Type	Reasoning
Qwen3.5-397B-A17B [39]	Open-weights	MoE	On (default)
Qwen3.6-35B-A3B [39]	Open-weights	MoE	On (default)
GPT-5.5-high [30]	Proprietary	Unknown	High effort
GPT-5.4-mini [29]	Proprietary	Unknown	Medium effort

IV-E Baselines

Direct-Synthesis: The baseline is the original ProgramBench pipeline [41], in which the code synthesis agent receives and proceeds directly to implementation without any preceding specification elicitation phase. This baseline establishes the performance ceiling of the standard approach and measures the net gain attributable to the entire specification elicitation.

Note that we use the same scaffold (i.e., official mini-swe-agent) as the code synthesis agent for both Direct-Synthesis and SpecFirst. In other words, the only difference between Direct-Synthesis and SpecFirst is that we introduce spec agent in SpecFirst, and other settings are kept the same.

IV-F Implementation details

Each run executes in an isolated, no-network Docker container from the task image by ProgramBench. The same model is used for both specification elicitation and code synthesis phases. Our local modifications to each are released as patches against those exact commits. All models are accessed through a single OpenAI-API-compatible routing proxy LiteLLM [4], with a fresh context per pipeline phase to prevent information leakage between the spec and exec runtimes. All experiments were conducted using model APIs accessed via a stable API provider.

V Results

V-A RQ1 - Effectiveness of SpecFirst

V-A1 Approach

We compare SpecFirst against Direct-Synthesis across all four models, with all other experimental conditions held constant. For each model, we run both configurations on the full 200-instance suite and compute the per-instance pass rate for each instance. To assess whether the improvement in average pass rate is statistically significant, we apply a paired two-sided Wilcoxon signed-rank test (). We report win/loss/tie counts to characterise the direction and prevalence of the effect across instances. To further understand whether SpecFirst improves not just the average performance, but the proportion of high-quality implementations, we report the fraction of instances achieving a pass rate above a threshold (i.e., 50% - 95%) for both SpecFirst and Direct-Synthesis.

媒体内容 · 前往原文查看

Model	Direct-Synthesis	SpecFirst	W/L/T
Qwen3.5-397B	33.66%	40.84% (+21.3%)	130/58/12
Qwen3.6-35B	27.51%	31.40% (+14.1%)	121/67/12
GPT-5.5-high	59.02%	65.14% (+10.4%)	150/38/12
GPT-5.4-mini	39.09%	41.78% (+6.9%)	119/69/12

V-A2 Results

SpecFirst consistently improves the average test pass rate across all four models, with relative gains ranging from 6.9% to 21.3%, up to 65.14% with GPT-5.5-high. Table II shows that SpecFirst consistently improves the average test pass rate across all four models, with relative gains ranging from 6.9% (GPT-5.4-mini) to 21.3% (Qwen3.5-397B-A17B). All improvements are statistically significant (), confirming that the gains are not attributable to random variation. The win/loss/tie counts reinforce this conclusion: across all models, SpecFirst outperforms the baseline at least on more than 59% of instances, with the strongest model (GPT-5.5-high) winning on 75% (150/200) instances. Taken together, these results confirm that explicitly eliciting a behavioral specification before implementation is a robust and model-agnostic intervention that materially improves program re-implementation fidelity.

SpecFirst does not merely lift average pass rates; it significantly expands the upper tail of the score distribution by driving a higher proportion of programs to near-perfection (i.e., the programs that pass almost all tests). Table III presents the proportion of programs, where the test pass rate is larger than a threshold (50% to 95%). For instance, at the most stringent thresholds, SpecFirst triples the rate of near-perfect solutions from 5.5% to 16.5% (for ) and more than quadruples it from 1.5% to 6.5% (for ). We observe a consistently similar trend across the other evaluated models.

媒体内容 · 前往原文查看

	Baseline	SpecFirst
	69.0% [62.3%, 75.0%]	74.0% [67.5%, 79.6%]
	42.5% [35.9%, 49.4%]	55.5% [48.6%, 62.2%]
	5.5% [3.1%, 9.6%]	16.5% [12.0%, 22.3%]
	1.5% [0.5%, 4.3%]	6.5% [3.8%, 10.8%]

媒体内容 · 前往原文查看

⬇

##FunctionSet

TheexecutableprovidesgomplatenamespacesandaliasesinadditiontoGotemplatebuilt-ins.Importantobservedfunctionsandaliasesinclude:

-Data:datasource/ds,include,datasourceExists,datasourceReachable...

-Collections:coll.Dict/dict,coll.Slice,coll.GoSlice...

##Flags

$./executable--help

ProcesstextfileswithGotemplates

Usage:

gomplate[flags]

Flags:

-d,--datasourcedatasourcedatasourceinalias=URLform.Specifymultipletimestoaddmultiplesources.

...

##EdgeCasesandCompatibilityNotes

--inplus--fileor--input-dir:invalid(exclusive-inputconflict).

--fileplus--input-dir:likewiseinvalid.

...

Listing 2 shows an example specification elicited for the gomplate instance. For example, SpecFirst records a thorough SPEC.md containing all function sets and edge cases. An exec agent that reads it before writing code can implement each of these behaviors correctly on the first attempt rather than guessing, which directly reduces the behavioral residual between and and explains the consistent pass rate lift across all four models.

V-B RQ2 - Effectiveness across different difficulty levels

V-B1 Approach

To assess the robustness of SpecFirst, we analyze its performance across different task-difficulty tiers: Easy (), Medium (), and Hard (), with the difficulty tags provided in the benchmark.

媒体内容 · 前往原文查看

Model	Easy ()	Medium ()	Hard ()
Qwen3.5-397B	42.9% 52.9% (+23.3%)	34.6% 42.1% (+21.7%)	20.0% 23.0% (+15.0%)
Qwen3.6-35B	37.8% 42.2% (+11.6%)	28.1% 32.1% (+14.2%)	14.5% 17.6% (+21.4%)
GPT-5.5-high	73.7% 78.9% (+7.1%)	61.9% 67.5% (+9.0%)	30.8% 40.0% (+29.9%)
GPT-5.4-mini	49.7% 57.4% (+15.5%)	40.7% 42.8% (+5.2%)	21.0% 21.9% (+4.3%)

V-B2 Result

SpecFirst consistently outperforms Direct-Synthesis across all difficulty levels and all evaluated models. For instance, GPT-5.5-high experiences its most pronounced surge, improving the average test pass rate from 30.8% to 40.0%, which represents a remarkable 29.9%. Rather than being restricted to a specific tier of tasks or a particular model architecture, the SpecFirst shows a universal improvement. More importantly, the performance gains are not confined to trivial or moderately difficult tasks. SpecFirst delivers substantial improvements on the hardest problems. Crucially, the largest absolute improvements often manifest where the baseline struggles the most. For instance, on Hard instances, the frontier model GPT-5.5-high experiences its most pronounced surge, improving the average test pass rate from 30.8% to 40.0%, which represents a remarkable 29.9% improvement. This pattern indicates that a rigorous, decoupled specification phase serves as a vital cognitive scaffold, allowing models to systematically decompose intricate logic that would otherwise cause a direct code-generation attempt to fail.

V-C RQ3 - Effectiveness of Spec Agent

V-C1 Approach

The spec agent is dedicated to probing the binary and eliciting a behavioral specification before implementation begins. To assess whether it actually improves exploratory coverage, we measure probing coverage of SpecFirst and compare it against Direct-Synthesis, which performs no dedicated elicitation phase.

V-C2 Results

媒体内容 · 前往原文查看

Model	SpecFirst	Direct-Synthesis	Win
	Spec / Synthesis / Union
Qwen3.5-397B	58.0%/51.0%/59.8%	51.9%	77.2%
Qwen3.6-35B	54.9%/51.3%/58.3%	49.2%	71.2%
GPT-5.5-high	58.3%/31.2%/60.3%	55.1%	66.2%
GPT-5.4-mini	56.5%/41.7%/58.6%	52.1%	63.5%

SpecFirst consistently elicits broader behavioral coverage than Direct-Synthesis significantly, achieving 9.4%–18.5% improvement. The gain originates from the spec agent rather than from the code synthesis agent. Table V reports probing coverage across all four models. SpecFirst achieves 58.3%–60.3% total coverage, an improvement of 9.4%–18.5% over Direct-Synthesis on every model. All improvements are statistically significant (). Its spec agent covers more behaviors than the baseline in 63.5%–77.2% of instances.

Decomposing SpecFirst by component reveals that the spec agent alone (54.9%–58.3%) consistently exceeds the code synthesis agent (31.2%–51.3%) on all models, and the union adds only marginally over the spec agent. This confirms that the coverage gain is driven by the dedicated elicitation phase, not by incidental exploration during code synthesis.

V-D RQ4 - Impact of Spec Agent on Code Synthesis Agent’s behavior

V-D1 Approach

To understand how the elicited specification shapes the code synthesis agent’s behavior during the run, we record the codebase size (lines of code) at every agent–environment turn for both SpecFirst and Direct-Synthesis, and normalise each turn index to by dividing by the run’s total number of turns.

V-D2 Results

Providing a persistent behavioral specification before implementation shifts the code synthesis agent from a probe-then-build pattern to an earlier, more sustained building phase. Figure 2 tracks the growth of the reconstructed codebase as the code synthesis agent progresses through its turns. SpecFirst starts implementation earlier than baseline. In the baseline, the early portion of the run is dominated by exploratory probing. For instance, Figure 3 compares the probing phase and implementation phase of SpecFirst and the baseline on four instances. On age, for example, the baseline spends most turns inspecting behavior before attempting to dump the entire 841-line implementation in a single action, inevitably failing due to an exhausted debugging budget. The baseline typically exhausts 11 to 20 turns probing the binary’s behavior. In contrast, SpecFirst directly resolves this bottleneck by externalizing exploration into a persistent SPEC.md during the dedicated spec phase. Instead of wasting implementation turns on trial-and-error probing, the code synthesis agent spends just 2 to 9 turns establishing context before immediately shifting to implementation.

Direct-Synthesis agents stop early, not because they run out of budget, but because they run out of behavioral understanding. One possible attribute of the weaker performance of Direct-Synthesis to insufficient turn budget: if probing consumes too many turns, the agent has little budget left for implementation. To rule out this explanation, we examine the number of turns consumed by Direct-Synthesis. The results are striking: zero instances reach the 1,000-turn step limit, and all instances terminate early by agent choice, consuming a median of only 22–177 turns (2–18% of the available budget). The bottleneck is therefore not turn budget but the quality of behavioral understanding the agent constructs before it stops: agents terminate voluntarily because they believe they have understood enough, not because they exhausted their budget. In other words, simply allocating more compute would not help. SpecFirst addresses this by equipping the code synthesis agent with a persistent specification established through the dedicated spec agent prior to implementation, enabling higher coverage of the specification, as demonstrated in RQ3.

Introducing the spec agent leads to larger codebases and higher pass rates across all models. The final codebase produced under SpecFirst is 7–29% larger across models (shaded IQR), consistent with agents writing more complete implementations rather than terminating early under behavioral uncertainty. This aligns with RQ3, where we demonstrate that a higher probing coverage from the spec agent yields a more complete specification, which in turn drives broader implementation coverage.

Together, these results show that specification elicitation does not merely provide additional context—it fundamentally restructures how the exec agent allocates its effort, shifting the run profile from a probe-then-build pattern toward an earlier, more sustained building phase.

VI Related Work

VI-A LLM-based Code Generation and Program Synthesis

The application of large language models to code synthesis has progressed from template-filling approaches [18] through neural program induction [10] to contemporary autoregressive models trained on hundreds of billions of tokens of source code [8, 27, 31, 19, 25]. These models achieve impressive performance on function-level benchmarks such as HumanEval [8] and MBPP [2], where a docstring and a handful of unit-test examples provide a highly constrained, unambiguous specification. Scaling the same paradigm to repository-level or whole-program tasks reveals a pronounced degradation. SWE-bench [40] demonstrates that resolving real GitHub issues, which requires understanding repository context far harder than synthesizing isolated functions. Agentic frameworks [36, 40, 15, 42] couple LLMs with tool-use loops (shell, file editors, web search) to extend their reach, yet they still rely on a task description written by a human that is treated as a fixed, complete specification. ProgramBench [41] introduces a harder variant: the reference behavior is embodied in a black-box binary, not a text description, forcing the agent to discover the specification rather than consume it. Our work is the first to explicitly design the discovery step as a first-class, reusable pipeline component when building programs from scratch.

VI-B Requirements Elicitation in Software Engineering

Requirements engineering (RE) is among the most studied disciplines in SE [34, 38, 28], with a recurring finding that written artifacts systematically omit behavioral detail, contain ambiguities, and conflict with one another [5, 13], making elicitation through interaction the standard remedy [7, 44]. NLP and LLM techniques have been applied to automate parts of this process, including ambiguity detection [13], traceability recovery [20], and requirements inconsistency detection [3], but these approaches treat requirements as textual artifacts to be processed rather than behaviors to be discovered through interaction. SpecAgent takes the complementary view, instantiating an LLM as an active elicitor that probes a running system rather than analyzing static documents—closest in spirit to behavior-driven development [33, 22], but producing a requirements artifact rather than a test suite.

VI-C Black-box Program Analysis and Specification Extraction

The closest prior work to SpecFirst’s elicitation loop is the line of research on protocol reverse engineering, which probes network binaries to extract message formats and state machines [6, 9]. These approaches share our black-box observation model but produce machine-readable protocol grammars for security analysis rather than natural-language behavioral specifications for code generation. Specification mining [1] and dynamic invariant detection [12] infer formal properties from execution traces, but require either source instrumentation or passive observation of existing traces rather than active, agent-directed probing. LLM-based decompilation [35] recovers source from binaries at the instruction level and requires disassembly access that our setting explicitly prohibits. Program translation approaches [32] operate source-to-source and assume a complete, readable specification is already available. SpecFirst occupies a distinct position: it actively probes a black-box binary under maximal access restrictions and produces a natural-language behavioral specification consumable directly by a downstream LLM code synthesis agent.

VII Discussion

VII-A Failure case analysis

To understand the failure case and identify their root causes, we randomly sample 50 test failure cases across four models and manually label each by examining three artifacts jointly: (i) the failing-test message (what behavior the test exercises and what concrete output, exit code, or error string it expects), (ii) the Spec.md (whether and how precisely the behavior is described), and (iii) the relevant source snippet (whether the implementation contains the corresponding code path and whether it matches Spec.md). We observed the following five types of failures:

F1: Spec Omission (10%). Spec.md never documents the features that the test exercises (e.g., the CLI flag, subcommand, error path, or output channel). It means that the spec agent failed to discover and collect the behavior from the binary.

F2: Wrong Spec Description (4%). Spec.md describes the behavior incorrectly (e.g., attributing a message to stderr when the test asserts stdout). The implementation was faithful to Spec.md, but Spec.md itself contradicted the binary’s actual behavior.

F3: Inaccurate Spec Description (26%). Spec.md documents the feature under test, but the description is insufficiently precise, forcing the implementation agent to guess. For example, Spec.md may document that invalid input produces an error without specifying the exit code or output channel; the implementation chooses rc=1 with a stderr message, while the test expects rc=2 with a stdout message.

F4: Execution Fault (52%). Spec.md describes the behavior correctly and completely, yet the implementation diverges. Common shapes include a subcommand recognized in help text but rejected by the dispatcher, a flag was parsed but its effect never applied, or an error format that contradicts Spec.md’s documented convention. This is the dominant failure mode, indicating that even a correct Spec.md does not guarantee a correct implementation for complex behavioral constraints.

F5: Environment-dependent Error (8%). Failures are environment-dependent (e.g., NS server strings embedded in error messages, missing test-wrapper paths, or subprocess timeouts from absent toolchains). These reflect benchmark infrastructure noise rather than agent deficiencies.

Implications. Although SpecFirst prove the use of spec agent is a promising direction for building a program from scratch. Our failure analysis suggests the following research direction. The dominant failure mode (F4, 52%) indicates that closing the remaining gap requires stronger execution-stage reasoning (e.g., better instruction following, self-consistency checking against Spec.md, and test-driven repair loops), rather than improvements to spec elicitation alone. F1, F2, and F3 (40% combined) suggest that elicitation coverage and behavioral fidelity remains an open problem; future work could further explore targeted probing strategies that systematically exercise under-documented surfaces such as error paths, flag interactions, and output routing.

VII-B Does the specification’s format matter?

To understand the impact of specification format, we compared the following formats, differing only in the deliverable-instruction paragraph of the task prompt; all other prompt text, probing rules, and limits are identical across conditions. Note that all the results reported in RQs are based on Sections.

Sections. The format we used is introduced in Section VII-B, where the agent is asked to write SPEC.md as structured Markdown under named headings.

Freeform. The agent chooses its own structure and coverage criteria. This condition isolates whether imposed structure helps or hurts downstream exec quality.

OpenSpec. The agent writes SPEC.md in the OpenSpec format [14], structuring the document as a ## Purpose section followed by ### Requirement: blocks, each annotated with RFC 2119 keywords (must/shall, should, may), and concrete #### Scenario: examples in GIVEN/WHEN/THEN form. This condition tests whether a more formal, requirements-engineering vocabulary improves the precision of the elicited specification.

We compare the average test pass rate of SpecFirst using the above different format with GPT-5.4-mini on randomly sampled 50 instances²²2We choose to use GPT-5.4-mini and a subset due to budget limit.. Table VI compares the effectiveness of different formats. As shown, every specification format improves on the baseline, while Sections improves the most and outperforms other formats.

媒体内容 · 前往原文查看

Format	Pass Rate (%)
Baseline (No Spec)	55.9%
Freeform	60.7%
OpenSpec	61.7%
Sections	62.6%

VII-C Cost of SpecFirst

媒体内容 · 前往原文查看

Model	Direct-Synthesis	SpecFirst (Spec/Synthesis/Total)
Qwen3.5-397B	$2.56	$0.95 / $2.84 (+11%) / $3.79 (+48%)
Qwen3.6-35B	$2.03	$0.50 / $2.68 (+32%) / $3.17 (+56%)
GPT-5.4-mini	$0.35	$0.25 / $0.29 (-17%) / $0.53 (+51%)
GPT-5.5-high	$2.54	$3.16 / $2.69 (+6%) / $5.85 (+130%)

Table VII reports the mean per-instance cost for Direct-Synthesis and SpecFirst. The total cost of SpecFirst is higher by 48%–130% across models, which is expected: the spec agent constitutes an additional inference phase ($0.25–$3.16 per instance), and the more complete specification produced leads the code synthesis agent to generate larger, more feature-complete implementations, modestly increasing its own cost as well (+6%–+32% for most models). The exception is GPT-5.4-mini, where synthesis cost decreases by 17%, suggesting that a clearer upfront specification reduces exploratory overhead during coding. The largest overhead occurs with GPT-5.5-high (+130%), driven primarily by the cost of the spec agent itself rather than by synthesis. Overall, the cost increase is a direct reflection of SpecFirst doing more useful work—broader behavioral exploration and more complete implementation—rather than redundant computation.

VII-D Threats to Validity

Internal Validity The SpecFirst uses two agents instead of one, meaning it spends more budget than the baseline (without spec agent). The risk is that the improvement is not actually from the value of having a specification, but simply because the model got a larger compute budget. To alleviate this threat, as shown in RQ4, 100% of the runs ended early by choice. Therefore, giving the baseline models more compute wouldn’t have naturally increased their test pass rate.

External Validity Our evaluation covers deterministic command-line programs in ProgramBench; results may not generalize to programs with non-deterministic behavior, graphical interfaces, or complex inter-process communication. We evaluate four models on a single scaffold (SWE-agent); while consistent gains across two model families suggest robustness, results may differ for other models and scaffolds. Although SpecFirst requires an executable binary for spec elicitation, the binary serves as a proxy for any executable oracle that exposes a probe-and-observe interface, such as a running REST API, a containerized service, a compiled SDK, or a remote CLI are all equivalent from the spec agent’s perspective. The key prerequisite is not a binary specifically but any queryable reference or human, a condition met by most software maintenance, migration, and integration scenarios.

Construct Validity The study’s main evaluation metric (a 0–100% scale average test pass rate) combines two different things: whether the agent can actually build a working program, and whether that program behaves correctly. There was a risk that failures such as compilation failures (like a missing compile.sh) would unfairly tank the average scores. However, these total technical failures were extremely rare (at most 3 out of 200 runs). This means when an agent scored a 0%, it was not because it could not build a working program; it was because the program’s actual behavior was completely wrong.

VIII Conclusion

We presented SpecFirst, a two-stage agentic framework that explicitly decouples behavioral specification elicitation from code synthesis for from-scratch program re-implementation. By dedicating a separate spec agent to systematically probe the binary and produce a structured specification prior to implementation, SpecFirst addresses three fundamental limitations of single-loop baselines: insufficient behavioral exploration, specification loss over multi-turns, and uncorrected error propagation. Experiments on all 200 ProgramBench instances across four frontier models demonstrate consistent improvements in test pass rates (6.9%–21.3%) and binary exploration coverage (9.4%–18.5%). Our results suggest that treating requirements elicitation as a first-class phase is a promising paradigm for advancing LLM agents on complex, from-scratch software construction tasks.

References

• [1] G. Ammons, R. Bodík, and J. R. Larus (2002) Mining specifications. ACM Sigplan Notices 37 (1), pp. 4–16. Cited by: §VI-C.
• [2] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. (2021) Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Cited by: §VI-A.
• [3] S. Bashir, A. Ferrari, A. Khan, P. E. Strandberg, Z. Haider, M. Saadatmand, and M. Bohlin (2025) Requirements ambiguity detection and explanation with llms: an industrial study. In 2025 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 620–631. Cited by: §VI-B.
• [4] LiteLLM: call 100+ llms using the openai input/output format Note: https://github.com/BerriAI/litellm External Links: Link Cited by: §IV-F.
• [5] D. M. Berry, E. Kamsties, and M. M. Krieger (2003) From contract drafting to software specification: linguistic sources of ambiguity. Springer, Berlin, Heidelberg. Cited by: §II-B, §VI-B.
• [6] J. Caballero, H. Yin, Z. Liang, and D. Song (2007) Polyglot: automatic extraction of protocol message format using dynamic binary analysis. In Proceedings of the 14th ACM conference on Computer and communications security, pp. 317–329. Cited by: §VI-C.
• [7] E. D. Canedo, A. T. S. Calazans, G. R. S. Silva, P. H. T. Costa, R. P. de Mesquita, and E. T. S. Masson (2022) Creativity and design thinking as facilitators in requirements elicitation. International Journal of Software Engineering and Knowledge Engineering 32 (10), pp. 1527–1558. Cited by: §VI-B.
• [8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. (2021) Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §I, §VI-A.
• [9] P. M. Comparetti, G. Wondracek, C. Kruegel, and E. Kirda (2009) Prospex: protocol specification extraction. In 2009 30th IEEE Symposium on Security and Privacy, pp. 110–125. Cited by: §VI-C.
• [10] J. Devlin, J. Uesato, S. Bhupatiraju, R. Singh, A. Mohamed, and P. Kohli (2017) Robustfill: neural program learning under noisy i/o. In International conference on machine learning, pp. 990–998. Cited by: §VI-A.
• [11] V. Dongre, R. A. Rossi, V. D. Lai, D. S. Yoon, D. Hakkani-Tür, and T. Bui (2025) Drift no more? context equilibria in multi-turn llm interactions. arXiv preprint arXiv:2510.07777. Cited by: §I.
• [12] M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao (2007) The daikon system for dynamic detection of likely invariants. Science of computer programming 69 (1-3), pp. 35–45. Cited by: §VI-C.
• [13] A. Ferrari, G. Lipari, S. Gnesi, and G. O. Spagnolo (2014) Pragmatic ambiguity detection in natural language requirements. In 2014 IEEE 1st International Workshop on Artificial Intelligence for Requirements Engineering (AIRE), pp. 1–8. Cited by: §II-B, §VI-B.
• [14] Fission AI (2026) OpenSpec: a lightweight spec-driven framework for ai-assisted development. Note: Accessed: 2026-06-10 External Links: Link Cited by: §I, §III-C, §VII-B.
• [15] P. Gao, Z. Tian, X. Meng, X. Wang, R. Hu, Y. Xiao, Y. Liu, Z. Zhang, J. Chen, C. Gao, et al. (2025) Trae agent: an llm-based agent for software engineering with test-time scaling. arXiv preprint arXiv:2507.23370. Cited by: §VI-A.
• [16] GitHub (2021) GitHub Copilot: Your AI Pair Programmer. Note: https://github.com/features/copilot Cited by: §I.
• [17] GitHub (2026) GitHub spec kit: toolkit to help you get started with spec-driven development. Note: Accessed: 2026-06-29 External Links: Link Cited by: §I.
• [18] S. Gulwani, O. Polozov, and R. Singh (2017) Program synthesis. Foundations and Trends in Programming Languages 4 (1-2), pp. 1–119. Cited by: §VI-A.
• [19] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, et al. (2024) DeepSeek-coder: when the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. Cited by: §VI-A.
• [20] J. H. Hayes, A. Dekhtyar, and S. K. Sundaram (2006) Advancing candidate link generation for requirements tracing: the study of methods. IEEE Transactions on Software Engineering 32 (1), pp. 4–19. Cited by: §VI-B.
• [21] C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan (2024) Swe-bench: can language models resolve real-world github issues?. In International Conference on Learning Representations, Vol. 2024, pp. 54107–54157. Cited by: §I.
• [22] K. Karhu, T. Repo, O. Taipale, and K. Smolander (2009) Empirical observations on software testing automation. In 2009 International Conference on Software Testing Verification and Validation, pp. 201–209. Cited by: §VI-B.
• [23] P. Laban, H. Hayashi, Y. Zhou, and J. Neville (2025) Llms get lost in multi-turn conversation. arXiv preprint arXiv:2505.06120. Cited by: §I.
• [24] A. B. Labate, V. M. de Sousa, S. R. Fiorini, L. G. Azevedo, R. M. Thiago, and V. T. da Silva (2025) Solving context window overflow in ai agents. arXiv preprint arXiv:2511.22729. Cited by: §I.
• [25] R. Li, L. B. Allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim, et al. (2023) Starcoder: may the source be with you!. arXiv preprint arXiv:2305.06161. Cited by: §VI-A.
• [26] Y. Ma and J. Liu (2025) Quantifying laziness, decoding suboptimality, and context degradation in large language models. arXiv preprint arXiv:2512.20662. Cited by: §I.
• [27] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong (2022) Codegen: an open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. Cited by: §VI-A.
• [28] B. Nuseibeh and S. Easterbrook (2000) Requirements engineering: a roadmap. In Proceedings of the Conference on the Future of Software Engineering, pp. 35–46. Cited by: §I, §VI-B.
• [29] OpenAI (2026-03-17)Introducing gpt-5.4 mini and nano(Website) External Links: Link Cited by: TABLE I.
• [30] OpenAI (2026-04-23)Introducing gpt-5.5(Website) External Links: Link Cited by: TABLE I.
• [31] B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, et al. (2023) Code llama: open foundation models for code. arXiv preprint arXiv:2308.12950. Cited by: §VI-A.
• [32] B. Roziere, M. Lachaux, L. Chanussot, and G. Lample (2020) Unsupervised translation of programming languages. Advances in neural information processing systems 33, pp. 20601–20611. Cited by: §VI-C.
• [33] J. F. Smart and J. Molak (2023) BDD in action: behavior-driven development for the whole software lifecycle. Simon and Schuster. Cited by: §VI-B.
• [34] I. Sommerville (2011) Software engineering, 9/e. Pearson Education India. Cited by: §I, §VI-B.
• [35] H. Tan, Q. Luo, J. Li, and Y. Zhang (2024) Llm4decompile: decompiling binary code with large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3473–3487. Cited by: §VI-C.
• [36] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. (2025) Openhands: an open platform for ai software developers as generalist agents. In International Conference on Learning Representations, Vol. 2025, pp. 65882–65919. Cited by: §VI-A.
• [37] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh, et al. (2025) Openhands: an open platform for ai software developers as generalist agents. In International Conference on Learning Representations, Vol. 2025, pp. 65882–65919. Cited by: §I, §II-A, §III.
• [38] K. Wiegers and J. Beatty (2013) Software requirements. Pearson Education. Cited by: §I, §VI-B.
• [39] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: TABLE I, TABLE I.
• [40] J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press (2024) Swe-agent: agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems 37, pp. 50528–50652. Cited by: §I, §I, §II-A, §III-E, §III, §VI-A.
• [41] J. Yang, K. Lieret, J. Ma, P. Thakkar, D. Pedchenko, S. Sootla, E. McMilin, P. Yin, R. Hou, G. Synnaeve, et al. (2026) ProgramBench: can language models rebuild programs from scratch?. arXiv preprint arXiv:2605.03546. Cited by: §I, §I, §I, §II-A, §IV-B, §IV-C, §IV-E, §VI-A.
• [42] X. Yang, J. Zhou, M. Pacheco, W. Zhu, P. He, S. Wang, K. Liu, and R. Pan (2025) Lingxi: repository-level issue resolution framework enhanced by procedural knowledge guided scaling. arXiv preprint arXiv:2510.11838. Cited by: §III-E, §VI-A.
• [43] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629. Cited by: §III-E.
• [44] D. Zowghi and C. Coulin (2005) Requirements elicitation: a survey of techniques, approaches, and tools. In Engineering and managing software requirements, pp. 19–46. Cited by: §I, §VI-B.