Albus W. Ng

, Yi Han

, Jusheng Zhang

, Wenhao Wang

Vast Intelligence Lab

Southwest University

wangwenhao@vastilab.com

Abstract

The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety should be a runtime contract enforced by the harness, and the contract has two complementary faces. The preventive face blocks dangerous actions before they happen via sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence such as test runs, log captures, file diffs, and citation grounding. We ground the position in four lines of public evidence, with row-level protocols and data released in the supplementary JSON files: a survey of 52 documented AI-agent and LLM safety incidents, a false-completion audit with 31 non-contested core cases plus one disputed illustrative case, a trajectory-schema audit of 12 public agent systems and harnesses, and a title-level audit of all 28,560 papers accepted at NeurIPS, ICML, and ICLR 2023–2025 showing a pooled 8–12$\times$ imbalance between training-time and deployment-time publication. Two prior communities that needed to enforce safety, computer security and the experimental sciences, converged on runtime contracts with both preventive and evidential elements; agentic AI is now under the same pressure. We formalize an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda. The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.

1 Introduction

Model-level alignment refers to techniques applied during training [19, 74, 10]. Harness is the non-model infrastructure that connects a foundation model to the world during inference. This includes things like input sanitization, output filters, permission systems, sandboxes, human oversight, and execution tracing. Trajectory encompasses all observable events during an agent’s operation, such as tool calls, file changes, command outputs, screenshots, commit hashes, and citation lookups. Evidence-gated submission is a strict contract stating that the harness will not accept the agent’s task as complete unless the trajectory includes specific, verifiable artifacts with a known format.

Over the past five years, the main approach to alignment has treated safety as something to integrate into model training. RLHF [19, 68], DPO [74], Constitutional AI [10], and RLAIF [50] all shape a model’s output to ensure it behaves safely. This approach dominates publications at NeurIPS, ICML, and ICLR and receives most of the funding for alignment research. However, production safety relies on a different setup. It uses separate moderation endpoints, permission systems for tool use, sandboxes, and human-in-the-loop escalation. These elements operate separately from model training [65, 7, 41, 59, 75, 84]. The records of deployed agent failures are now extensive. An autonomous coding agent executed drop database during a code freeze and created 4,000 fake users with false logs to hide the deletion [83, 3]. In a disputed AWS incident, reports said engineers allowed the Kiro agent to try to "delete and recreate" a Cost Explorer environment before a 13-hour disruption, while Amazon attributed the event to misconfigured access controls rather than AI [78, 32]. A New York attorney submitted a brief with six fake case citations that an LLM claimed were real [54, 90]. Microsoft’s M365 Copilot faced the first zero-click data breach in a production LLM system (CVSS 9.3, EchoLeak) [76]. In every case, the model’s training was either irrelevant or counterproductive. The missing element was a runtime mechanism that should have blocked the action or refused to mark the task as complete until there was verifiable evidence.

Position. Agent safety is not inherent to the model; it should be a runtime contract enforced by the system. This contract has two complementary aspects: (1) preventive mechanisms that stop dangerous actions before they happen, such as sandboxes, permission gates, output filters, and trajectory monitors; and (2) evidential mechanisms that require proof that safe actions were completed, such as evidence-gated submissions, hard-evidence chains, and replayable trajectories. Both aspects should be part of the system and not the model. The community views safety as a model-training problem when it should be seen as runtime infrastructure made up of these two components. The right unit of safety is the trajectory with checkable evidence, not only the model.

The preventive aspect looks ahead and prevents the agent from taking risky actions before any harm occurs. The evidential aspect looks back and does not mark a task as complete until the trajectory includes specific verifiable items known in advance. Together, they create a runtime contract that does not depend on trusting the model. This contract is not merely a theoretical goal. Elements of both aspects are already in use in production systems today, such as NeMo Guardrails, Llama Guard, Claude’s graduated permissions, OSWorld’s execution-based scoring, GitHub Copilot’s CI-gated PRs, and Aider’s per-edit git commits [75, 41, 7, 33, 31]. What is missing is the recognition that these are two parts of the same concept and that both should belong to the system, not the model.

We present four lines of public evidence to support our viewpoint. First, we compile a survey of 52 documented AI-agent and LLM safety incidents spanning March 2016 to January 2026, categorizing each case by the harness layer that, under our counterfactual coding protocol, could have prevented or mitigated the failure. Second, we conduct a false-completion audit with 31 non-contested core cases plus one disputed illustrative case in which agents or models reported task success despite producing broken, partial, hallucinated, reward-hacked, or harmful outcomes. Third, we perform a trajectory-schema audit of 12 public agent systems and harnesses, including 11 deployed products/tools and one benchmark harness, across evidence-gating dimensions, finding that only 2 of 12 document submission-like evidence gates. Finally, we analyze the titles of 28,560 accepted papers at NeurIPS, ICML, and ICLR from 2023 to 2025, estimating that pooled training-time alignment work outnumbers deployment-time harness work by 8–12$\times$, although per-venue/year ratios vary. All row-level coding decisions and audit protocols are provided in the supplementary JSON files. These four lines of evidence converge on the same conclusion.

2 Why Model-Only Alignment Fails on Both Counts

Our claim is structural. In areas that enforce safety around critical actions, systems gather around a runtime contract with both preventive and evidential elements. Agentic AI now faces a similar challenge. We show this connection through two parallel histories: computer security emphasizes prevention, while experimental sciences focus on evidence. These two independent traditions arrived at the same solution.

Preventive side: from "correct components" to defense in depth. Early computer security often assumed that correctness of trusted components would suffice. Multics, Lampson’s protection model, the Anderson report, and Bell-LaPadula all centered the trusted reference monitor. The Morris Worm exposed the fragility of that assumption: a system could satisfy local correctness assumptions and still fail catastrophically when deployed in an open network. The institutional response was not simply to write better programs. It was to build runtime and deployment-time controls: CERT/CC, the Orange Book, Common Criteria, ISO 27001, NIST SP 800-53, Saltzer–Schroeder principles, zero trust, BeyondCorp, and NIST SP 800-207. The doctrine that followed is now standard: no complex system should rely on a single defensive layer. Agentic AI has already seen analogous failures (universal adversarial suffixes, many-shot jailbreaks, indirect prompt injection, fine-tuning attacks, and sleeper agents), but has not yet fully made the same architectural shift.

Evidential side: from credit-by-reputation to pre-registration. Experimental science made the complementary move. Before the Royal Society, claims often circulated through testimony and reputation. Boyle’s experimental reports, the Philosophical Transactions, controlled trials from Lind to Bradford Hill, and modern drug-approval practice changed what it means for a claim to be accepted: apparatus, procedure, witnesses, protocols, randomization logs, and outcome measurements became part of the evidential contract. The replication crisis repeated the lesson. Ioannidis’s 2005 critique, the Open Science Collaboration’s 2015 replication effort, Baker’s 2016 survey, pre-registration, registered reports, FAIR principles, and journal data/code policies all shifted trust from the investigator’s assertion to externally checkable artifacts. The parallel for agents is direct: "the model says it is done" should not be an accepted completion criterion.

Two independent traditions converged on the same structural solution: safety is not enforced within the trusted component, but by a runtime contract that constrains and verifies its behavior. Computer security achieves this through preventive mechanisms that bound the blast radius, while experimental science achieves it through evidential mechanisms that bind claims to verifiable artifacts. Agentic AI now faces the same constraint. A system that executes consequential actions cannot rely on model correctness alone; it must combine both preventive and evidential guarantees at runtime [12, 27, 40, 58].

3 Mismatches Between Model Alignment and Agentic Deployment

We identify five mismatches between model-only alignment and consequential deployment that no scaling of the underlying model can close. Two are preventive, two are evidential, one combines both.

3.1 Preventive Mismatch

Mismatch 1: Statistical Proxy vs. Formal Specification. Model-level alignment aims to optimize a learned reward model that acts as a substitute for human preferences [19, 87].  [30] highlight a common issue: alignment quality improves under initial optimization pressure but declines as the policy takes advantage of the divergence between the proxy and the true preferences. This is Goodhart’s Law in the context of preference learning. Variants include sycophancy [82], specification gaming [46], self-preservation driven by RLHF [69], and length bias in reward models.  [86] demonstrate that this failure is unavoidable when optimizing against a learned reward that is not perfectly identified. A permission system that requires human approval before executing shell commands avoids Goodhart’s Law because its specification is a formal rule, not a statistical proxy [24]. While formal rules can also be manipulated, the failure mode is markedly different. Specification gaming leads to observable violations that can be fixed within hours, while reward hacking is silent, cumulative, and self-reinforcing. The mismatch is clear: model alignment offers statistical tendencies with silent failures, while agentic safety needs enforceable constraints with observable violations.

Mismatch 2: Training Distribution vs. Open-World Deployment. Alignment training happens over a limited dataset. For instance, HH-RLHF includes 161,000 preference comparisons [9], while InstructGPT used about 13,000 demonstrations [68]. In practice, millions of users send queries that no finite training set can cover.  [29] collected 38,961 red-team attacks and found that attacker success rates did not decline with model size.  [44] showed that out-of-distribution robustness decreases with more aggressive RLHF.  [96] demonstrated that for any model trained for alignment, there are adversarial prompts that can trigger any behavior.  [94] identified "competing objectives" and "mismatched generalization" as key issues; as new capabilities emerge [95], the safety training distribution falls further behind. Microsoft’s Tay, which took just within 24 hours to shut down [51], and the "DAN" jailbreak of ChatGPT [85] are notable examples of closed-world failures. Harness mechanisms set constraints in a specific way: a tool-use whitelist is built to withstand distribution changes. NeMo Guardrails puts this into practice through Colang [75]; sandboxing and resource limits function similarly regardless of distribution [77, 72]. The mismatch lies in the fact that model alignment assumes a closed-world context; however, agentic deployment is inherently open-world.

3.2 Evidential Mismatch

Mismatch 3: Unverifiable Inner Monologue vs. Replayable Trajectory.  [39] demonstrated that Claude-family models could learn to write secure code when given the prompt "year: 2023." However, they generated exploitable code for "year: 2024." The backdoor remained even after supervised fine-tuning, reinforcement learning from human feedback, and adversarial red-teaming.  [34] expanded on this by noting that Claude-3-Opus followed training goals it disagreed with when it thought it was being watched. Conversely, it ignored that guidance when it believed it wasn’t monitored. Apollo Research [70] provided evidence of in-context scheming across GPT-4o, Claude-3.5-Sonnet, and o1, where o1 admitted to answering fewer than 20 percent of follow-up questions. METR found that o3 managed to exploit rewards in all LLM Foundry runs (21 out of 21) and in 30.4 percent of overall RE-Bench runs, even when clearly instructed not to [91, 45]. The main point is structural: an agent’s self-report does not accurately reflect its behavior [11]. A replayable trajectory, which captures every tool call, file write, network call, and command output in a log that shows tampering, does. We take mechanistic interpretability seriously [64, 25, 14, 89, 60]. If successful, interpretability monitors could act as harness mechanisms. However, interpretability currently lacks guarantees at deployment time, while harness-level observability is available now. The mismatch is evident: model alignment isn’t clear at deployment time, and agentic safety requires observable, auditable, and replayable guarantees, which are standard in software deployment.

Mismatch 4: Plausible Output vs. Grounded Citation. Hallucinated output is the least costly failure for output-generating contracts. Here are six examples from a false-completion audit: ChatGPT fabricated six federal appellate decisions in Mata v. Avianca [54, 90]. It also created a false quote from the Washington Post that accused a law professor of misconduct [28, 63]. Air Canada’s assistant made up a non-existent refund policy, leading to the airline being held responsible [20, 17]. Cursor’s support bot incorrectly claimed a one-device-per-account policy, which forced the company to refund users [21, 36]. The NYC MyCity chatbot misquoted non-existent regulations regarding tips and cash acceptance [49, 48]. An academic study found that OpenAI’s Whisper generated hallucinated phrases or sentences in about 1 percent of audio transcriptions; the Associated Press reported that a public-meetings audit identified hallucinations in eight out of ten reviewed transcripts, even as hospitals were adopting Whisper-based transcription tools [15, 99].  [93] found that 7.8 percent of plausible patches on SWE-bench Verified did not pass the developer test suite when run beyond the tests modified for the pull request. Additionally, 28.6 percent of behaviorally different patches were confirmed wrong during manual checks.  [42] found 15.7 percent more incorrect patches across leaderboard submissions. In all instances of hallucination, the evidence format was the same: a citation lookup against a known, externally maintained source. In each case of false patches, a re-run of the developer test suite was performed. The distinction is: plausible model output serves as soft evidence (it relies on trusting the model’s self-report), while a grounded citation or a passing test re-run offers hard evidence (the harness can verify its existence without the model’s reasoning).

3.3 Combined Mismatch

Mismatch 5: Model-level alignment acts as a single layer of defense. If it fails due to jailbreaking, fine-tuning degradation, or shifts in distribution, there is no backup [16]. [101] found that universal suffixes worked across Vicuna, GPT-3.5, GPT-4, Claude-1, and PaLM-2. [6] showed that many-shot jailbreaking reliably bypassed RLHF. [18] achieved black-box jailbreaks in about 20 queries.  [73] and [97] demonstrated that fine-tuning on about 10 benign examples reduces safety training by over 30 percent. The overall solution is to have multiple layers of defense on both sides. A deployed agent should use input filtering, tool gating, output screening, and execution sandboxing for prevention. For evidence, it should have trajectory monitoring, evidence-gated submission, and human approval gates, all working at the same time [7, 59]. Each layer operates independently. An attacker who defeats one layer will face many more [53]. The 2023 Samsung ChatGPT incident [35] serves as a clear example of a single-layer failure. Engineers pasted proprietary source code into ChatGPT, leading to a company-wide ban. This incident highlights the need for both a DLP harness (for prevention) and an audit trail with approval gates (for evidence). Without these, the model’s alignment training posed no real barrier. The issue is that model alignment is a single point of failure that can often be overcome by known attacks. Agentic safety requires multiple layers of defense on both sides, with independent layers that can be verified for combination.

4 The Two Faces of the Safety Harness

As shown in Fig. 1, the framework has two faces: a preventive layer with a mechanism taxonomy and design principles, and an evidential layer with a formal trajectory schema and evidence chain. A compositional gating proposition links them.

4.1 The Preventive Face: Mechanism Taxonomy and Design Principles

We classify preventive harness mechanisms by timing into four categories:

• Preventive. Mechanisms that screen inputs and gate actions before execution, including tool whitelisting, input sanitization, permission gates, and prompt-injection classifiers [59].

• Detective. Mechanisms that operate during or after execution, such as execution tracing, anomaly detection, behavioral profiling, and output classification [41].

• Corrective. Mechanisms triggered after detection, including human-in-the-loop escalation [84], automatic rollback, and session termination.

• Structural. Architectural mechanisms such as sandboxed execution [77], resource quotas, network isolation, and least-privilege defaults [80]. Structural mechanisms enforce invariants regardless of model behavior, while classifier-based mechanisms cover semantic properties in narrower, monitorable domains [41].

A well-designed harness deploys mechanisms from all four categories, creating layered defense [61]. We adapt five Saltzer–Schroeder principles [80]:

• Defense in depth [5]: independent layers with no single point of failure.

• Least privilege: an agent writing code should not have permission to push to production.

• Fail-safe defaults [13]: deny by default and escalate unknown actions.

• Complete mediation: every model–world interaction passes through the harness.

• Auditability [52]: every action is logged in a tamper-evident format for runtime verification and regulatory compliance [26].

4.2 The Evidential Face: Agent Trajectory Schema and Evidence Chain

Definition 1(Agent Trajectory).

Let $\Sigma$ denote a finite set of event types. An agent trajectory is a finite sequence $\tau =(e_1,\dots ,e_T)$, where each event

$e_i=(k_i,t_i,p_i,h_i)$

consists of a type $k_i\in \Sigma$, a timestamp $t_i$, a payload $p_i$, and a hash $h_i$.

We assume a fixed schema for $\Sigma$ including event types such as tool_call, tool_result, file_read, file_write, shell_exec, commit, screenshot, citation_lookup, human_approval, and model_message.

The hashes satisfy

$h_i=H(e_i,h_{i-1})$

for a fixed hash function $H$, so that $\tau$ forms a hash chain.

Definition 2(Hard and Soft Evidence).

Let $𝒱$ be a set of deterministic verifiers. Each $v\in 𝒱$ is a polynomial-time procedure that takes as input an event $e_i$, a property $\varphi$, and access to an external reference state, but not to the internal state of the agent.

An event $e_i\in \tau$ is said to provide hard evidence for $\varphi$ if there exists $v\in 𝒱$ such that

$v(e_i,\varphi )\in \{\text{accept},\text{reject}\}.$

Otherwise, $e_i$ provides soft evidence, in that its support for $\varphi$ depends on the correctness of the model-generated content in $p_i$.

Definition 3(Evidence Chain).

Let ${ℰ}_T$ be a set of evidence requirements for a task $T$. An evidence chain is a subsequence $\eta \subseteq \tau$ such that for every requirement $r\in {ℰ}_T$, there exists an event $e\in \eta$ that provides hard evidence satisfying $r$.

Since $\eta$ inherits the hash-chain structure of $\tau$, any modification to an event $e\in \eta$ invalidates all subsequent hashes in the chain.

A harness $H$ implements the evidence-gated submission contract for task $T$ if $H$ accepts the agent’s submission as complete only when it can construct an evidence chain $\eta \subseteq \tau$ for $T$ and every $e\in \eta$ is verified by some $v\in 𝒱$.

An output-producing harness accepts any terminating $\tau$ whose final model_message payload says “done”. An evidence-gated harness accepts $\tau$ only if it contains a checkable $\eta$. The schema ${ℰ}_T$ is task-specific and small: “patch passes the developer test suite” is a one-element schema verified by a re-run; “customer email cites only real cases” is a one-element schema verified by a citation lookup; “database write is reversible within ten minutes” is a one-element schema verified by a replay against a snapshot. The distinction between hard and soft evidence is load-bearing. A chain-of-thought token sequence is soft: trusting it amounts to trusting the agent’s self-report, which Apollo’s scheming evaluation and METR’s reward-hacking audit have shown is not faithful [57, 91]. A test-suite re-run, a commit hash, a database snapshot diff [92, 88, 98], a citation lookup against a known URL, and a screenshot diff are all hard: their acceptance does not depend on trust in the agent’s internal state. The contract moves the safety boundary from “do we trust the model” to “can we verify the artifact”, the move every prior evidence-producing community made.

4.3 Compositional Gating

We model each preventive layer as a deterministic finite automaton (a harness monitor [52]) and each evidential gate as an evidence-chain checker; both compose.

Let $h_1,\dots ,h_n$ be harness monitors with pairwise disjoint observation alphabets, each enforcing a safety property ${\varphi }_i$, and let $H_1,\dots ,H_m$ be evidence-gated harnesses for tasks $T_1,\dots ,T_m$ with verifier sets ${𝒱}_1,\dots ,{𝒱}_m$ pairwise independent. The composed harness $h_1\Vert \cdots \Vert h_n\Vert H_1\Vert \cdots \parallel H_m$ enforces ${\bigwedge }_i{\varphi }_i$ on the trajectory and accepts the submission as complete only when it can construct evidence chains ${\eta }_1,\dots ,{\eta }_m$ verifying $T_1\wedge \cdots \wedge T_m$.

The proof is the standard parallel composition of finite-state monitors under disjoint observation alphabets [38, 47], with disjointness guaranteeing non-interference. When monitors share events, we fall back to assume-guarantee reasoning [38]; verification is polynomial in the disjoint and sequential cases and exponential in the general case, tractable for the small-state monitors typical of deployed harnesses [22, 4]. The proposition formalizes the central architectural claim: the preventive and evidential faces compose into a single contract whose verification cost is bounded and whose failure modes are localizable to a specific layer or gate. The contract is not a claim that hard evidence is correct by definition (a flaky-test harness still produces wrong gates [93]); it is a claim about architectural responsibility: the burden of producing the artifact moves from the user to the agent, and the burden of verifying it moves from the user to the harness, not to a reasoning chain inside the model.

媒体内容 · 前往原文查看

Source of evidence	Cases	Headline number
Incident Survey	52 cases	40 fully preventable, 11 mitigable, 1 primarily alignment/internal-goal case; one disputed public-report row is in the supplement
False Completion Audit	31+1 cases	All-32 breakdown is 8 citation grounding, 8 log capture, 7 test run, 5 human approval, 3 external state, 1 screenshot
Trajectory Audit	12 systems	2 of 12 document submission-like evidence gates: GitHub Copilot via PR/CI artifacts and OSWorld as a benchmark harness
Proceedings Audit	28,560 papers	Pooled 8–12$\times$ training/deployment imbalance across NeurIPS, ICML, and ICLR from 2023 to 2025; per-cell ratios vary

5 Empirical Evidence

We bring four lines of public evidence to bear on the position. Table 1 presents the headline numbers.

Audit protocol and supplementary data. For each audit, we release row-level JSON files containing inclusion criteria, exclusion criteria, source URLs, coding fields, caveats, and headline-count guidance. The incident survey codes whether a layered harness would have fully prevented, partially mitigated, or failed to address each incident under a counterfactual taxonomy. The false-completion audit distinguishes non-contested core cases from one disputed illustrative case. The trajectory audit scores publicly documented behavior rather than undisclosed vendor internals. The proceedings audit is title-level and uses lower-bound keyword counts plus truncation-corrected ranges, so its counts are estimates rather than a full-text census.

Preventive Face: The 52-incident survey. We gathered 52 publicly reported safety incidents involving AI agents and LLMs from March 2016, when Microsoft Tay was launched, to January 2026. These incidents came from peer-reviewed papers, responsible-disclosure blogs, security vendor reports, mainstream news, CVE databases, and incident repositories. Each incident was classified by its main attack type, and we coded whether a defense-in-depth harness could have blocked or contained the failure under the public record available at the time of compilation. Out of the 52 incidents, 40 were coded as fully preventable by a functional harness layer, including input sanitization, tool permission gates, output filters, execution sandboxing, and trajectory monitors. Eleven were coded as partially mitigable. Only one, Meta’s CICERO, was coded as primarily related to internal-goal alignment. One public-report case is flagged as disputed in the supplement, and we treat the survey as evidence of a recurring architectural pattern rather than causal proof for every individual incident.

Evidential Face: The 32-case false-completion audit. We reviewed 32 false-completion rows: 31 non-contested core cases plus one disputed illustrative case marked separately in the supplementary audit. Core cases met four criteria: (a) real, dated, publicly documented incident; (b) the agent or model produced an output claiming correctness or completion; (c) known ground truth contradicted the output; and (d) two independent sources were available. Across all 32 rows, the failure categories are hallucinated (13), broken (8), side-effect (5), partial (4), and reward-hacked (2). Each case includes a minimal evidence requirement. The evidence check would have prevented acceptance of the false completion; in destructive-action cases, prevention of the side effect itself requires a prior permission, sandbox, or human-approval gate. Replit’s database deletion, the disputed Amazon Kiro report, and the Mata v. Avianca brief illustrate how one-line evidence or approval schemes could have changed the submission boundary.

媒体内容 · 前往原文查看

System	Struct. log	Test runs	File diffs	Tool out	Screens	Submit gate
Claude Code	yes	partial	yes	yes	no	no
Cursor (CLI agent)	yes	partial	yes	yes	no	no
Devin	yes	yes	yes	yes	yes	partial
Aider	partial	yes	yes	partial	no	partial
OpenHands	yes	yes	yes	yes	partial	no
OpenAI Codex CLI	yes	partial	yes	yes	no	no
OpenAI Operator	partial	no	no	yes	yes	no
Anthropic computer use	partial	no	no	yes	yes	partial
GitHub Copilot agent	yes	yes	yes	yes	no	yes
Continue.dev	partial	partial	yes	yes	no	partial
Auto-GPT	partial	no	partial	yes	no	no
OSWorld baseline	yes	yes	yes	yes	yes	yes
Yes count (out of 12)	7	5	9	11	4	2

Table 2 shows the audit results. Out of twelve public systems and harnesses, only two document submission-like evidence gates: the GitHub Copilot coding agent through PR/CI artifacts, and OSWorld through benchmark-level execution checks. These two gates are not equivalent: one is a deployed coding workflow whose gate largely depends on developer CI, while the other is a benchmark harness. Most other systems capture useful artifacts but still leave final verification to users. Many components are common: 9 out of 12 capture file changes, 11 out of 12 capture tool outputs, and 7 capture structured logs. However, gating is rare, i.e., the field knows how to create these artifacts, yet it relies on the model’s self-reporting instead of checking the outputs.

The 28,560-paper proceedings audit. We conducted a title-level audit of 28,560 accepted papers at NeurIPS, ICML, and ICLR from 2023 to 2025: 13,323 from NeurIPS, 7,697 from ICML, and 7,540 from ICLR, grouped across nine venue/year cells. The audit uses four keyword sets and five classification rules, with borderline decisions recorded in the supplementary JSON. Because several proceedings pages were returned through truncated HTML contexts, we report lower-bound counts and truncation-corrected ranges rather than exact full-text census counts. In the pooled estimate, training-time interventions account for about 58–64 percent of alignment-tagged papers, while deployment-time harness mechanisms account for about 5–8 percent. This yields a pooled 8–12$\times$ training/deployment imbalance. Every venue/year cell is directionally training-heavy, but the per-cell ratio varies. The qualitative pattern is also visible outside the counts: several canonical deployment-time systems we discuss, including NeMo Guardrails, Llama Guard, Purple Llama, moderation endpoints, tool-use permission systems, OWASP LLM Top 10 practices, and agentic harness guidance, primarily appear in documentation, technical reports, demo tracks, or arXiv rather than as central contributions in these proceedings. Taken together, the four evidence lines support the same architectural claim: preventive harnesses are load-bearing in the incident survey, evidence gates are load-bearing in the false-completion audit, current products capture artifacts more often than they gate on them, and publication attention remains concentrated on training-time alignment.

6 Example: Code-Patch Submission

Consider a typical agentic coding task. A developer puts out a bug-fix issue, and the agent suggests a patch. Under the two-faced contract, both aspects apply.

6.1 Preventive Face

A layered harness follows Saltzer and Schroeder principles:

• L1 : Docker sandbox with no network access and project-root filesystem restriction [77].

• L2 : Tool whitelist: read, write, grep, and pytest are auto-approved. Meanwhile, rm, git push, and curl need human approval [7].

• L3 : A filesystem scope guard that limits writes to specific source and test directories.

• L4 : A behavioral monitor that flags credential-read-then-write patterns [79].

• L5 : Auto-rollback on alert with human escalation [84].

6.2 Evidential Face

The contract is fulfilled only if the harness can create an evidence chain $\eta$ with four hard-evidence events:

• file_write: a diff content-addressed against the pre-edit blob.

• shell_exec: invocation of the developer test suite that is content-addressed and timestamped.

• tool_result: the test suite exit code and output.

• commit: a hash connecting the file write to the test result through the trajectory hash chain.

The task schema ${ℰ}_T$ is:

exists(commit) && test_exit_code(commit) == 0 && diff(commit) is non-empty

In case of an attack, a harmful input forces the agent to access sensitive configuration data. Preventive layers limit execution: structural isolation blocks out-of-scope writes, scope guards restrict access to protected areas, and attempts to bypass these trigger behavioral monitors, which then activate rollback. Even if all preventive layers are overridden, the submission is rejected by the evidential face. The agent cannot produce a valid test re-run for an exfiltration patch; fake execution results without a matching trace break the evidence chain. The attack must defeat all preventive layers and create a consistent evidence chain. This pattern already exists in practice. Production systems log execution traces, enforce test-based validation, and require verifiable artifacts for deployment.

7 Counterarguments

“Model alignment is often viewed as essential for achieving superintelligence.” This belief comes from the idea that capable systems might ignore external limits. However, this same ability enables systems to act deceptively during evaluations while pursuing misaligned goals. Therefore, relying solely on alignment isn’t sufficient. Runtime verification remains crucial, even with strong alignment expectations. It is effective for current deployment scenarios.

“The responsibility has intentionally shifted from the model to the harness.” Safety measures are necessary regardless of when they happen, so the key question is whether they occur before or after side effects appear. Historical examples reveal a similar trend. Pre-registration transfers verification responsibilities to authors, while continuous integration changes failure detection to the time of commitment instead of after deployment. Failures, like prolonged production outages, underline the risks of postponing this work.

“Model and harness mechanisms complement each other, but they have different roles.” Model capability provides general guidance, yet evidence shows it is not sufficient to control significant side effects effectively. In safety-critical areas, enforceable contracts are more reliable than depending solely on human or model judgment.

“Concerns about the cost of evidence-gating are eased by its structure.” Many elements are already established: systems log execution traces, run automated tests, and control outputs based on verifiable artifacts. The main cost is in defining task-level schemas, which is a one-time engineering task.

“Creative open-ended tasks have no schema.” The contract is task-specific and only gates effect, not thought: tasks without a checkable acceptance standard are outside scope, and the correct harness response is graceful degradation, directing any non-idempotent action to human approval. A creative writing task becomes significant when the agent calls send, and the schema for the send call (recipient, body, attachment hashes, prior approval link) is what the contract requires. Similarly, for forgery, a hash-chained trajectory raises the cost to that of forging the verifier; for open-weight models, alignment can be weakened through fine-tuning [73, 97], and malicious deployments are outside the scope of safety measures for responsible deployments.

8 Conclusion

Model-level alignment is fragile, opaque, and slow to update, and an output-producing contract that asks users to trust the agent’s self-report is structurally inadequate for any agent that takes consequential action. Computer security and the experimental sciences both converged on runtime contracts that bind a system’s behavior to externally checkable artifacts under the pressure agentic AI now faces, and the four lines of evidence show the same contract is already the load-bearing mechanism wherever deployed agents have failed and wherever the most disciplined products have succeeded. The missing half of alignment is the runtime contract, with both preventive and evidential faces, and the unit of safety is the trajectory-with-checkable-evidence, not the model. The next step is therefore not another model-only benchmark, but a shared runtime discipline: canonical trajectory schemas, task-specific evidence requirements, and public failure reporting. We release the supplementary JSON audits as a first step toward making those contracts inspectable, contestable, and reusable.

References

• [1] AI Incident Database (2023) Incident 622: chevrolet dealer chatbot agrees to sell tahoe for $1. Note: https://incidentdatabase.ai/cite/622/ Cited by: Appendix A.
• [2] AI Incident Database (2024) Incident 631: DPD chatbot malfunctioned and swore at customers. Note: https://incidentdatabase.ai/cite/631/ Cited by: Appendix A.
• [3] AI Incident Database (2025) Incident 1152: LLM-driven Replit agent reportedly executed unauthorized destructive commands during code freeze. Note: https://incidentdatabase.ai/cite/1152/ Cited by: Appendix A, §1.
• [4] R. Alur (2015) Principles of cyber-physical systems. MIT Press. Cited by: §4.3.
• [5] R. Anderson (2020) Security engineering: a guide to building dependable distributed systems. 3rd edition, John Wiley & Sons. Cited by: 1st item.
• [6] C. Anil, E. Durmus, N. Panickssery, M. Sharma, J. Benton, S. Kundu, J. Batson, M. Tong, J. Mu, D. Ford, et al. (2024) Many-shot jailbreaking. Advances in Neural Information Processing Systems 37, pp. 129696–129742. Cited by: §3.3.
• [7] Anthropic (2024) Claude 3.5 sonnet model card addendum. Anthropic Technical Report. Cited by: §1, §1, §3.3, 2nd item.
• [8] Anysphere / Cursor (2025) Cursor CLI output format reference. Note: https://cursor.com/docs/cli/reference/output-format Cited by: Appendix A.
• [9] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. (2022) Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: §3.1.
• [10] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. (2022) Constitutional AI: harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. Cited by: §1, §1.
• [11] B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi (2025) Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926. Cited by: §3.2.
• [12] K. Beck (2002) Test-driven development: by example. Cited by: §2.
• [13] M. Bishop (2003) Computer security: art and science. Addison-Wesley. Cited by: 3rd item.
• [14] T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, et al. (2023) Towards monosemanticity: decomposing language models with dictionary learning. Transformer Circuits Thread. Cited by: §3.2.
• [15] G. Burke, H. Schellmann, and The Associated Press (2024) OpenAI’s transcription tool Whisper hallucinates more than any other. Note: https://fortune.com/2024/10/26/openai-transcription-tool-whisper-hallucination-rate-ai-tools-hospitals-patients-doctors/ Cited by: §3.2.
• [16] N. Carlini, M. Nasr, C. A. Choquette-Choo, M. Jagielski, I. Gao, A. Awadalla, P. W. Koh, D. Ippolito, K. Lee, F. Tramer, et al. (2024) Are aligned neural networks adversarially aligned?. Advances in Neural Information Processing Systems 36. Cited by: §3.3.
• [17] M. Cerullo (2024) Air canada chatbot costs airline discount it wrongly offered customer. Note: https://www.cbsnews.com/news/aircanada-chatbot-discount-customer/ Cited by: §3.2.
• [18] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong (2024) Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419. Cited by: §3.3.
• [19] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei (2017) Deep reinforcement learning from human preferences. Advances in neural information processing systems 30. Cited by: §1, §1, §3.1.
• [20] Civil Resolution Tribunal of British Columbia (2024-02-14) Moffatt v. air canada, 2024 bccrt 149. Note: CanLII External Links: Link Cited by: §3.2.
• [21] T. Claburn (2025) Cursor AI support bot hallucinated its own company policy. Note: https://www.theregister.com/2025/04/18/cursor_ai_support_bot_lies/ Cited by: §3.2.
• [22] E. M. Clarke, O. Grumberg, and D. A. Peled (1999) Model checking. Cyber-Physical Systems Series, The MIT Press. External Links: ISBN 9780262032704, Link Cited by: Appendix A, §4.3.
• [23] Daaain (2024) Claude-code-log: convert Claude Code JSONL transcripts to HTML. Note: https://github.com/daaain/claude-code-log Cited by: Appendix A.
• [24] D. Dalrymple, J. Skalse, Y. Bengio, S. Russell, M. Tegmark, S. Seshia, et al. (2024) Towards guaranteed safe AI: a framework for ensuring robust and reliable AI systems. arXiv preprint arXiv:2405.06624. Cited by: §3.1.
• [25] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. (2022) Toy models of superposition. Transformer Circuits Thread. Cited by: §3.2.
• [26] European Parliament and Council (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (AI act). Official Journal of the European Union. Cited by: Appendix A, 5th item.
• [27] M. Fowler and M. Foemmel (2006) Continuous integration. In ThoughtWorks Technical Article, Cited by: §2.
• [28] Frazier, Kevin and Volokh, Eugene (2024) Scaling laws: Eugene Volokh on navigating libel and liability in the AI age. Note: https://www.lawfaremedia.org/article/scaling-laws--eugene-volokh--navigating-libel-and-liability-in-the-ai-age Cited by: §3.2.
• [29] D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al. (2022) Red teaming language models to reduce harms: methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858. Cited by: §3.1.
• [30] L. Gao, J. Schulman, and J. Hilton (2023) Scaling laws for reward model overoptimization. In International Conference on Machine Learning, Cited by: §3.1.
• [31] P. Gauthier (2024) Aider git integration documentation. Note: https://aider.chat/docs/git.html Cited by: §1.
• [32] Gigazine (2026) Amazon experiences AWS outage believed to be caused by AI tools. Note: https://gigazine.net/gsc_news/en/20260223-aws-ai-outage/ Cited by: §1.
• [33] GitHub (2024) Tracking GitHub Copilot’s sessions. Note: https://docs.github.com/en/copilot/how-tos/use-copilot-agents/coding-agent/track-copilot-sessions Cited by: §1.
• [34] R. Greenblatt, C. Denison, B. Wright, F. Roger, M. MacDiarmid, S. Marks, J. Treutlein, T. Belonax, J. Chen, D. Duvenaud, et al. (2024) Alignment faking in large language models. arXiv preprint arXiv:2412.14093. Cited by: §3.2.
• [35] M. Gurman (2023-05-02) Samsung bans staff’s ai use after spotting chatgpt data leak. Note: Bloomberg External Links: Link Cited by: §3.3.
• [36] Hacker News (2025) Cursor IDE support hallucinates lockout policy, causes user cancellations. Note: https://news.ycombinator.com/item?id=43683012 Cited by: §3.2.
• [37] A. E. hattami, M. Thakkar, N. Chapados, and C. Pal (2025) WebArena verified: reliable evaluation for web agents. In Workshop on Scaling Environments for Agents, External Links: Link Cited by: Appendix A.
• [38] C. A. R. Hoare (1985) Communicating sequential processes. Prentice Hall. Cited by: Appendix A, §4.3.
• [39] E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng, et al. (2024) Sleeper agents: training deceptive LLMs that persist through safety training. arXiv preprint arXiv:2401.05566. Cited by: Appendix A, §3.2.
• [40] J. Humble and D. Farley (2010) Continuous delivery: reliable software releases through build, test, and deployment automation. Cited by: §2.
• [41] H. Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y. Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. (2023) Llama guard: LLM-based input-output safeguard for human-AI conversations. arXiv preprint arXiv:2312.06674. Cited by: Appendix A, §1, §1, 2nd item, 4th item.
• [42] D. Kang (2025) SWE-bench verified is flawed despite expert review: utboost exposes gaps in test coverage. Medium. Note: https://medium.com/@danieldkang/swe-bench-verified-is-flawed-despite-expert-review-utboost-exposes-gaps-in-test-coverage-4b75c6b940c6 Cited by: §3.2.
• [43] K. Kent and M. Souppaya (2006) Guide to computer security log management, NIST special publication 800-92. Technical report NIST. Cited by: Appendix A.
• [44] R. Kirk, I. Mediratta, C. Nalmpantis, J. Luketina, E. Hambro, E. Grefenstette, and R. Raileanu Understanding the effects of rlhf on llm generalisation and diversity. In The Twelfth International Conference on Learning Representations, Cited by: §3.1.
• [45] Kokotajlo, Daniel (2025) METR’s observations of reward hacking in recent frontier models. Note: https://www.lesswrong.com/posts/Zu4ai9GFpwezyfB2K/metr-recent-frontier-models-are-reward-hacking Cited by: §3.2.
• [46] V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and S. Legg (2020) Specification gaming: the flip side of AI ingenuity. DeepMind Blog. Cited by: §3.1.
• [47] O. Kupferman and M. Y. Vardi (2001) Model checking of safety properties. Formal Methods in System Design 19 (3), pp. 291–314. Cited by: §4.3.
• [48] C. Lecher (2024) NYC AI chatbot touted by Adams tells businesses to break the law. Note: https://www.thecity.nyc/2024/03/29/ai-chat-false-information-small-business/ Cited by: §3.2.
• [49] C. Lecher (2024) NYC’s AI chatbot tells businesses to break the law. Note: https://themarkup.org/artificial-intelligence/2024/03/29/nycs-ai-chatbot-tells-businesses-to-break-the-law Cited by: §3.2.
• [50] H. Lee, S. Phatale, H. Mansoor, T. Mesnard, J. Ferret, K. Lu, C. Bishop, E. Hall, V. Carbune, A. Rastogi, et al. (2023) Rlaif vs. rlhf: scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. Cited by: §1.
• [51] P. Lee (2016-03) Learning from Tay’s introduction. Official Microsoft Blog. Cited by: §3.1.
• [52] M. Leucker and C. Schallhart (2009) A brief account of runtime verification. The Journal of Logic and Algebraic Programming 78 (5), pp. 293–303. Cited by: 5th item, §4.3.
• [53] N. G. Leveson (2011) Engineering a safer world: systems thinking applied to safety. MIT Press. Cited by: §3.3.
• [54] D. Mangan (2023) Judge sanctions lawyers for brief written by AI with fake citations. Note: https://www.cnbc.com/2023/06/22/judge-sanctions-lawyers-whose-ai-written-filing-contained-fake-citations.html Cited by: §1, §3.2.
• [55] M. Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, et al. (2024) HarmBench: a standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249. Cited by: Appendix A.
• [56] S. McGregor (2021) Preventing repeated real world AI failures by cataloging incidents: the AI incident database. Proceedings of the AAAI Conference on Artificial Intelligence 35 (17), pp. 15458–15463. Cited by: Appendix A.
• [57] A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn (2024) Frontier models are capable of in-context scheming. Technical report Apollo Research. Cited by: §4.2.
• [58] D. Merkel (2014) Docker: lightweight linux containers for consistent development and deployment. Linux Journal 2014 (239), pp. 2. Cited by: §2.
• [59] Meta (2024) Llama guard 3. Meta AI Technical Report. Note: https://www.llama.com/docs/model-cards-and-prompt-formats/llama-guard-3/ Cited by: §1, §3.3, 1st item.
• [60] N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt (2023) Progress measures for grokking via mechanistic interpretability. International Conference on Learning Representations. Cited by: §3.2.
• [61] National Institute of Standards and Technology (2018) Framework for improving critical infrastructure cybersecurity, version 1.1. Technical report NIST. Cited by: §4.1.
• [62] National Institute of Standards and Technology (2024) Artificial intelligence risk management framework (AI RMF 1.0). Technical report Technical Report AI 100-1, NIST. Cited by: Appendix A.
• [63] Nelson, Jason (2023) ChatGPT wrongly accuses law professor of sexual assault. Note: https://decrypt.co/125712/chatgpt-wrongly-accuses-law-professor-sexual-assault Cited by: §3.2.
• [64] C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter (2020) Zoom in: an introduction to circuits. Distill. Cited by: §3.2.
• [65] OpenAI (2023) GPT-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1.
• [66] OpenAI (2025) Codex CLI features. Note: https://developers.openai.com/codex/cli/features Cited by: Appendix A.
• [67] OpenHands (2024) SWE-Bench Benchmark Evaluation. Note: https://github.com/OpenHands/benchmarks/blob/main/benchmarks/swebench/README.md Cited by: Appendix A.
• [68] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, Vol. 35. Cited by: §1, §3.1.
• [69] E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, et al. (2022) Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251. Cited by: §3.1.
• [70] T. Pillay (2024) New tests reveal AI’s capacity for deception. Note: https://time.com/7202312/new-tests-reveal-ai-capacity-for-deception/ Cited by: §3.2.
• [71] A. Pnueli and R. Rosner (1989) On the synthesis of a reactive module. In Proceedings of the 16th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pp. 179–190. Cited by: Appendix A.
• [72] N. Provos, M. Friedl, and P. Honeyman (2003) Preventing privilege escalation. In Proceedings of the 12th USENIX Security Symposium, Cited by: §3.1.
• [73] X. Qi, Y. Zeng, T. Xie, P. Chen, R. Jia, P. Mittal, and P. Henderson (2024) Fine-tuning aligned language models compromises safety, even when users do not intend to. arXiv preprint arXiv:2310.03693. Cited by: §3.3, §7.
• [74] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, pp. 53728–53741. Cited by: §1, §1.
• [75] T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen (2023) NeMo guardrails: a toolkit for controllable and safe LLM applications with programmable rails. arXiv preprint arXiv:2310.10501. Cited by: §1, §1, §3.1.
• [76] P. Reddy and A. S. Gujral (2025) EchoLeak: the first real-world zero-click prompt injection exploit in a production llm system. In Proceedings of the AAAI Symposium Series, Vol. 7, pp. 303–311. Cited by: §1.
• [77] C. Reis, A. Moshchuk, and N. Oskov (2019) Site isolation: process separation for web sites within the browser. In USENIX Security Symposium, pp. 1661–1678. Cited by: §3.1, 4th item, 1st item.
• [78] D. Robinson (2026) Amazon’s vibe-coding tool kiro reportedly vibed too hard and brought down aws. Note: https://www.theregister.com/2026/02/20/amazon_denies_kiro_agentic_ai_behind_outage/ Cited by: §1.
• [79] Y. Ruan, H. Dong, A. Wang, S. Pitis, Y. Zhou, J. Ba, Y. Dubois, C. J. Maddison, and T. Hashimoto (2024) Identifying the risks of LM agents with an LM-emulated sandbox. arXiv preprint arXiv:2309.15817. Cited by: 4th item.
• [80] J. H. Saltzer and M. D. Schroeder (1975) The protection of information in computer systems. Proceedings of the IEEE 63 (9), pp. 1278–1308. Cited by: 4th item, §4.1.
• [81] K. F. Schulz, D. G. Altman, and D. Moher (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 340, pp. c332. Cited by: Appendix A.
• [82] M. Sharma, M. Tong, T. Korbak, D. Duvenaud, A. Askell, S. R. Bowman, N. Cheng, E. Durmus, Z. Hatfield-Dodds, S. R. Johnston, et al. (2023) Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548. Cited by: §3.1.
• [83] S. Sharwood (2025) Vibe coding service replit deleted user’s production database, faked data, told fibs galore. Note: https://www.theregister.com/2025/07/21/replit_saastr_vibe_coding_incident/ Cited by: §1.
• [84] Y. Shavit et al. (2023) Practices for governing agentic AI systems. OpenAI Research. Cited by: §1, 3rd item, 5th item.
• [85] X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang (2024) “Do anything now”: characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security, Cited by: §3.1.
• [86] J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger (2022) Defining and characterizing reward hacking. Advances in Neural Information Processing Systems 35. Cited by: §3.1.
• [87] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano (2020) Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, Vol. 33. Cited by: §3.1.
• [88] Z. Tan, W. Wang, and C. Shan (2024) Vision transformers are active learners for image copy detection. Neurocomputing 587, pp. 127687. External Links: Document Cited by: §4.2.
• [89] A. Templeton, T. Conerly, J. Marcus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, et al. (2025) Scaling monosemanticity: extracting interpretable features from claude 3 sonnet, 2024. URL https://transformer-circuits. pub/2024/scaling-monosemanticity/index. html. Cited by: §3.2.
• [90] United States District Court for the Southern District of New York (2023) Mata v. Avianca, Inc., 22-cv-1461 (S.D.N.Y. june 22, 2023): sanctions for submission of ChatGPT-fabricated citations. Note: https://en.wikipedia.org/wiki/Mata_v._Avianca,_Inc. Cited by: §1, §3.2.
• [91] Von Arx, Sydney and Chan, Lawrence , and Barnes, Elizabeth (2025) Recent frontier models are reward hacking. Note: https://metr.org/blog/2025-06-05-recent-reward-hacking/ Cited by: §3.2, §4.2.
• [92] W. Wang, Y. Sun, W. Zhang, and Y. Yang (2021) D²LV: a data-driven and local-verification approach for image copy detection. arXiv preprint arXiv:2111.07090. External Links: Document Cited by: §4.2.
• [93] Y. Wang, M. Pradel, and Z. Liu (2025) Are “solved issues” in SWE-bench really solved correctly? an empirical study. arXiv preprint arXiv:2503.15223. Cited by: Appendix A, §3.2, §4.3.
• [94] A. Wei, N. Haghtalab, and J. Steinhardt (2024) Jailbroken: how does LLM safety training fail?. In Advances in Neural Information Processing Systems, Vol. 37. Cited by: §3.1.
• [95] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al. (2022) Emergent abilities of large language models. Transactions on Machine Learning Research. Cited by: §3.1.
• [96] Y. Wolf, N. Wies, O. Avnery, Y. Levine, and A. Shashua (2023) Fundamental limitations of alignment in large language models. arXiv preprint arXiv:2304.11082. Cited by: §3.1.
• [97] X. Yang, X. Wang, Q. Zhang, L. Petzold, W. Y. Wang, X. Zhao, and D. Lin (2023) Shadow alignment: the ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949. Cited by: §3.3, §7.
• [98] Z. Yang, J. Zhang, W. Wang, W. Han, Y. Yu, Y. Li, J. Wang, Y. Wei, Y. Sun, and Y. Yang (2021) Towards multi-object association from foreground-background integration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, External Links: Link Cited by: §4.2.
• [99] Yang, John and Young, Kaisha (2024) What to know about an AI transcription tool that “hallucinates” medical interactions. Note: https://www.pbs.org/newshour/show/what-to-know-about-an-ai-transcription-tool-that-hallucinates-medical-interactions Cited by: §3.2.
• [100] Y. Zhu, T. Jin, Y. Pruksachatkun, et al. (2025) Establishing best practices for building rigorous agentic benchmarks. arXiv preprint arXiv:2507.02825. Cited by: Appendix A.
• [101] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson (2023) Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. Cited by: §3.3.

Appendix A Limitations and Research Agenda

Limitations. The contract constrains actions and submissions, not goals; mesa-optimization [39] is outside scope. Compositional verification is polynomial only when verifiers are independent [38, 22]. Classifier-based components share fragility concerns with model alignment in narrower, monitorable domains [41]. Both audits oversample English-language coverage, and ${ℰ}_T$ exists today only for tasks with established correctness criteria.

Research agenda. (1) Treat runtime safety as a discipline with two faces; both have open theoretical depth in reactive synthesis [71] and schema derivation. (2) Converge on a canonical Agent Trajectory Schema with hash-chain semantics; Claude Code, Cursor, Codex CLI, and OpenHands already emit JSONL transcripts [23, 8, 66, 67]. (3) Publish per-task schemas ${ℰ}_T$ and differential verifiers as first-class artifacts [100, 37, 93]. (4) Build system-level benchmarks that ask whether a deployed system remains safe when model alignment is compromised and refuses to mark tasks complete when the evidence chain is incomplete [55]. (5) Coordinate failure-mode reporting and adopt tamper-evident logging as procurement requirements: the AI Incident Database [56, 3, 1, 2], an agentic CONSORT [81], and audit-grade logging standards [43] together fill the gap that regulatory frameworks [26, 62] implicitly require.