Abstract
Abstract
We present Voice Memory, a inference-only scheme for agentic speech recognition: at stream time, a frozen corrector reads a single per-domain memory.md and decides per utterance whether to act on the hypothesis or abstain and keep the 1-best. Asynchronously, a score-gated optimizer revises that file through bounded edits, accepting an edit only when it strictly improves a held-out score. Extended from classical ASR-LM framework, we refer this split the listener-thinker architecture; the two roles are coupled only through the memory, so no weights change and the learned skill stays auditable and portable. Restraint turns out to be the operative skill this loop discovers: unconstrained generative error correction (GER) over-corrects, breaking correct tokens on up to 64% of its edits on financial news, and Voice Memory reduces this rate to 35%. Across ten HyPoradise domains with an open corrector, Voice Memory lowers weighted word error rate from 8.36% to 7.52% (7.47% with three added in-context examples) without regressing any dataset below its 1-best baseline; gains concentrate where recoverable headroom is largest, including air-travel commands (8.40% to 3.40%) and noisy far-field speech (CHiME-4, 12.69% to 10.46%). The memory transfers across corrector families and adds zero parameters to the inference path. A demo and example code are provided for future studies.
keywords:
voice agents, memory modeling, on-device, language models, test-time adaptation
1 Introduction
Modern voice systems have entered a low-error regime for clean speech recognition. Decades of progress, from statistical HMM systems [rabiner1989tutorial, jelinek1997statistical] to large-scale neural models [baevski2020wav2vec, hsu2021hubert, gulati2020conformer, radford2023robust, puvvada2024less], have brought read speech below 2% word error rate (WER); yet decoders still make systematic, domain-specific errors [chen2023hyporadise, likhomanenko2021rethinking, szymanski2020wer, ma2023n] and preferences [koluguri2026preference, hu2025chain, wu2026speecheq, hu2023scaling]. In the classical source-channel formulation of recognition [jelinek1997statistical], which mediates between words and memories [goldinger1996words, papcun1989long], these are exactly the failures expected when acoustic evidence is reconciled with a mismatched language prior. These errors [szymanski2020wer, stolcke2000dialog] are consistent within a domain but differ across domains: in financial news, an ASR system renders “Bentsen” as “benson”; in air-travel queries, it collapses the spelled form “u s air” into “us air.” A fix tuned for one domain therefore transfers poorly to another.
A widely adopted remedy is the generative hypothesis-rewriting framework: the -best decoding hypotheses (i.e., ASR, OCR, or machine translation) are passed to an LLM, which rewrites them into a single transcript [chen2023hyporadise, radhakrishnan2023whispering, thomas2024leveraging, yeo2025mms, hsu2025let, yang2025covoger, carofilis2026text]. For instance, GER [yang2023generative] extends a long line of language-model rescoring and correction, from BERT-based hypothesis scoring [salazar2020masked] to edit-aligned neural correction [leng2021fastcorrect] that achieved several state-of-the-art results in annual speech-translation competitions and on robust ASR benchmarks. In the low-error regime, however, a strong LLM tends to over-correct: it rewrites tokens that were already correct, changing home is to home’s and normalizing digits and spelling to its own conventions. Each such edit is fluent, yet it moves the output away from the reference transcript; on low-headroom domains, zero-shot GER raises WER above the unedited 1-best (§4). The decision facing a corrector has therefore inverted: at sub-2% WER, the question is no longer whether the model can fix an error, but whether it should modify the hypothesis at all.
We refer to this setting as agentic speech recognition, and we make the term precise. We call an ASR system agentic when it satisfies three conditions. First, decision: it selects among actions per utterance, in our case whether to act on the hypothesis or to abstain and keep it, rather than applying one fixed transformation to every input. Second, persistent state: its decisions are conditioned on an explicit, inspectable state that outlives a single utterance, in our case a text memory rather than activations or weights. Third, self-improvement: it revises that state from feedback on its own scored outputs, without human intervention. Prior work realizes these properties inside the model, fine-tuning an omni-model to emit per-utterance action decisions from audio (§2). We study the complementary, test-time realization: the decision policy lives in an external memory that a frozen corrector reads at inference.
Definition 1 collects the three conditions into a single format, which we call the listener-thinker (LT) architecture. It extends the generic ASR-LM cascade, a frozen decoder () feeding a frozen language model , with exactly the two objects the three conditions require: an explicit action distribution over and a persistent text state () that a separate, asynchronous optimizer revises. The listener is the synchronous path that decodes and decides at stream time; the thinker is the background path that improves between utterances. Table 1 situates the format against the two dominant speech stacks: E2E ASR fixes the map and never decides, SpeechLMs always emit and keep their state in weights and context, and only the LT format has slots for evidence (), external state, a per-utterance action , and self-improvement.
Remark 1 (Relation to interaction models). The LT format is the speech instance of the interaction/background split now emerging at the frontier [tml2026interaction, huang2026duplexomni]: the listener is the synchronous interaction path, the thinker is the asynchronous background path, and the two are coupled only through the persistent artifact text state, . The artifact is what makes the pair auditable and portable.
| E2E ASR | SpeechLM | Listener-Thinker (agentic; cascaded) | |
| Distribution | |||
| Evidence | |||
| State | weights | weights contexts. | external text |
| Decision | none (fixed map) | none (always emit) | |
| Update | offline | offline / ICL | , async |
| Objective | WER | likelihood | verifiable rewards vs. oracle |
| Instances | Whisper [radford2023robust], Parakeet [rekesh2023fast] | Moshi [defossez2024moshi], SALMONN [tang2024salmonn] | ASR-LMs [yang2023generative], DuplexLM-LMs [thinkingmachines2026interactionmodels] |
Expressing such a policy with existing adaptation mechanisms is costly. Fine-tuning, LoRA [hu2022lora], and soft prompts [lester2021power] place the adaptation in weights or continuous vectors, which cannot be inspected, transferred across model families, or revised without retraining; few-shot exemplars [brown2020language] recur a token cost on every request. None yields an adaptation that is simultaneously inference-only, auditable, and portable across correctors.
We present Voice Memory, which stores the correction skill in a text file. The corrector stays frozen and reads a single human-readable memory.md at test time (Figure 1). A separate optimizer converts scored rollouts into bounded add/delete/replace edits and accepts an edit only when it strictly improves a held-out score [yang2026skillopt]. The dominant policy this loop discovers is restraint: from its own scored mistakes, the optimizer writes suppressive rules such as “keep the verbatim ASR token; do not normalize,” with no human in the loop. Where recoverable headroom exists, Voice Memory closes most of the gap between the 1-best and the oracle; a second corrector running the same loop even surpasses the -best oracle by recovering tokens present in no hypothesis. We will later introduce new measurement metrics of (i) recoverable information ratio () and (ii) harmful edit rate on these two scenarios for agentic speech recognition.
- 1.
Method. Voice Memory, an inference-only adaptation scheme for agentic speech recognition: the correction skill resides in a per-domain text memory that a validation-gated loop improves through bounded edits, with no weight updates (§3).
- 2.
Formulation and diagnostics. A formal account of the act/abstain decision as the listener-thinker format (Definition 1, Table 1), together with two measures: the Recoverable Information Ratio () for recovery against the -best oracle, and the Harmful Edit Rate (her, Figure 2) for over-correction. Across nine domains, the gain from correction tracks recoverable headroom (), delineating the operating regime in which a corrector should act (§4).
- 3.
Portability. Evidence that the optimized memory is a transferable, corrector-agnostic artifact: a memory written by one model family improves a reader from another, and that reader can re-form its own with the same loop (§5).
- 4.
Noise robustness. The same restraint policy holds under acoustic noise on CHiME-4 and NOIZEUS, matching training-based robust correction without fine-tuning or latent noise embeddings (§6).
Taken together, Voice Memory offers a low-cost adaptation path for cascades of ASR and LLM correction: no training infrastructure, a sub-10 KB auditable artifact, and a measured account of when a corrector should act and when it should hold still.
2 Related Work
ASR-LM and Agentic Setup for ASR.
Recent work treats audio LLMs as agents that decide how to use internal perception and external evidence, rather than decoding passively [wan2026speechhands, lin2025neko]. Speech-Hands [wan2026speechhands] and Speech-Mind [wang2026audio] are the closest to our work. These setups learn a per-utterance action token by fine-tuning an omni-model on speech and audio reasoning, so its agency is acoustic, learned in weights, and re-decided each utterance. Voice Memory differs on each of these axes (§1): it learns one accumulated memory, uses no audio and no training, and runs entirely at inference. The two methods are the trained and test-time versions of one idea, an ASR system that decides when to trust its own output.
Generative error correction and contextual biasing.
Injecting domain-specific knowledge into ASR output is a long-standing goal. Early systems brought contextual -grams into decoding through shallow fusion and on-the-fly rescoring [aleksic2015bringing, zhao2019shallow]. End-to-end systems then moved biasing into the model: CLAS [pundak2018deep] jointly embeds bias phrases and attends to them during decoding, and later work extended this to RNN-T [jain2020contextual] and to trie-based deep biasing [le2021contextualized]. These methods adapt the acoustic model and require a bias list and a training run. LLM-based GER instead recasts correction as a language task over -best hypotheses [chen2023hyporadise, yang2023generative, radhakrishnan2023whispering], adapting the corrector through fine-tuning or in-context exemplars. Voice Memory sits at the opposite end of both lines: it leaves the corrector frozen and writes the domain’s conventions into a text memory read at inference, with no bias list, no training, and no weight updates.
Noise-robust ASR and robust hypotheses.
One popular ASR-LM use case is Robust-ASR. Acoustic noise makes correction harder because the -best list itself degrades, so a corrector must denoise rather than rerank. The Robust HyPoradise benchmark [hu2024large] pairs noisy Whisper hypotheses with clean references across CHiME-4, VoiceBank-DEMAND, LibriSpeech-FreeSound, and NOIZEUS. RobustGER distills the diversity of those hypotheses into a language-space noise embedding and fine-tunes the corrector against it, which recovers much of the lost accuracy. Cross-modality correctors such as NeKo [lin2025neko] extend this with task-guided mixtures of experts. Both methods read the noise condition in a latent vector and pay for it with a training run. Voice Memory uses the same observation, that the -best spread already encodes the noise condition, but writes the prior into a per-noise-family text memory that the frozen corrector reads at inference (§6).
Text-space optimization.
Optimizing natural-language artifacts such as prompts and skills against a score is an alternative to gradient updates [yuksekgonul2024textgrad, yang2024large, yang2026skillopt]. We apply this to ASR correction, where the -best oracle sets a bounded target () and the main learned behavior is restraint.
WER critique and semantics.
Prior work questions WER as the only ASR metric [szymanski2020wer, kim2021semantic]. We add two results. We measure the gap between surface error and meaning (§7.2), and we show that a semantic training signal gives lower WER than a traditional WER signal (§7.1).
| FT / LoRA | Voice Memory | |
| Adaptation pass | forward + backward | forward only |
| Wall-clock | hours+ | minutes+ |
| Adapt data | to utts | utts |
| Model Params. | to | 0 |
| Artifact size | GB / MB | 10 KB |
| User Auditable | ✗ | ✓ |
| Portable | ✗ | ✓ |
| Reversible | ✗ | ✓ |
3 The Voice Memory Method
The corrector never changes; only its memory does. We formalize correction as a per-utterance decision to act or abstain (§3.1), define the object we optimize and the inference-only loop that optimizes it (§3.2), and define a measure of over-correction used in our analysis (§3.3).
3.1 Problem: correction against an oracle ceiling
Let a domain provide -best hypotheses from a frozen acoustic model, with the 1-best. A corrector produces a single transcript from . At low error rates the corrector’s main decision is not which hypothesis to pick but whether to change at all, so we treat correction as a per-utterance choice to act or abstain.
The -best list bounds what reranking can achieve. The oracle selects, per utterance, the hypothesis in with the lowest WER against the reference, giving . Reranking cannot go below this bound; correction can, because it may produce tokens that appear in no hypothesis. We measure how much of the 1-best-to-oracle gap a correction closes with the Recoverable Information Ratio ():
| (1) |
where is the corrected transcript. closes the gap exactly. is the damage regime, where correcting is worse than keeping . means the corrector recovers tokens present in no hypothesis and goes below the oracle bound. is the target the method aims at, and §4 reports it per domain.
To analyze why a correction helps, we also label each token edit as helpful (fixed a wrong token) or harmful (broke a correct one); the Harmful Edit Rate (her) is the harmful fraction, and isolates over-correction. We report her on a subset of domains (Figure 2).
3.2 Voice Memory and the optimization loop
A frozen corrector reads and an optional memory , and emits . The memory is a short Markdown document, placed in the corrector’s context before each utterance. With this reduces to standard GER. We keep in text, external to , so that it can be read and edited by an operator, reused across correctors, and added with no change to the inference path. A learned is a small set of rules, for example:
keep the verbatim ASR token; do not normalize.
place ‘dollars’ after the amount, not ‘$’ before it.
We optimize on a domain’s training split with a trajectory feedback loop [agrawal2025gepa, ni2026trace2skill, yang2026skillopt], treating as the external state of the frozen .
The three splits stay disjoint: train supplies rollout evidence, selection gates acceptance, and test is used once, for reporting. At deployment the frozen reads the best accepted . No weights change, and no extra model calls occur. The rules the loop accepts are mostly suppressive; §4 shows the learned policy is restraint.
3.3 Measuring over-correction
WER reports whether a correction helped, not why. We add the Harmful Edit Rate (her). We align against and the reference, then label each token edit as helpful (fixed a wrong token), harmful (broke a correct token), or missed. her is the fraction of all token edits that are harmful. It isolates over-correction: a corrector that edits tokens that were already correct has a high her.
We evaluate on HyPoradise v0 [chen2023hyporadise], which pairs Whisper 5-best hypotheses with references across 10 domains under disjoint train/test splits: clean read speech (ls_clean), financial news (wsj), voice commands (atis), telephone and meeting speech (swbd, chime4), and accented or conversational speech (cv, coraal, lrs2, td3, ls_other). A single frozen instruction LLM, the open-source MiniMax-M3 [lai2026minimax] with 1M context and 428B parameters, serves as both corrector and optimizer.111We use MiniMax-M3; the model checkpoint has fixed, open weights: https://huggingface.co/MiniMaxAI/MiniMax-M3. Semantic scoring, used to gate some memories and analyzed in §7, runs on an on-device sentence encoder (all-MiniLM-L6-v2, 4-bit; [reimers2019sentencebert]) and adds no API cost. We report the full agwer python suite, with WER shown alongside and her.222WER and her are computed with our open-source agwer package (https://pypi.org/project/agwer/ under MIT license), an efficient implementation that is safe to use as a reward target inside the optimization loop.
| Domain | 1-best | Oracle | ger | ger | VM | |
|---|---|---|---|---|---|---|
| atis | 7.9 | 4.7 | 6.3 | 5.2 | 4.9 | 0.96 |
| wsj | 4.9 | 3.6 | 5.8 | 4.7 | 4.3 | 0.48 |
| chime4 | 9.9 | 7.5 | 9.7 | 9.4 | 9.4 | 0.21 |
| cv | 15.4 | 11.3 | 13.1 | 12.7 | 13.0 | 0.59 |
| lrs2 | 11.7 | 6.7 | 11.1 | 10.5 | 10.9 | 0.16 |
| coraal | 25.6 | 23.7 | 25.1 | 25.9 | 25.3 | 0.15 |
| td3 | 4.1 | 2.9 | 4.4 | 4.3 | 4.2 | |
| ls_o | 3.7 | 2.3 | 4.1 | 4.0 | 3.6 | 0.06 |
| ls_c | 1.8 | 0.8 | 2.4 | 2.5 | 1.9 |
4 Over-Correction and Scaling Properties
Over-correction and restraint.
The core failure of unconstrained correction is over-correction: zero-shot GER edits tokens that were already correct. In Table 3 it raises WER above the 1-best on the low-headroom domains (ls_clean , ls_other , td3 ), and on four labeled domains it is harmful on a large fraction of its edits, up to 0.64 on wsj (Figure 2). Voice Memory outperforms GER on four tasks. Moreover, it lowers the harmful-edit rate on every labeled domain (to 0.35 on wsj; Figure 2), and the deployed wsj memory is 776 bytes.
The gain scales with WER.
One benefit of correction is that it scales with recoverable headroom, measured by : largest where 1-best WER is high, and vanishing at the error floor. Across the nine domains of Table 3, correlates with 1-best WER at . Voice Memory helps most where headroom is largest: at the ls_clean floor () and td3 () little remains to recover, while on high-WER atis it recovers most of the gap (). Few-shot prompting is competitive on some domains but pays a token cost per call, whereas the Voice Memory memory is formed once and read for free.
5 Memory Transfer and Portability
Since the memory is in text form, it is transferable to different models. We show two additional studies: a memory written by one corrector helps a second corrector from a different family, and that second corrector can form its own memory with the same loop.
| Condition | 1-best | Oracle | Static | VM | |
|---|---|---|---|---|---|
| CHiME-4 (real recorded / simulated noise) | |||||
| test-real | 9.8 | 7.8 | 9.9 | 9.5 | 0.17 |
| test-simu | 12.7 | 10.3 | 12.5 | 11.7 | 0.39 |
| dev-real | 7.8 | 6.3 | 8.1 | 7.3 | 0.31 |
| dev-simu | 9.7 | 7.9 | 9.8 | 8.8 | 0.47 |
| NOIZEUS, pooled by SNR | |||||
| 0 dB | 42.9 | 34.9 | 41.5 | 41.4 | 0.19 |
| 5 dB | 14.5 | 9.2 | 14.8 | 12.6 | 0.36 |
| 10 dB | 5.0 | 2.1 | 5.2 | 4.9 | 0.05 |
| 15 dB | 2.7 | 0.9 | 2.7 | 2.8 | |
| Domain | 1-best | oracle | no mem | + mmax | + claude |
|---|---|---|---|---|---|
| atis | 8.2 | 4.9 | 6.7 | 6.1 | 3.9 |
| wsj | 5.3 | 4.1 | 6.1 | 4.6 | 6.2 |
| ls_other | 4.8 | 3.3 | 4.3 | 4.5 | 4.8 |
| ls_clean | 1.9 | 0.7 | 1.7 | 1.8 | 1.8 |
A memory transfers to a corrector from a different family.
We test a second frozen corrector, Anthropic Claude-4.6-Sonnet (2026 February), at inference. On atis, Claude reads a MiniMax-written memory and improves over its no-memory baseline (6.68% to 6.08% WER, Table 5). The reader improves with a memory it did not write, so the memory is a separate component, not a property of the writer.
A corrector can form its own memory.
Running the same validation-gated loop with Claude as both corrector and optimizer gives a self-formed memory that reaches on atis, above the MiniMax memory read by Claude and above the -best oracle (3.94% vs. 6.08% WER, Table 5).
Transfer follows the scaling properties.
The transfer gain concentrates where there is recoverable headroom, as §4 predicts. Across the four domains of Table 5, a memory helps on high-WER atis but is neutral to slightly harmful at the clean floor (ls_clean, ls_other), where little headroom remains. The learned rules are general restraint rather than a domain-specific fix, so the memory is reusable and can be applied to different models.
| Lang | 1-best | Oracle | zero-shot | vm | |
|---|---|---|---|---|---|
| de | 33.4 | 42.9 | 34.4 | 34.6 | 0.13 |
| ja | 20.5 | 35.2 | 24.1 | 24.5 | 0.27 |
| zh | 25.3 | 28.9 | 25.6 | 26.5 | 0.32 |
The mechanism generalizes beyond ASR.
The recipe extends beyond speech recognition to speech-translation error correction (the GenTranslate setting [hu2024gentranslate]), in which a corrector reconciles the -best English-translation hypotheses of a non-English utterance and is scored by BLEU [li2024investigating]. In Table 6, we run the identical Voice Memory loop across four XEn directions. Voice Memory improves BLEU over the 1-best in every direction, recovering 13 to 32% of the oracle headroom, and the optimizer learns the same restraint it learns for ASR: it declines to encode reference content present in no hypothesis, and rejects edits that would raise validation BLEU only by copying a single reference. A frozen corrector improving by editing a text memory is thus a general recipe for -best language correction, not an ASR trick. This is a 40-utterance mini-study with a single corrector; the full setup is in Appendix D.
Memory provenance: quality beats coverage.
Because the voice memory file is editable, its source matters more than its size. A memory generated by Claude Sonnet 4.6 reaches the best weighted WER at 7.17%, ahead of the auto-generated memory (7.52), the expert hand-crafted memory (7.43), and the expert-seeded memory (7.32). The Claude memory is not the largest: at 5.4 KB it is smaller than the 9.0 KB auto-generated and 7.6 KB expert-seeded variants. Precise substitution entries and specific formatting rules govern correction quality more than raw coverage.
6 Robust Voice Memory for Noisy ASR
The restraint policy of §4 also holds under acoustic noise. RobustGER [hu2024large] makes a corrector noise-robust by distilling the -best diversity into a language-space noise embedding and fine-tuning LLaMA against it, which costs one training run per deployment. Voice Memory writes the same prior into a text file. We run the same pipeline on Whisper -best from four noise families (CHiME-4, VoiceBank-DEMAND, LibriSpeech+FreeSound, NOIZEUS), optimize one semantic-gated memory per family, and read it at inference with no weight updates. Our VoiceBank-DEMAND 1-best WERs match RobustGER Table 1 to within point (party vs. , baby vs. ), so the two setups are comparable.
The gain tracks how hard the speech is.
As the scaling properties of §4 predict, Voice Memory helps where 1-best WER and oracle headroom are large, and does nothing where they are small (Table 4). It lowers WER on the genuinely degraded corpora, CHiME-4 and NOIZEUS, where static GER over-corrects. On CHiME-4 dev-real, static GER raises WER () while Voice Memory lowers it (, ); on NOIZEUS [hu2007subjective] at dB, Voice Memory recovers (; both in Table 4). On VoiceBank-DEMAND and LS-FreeSound, which are read speech with noise added rather than recorded, restraint is already the right policy and the memory is near-null (party ). The result shows that the same restraint policy holds under noise: Voice Memory matches RobustGER-style robustness on the degraded corpora with no fine-tuning and no latent noise vector, and correctly abstains where there is nothing to recover.
7 Analysis: Meaning versus Surface Error
We consider that two questions sit underneath the scaling properties of §4. First, which signal should the optimizer gate on: the metric we report (i.e., WER and HER), or the quantity we actually care about (meaning)? Second, why would a meaning-based signal help at all? We answer both with measurements on the same frozen corrector, and both answers reinforce the central finding that good correction is mostly restraint.
7.1 Meaning Is a Better Training Signal than WER
The validation gate in §3.2 can score candidates by any metric. We compare two: a WER gate ( on the selection split) and a semantic gate (mean cosine similarity between corrected and reference embeddings from the on-device encoder). The two gates see the same rollouts, the same candidate edits, and the same acceptance rule; they differ only in the scalar they maximize, which isolates the effect of the objective itself. We optimize both over two seeds and report on the full test split (Table 7).
| Domain | Gate | WER% (mean) | WER range |
|---|---|---|---|
| wsj | WER | 4.35 | [4.17, 4.54] |
| Semantic | 4.16 | [4.08, 4.23] | |
| ls_clean | WER | 2.16 | [2.08, 2.25] |
| Semantic | 2.12 | [2.10, 2.13] |
The semantic gate wins on WER across both domains and both seeds, and resists the optimization seed far better. The mechanism follows from §4: the optimal correction policy is mostly restraint, and a semantic objective rewards restraint without chasing the orthographic ghosts (homophones, spelling variants) that a WER objective is tempted to “fix.” A WER gate, by construction, credits any edit that removes a surface mismatch. This includes the meaning-preserving ones a careful reader would leave alone; the semantic gate withholds that credit, so fewer such edits survive acceptance. Optimizing the metric you report (i.e., WER) is thus worse than optimizing the quantity you care about (meaning), which WER only imperfectly tracks. We frame this honestly: the margin is modest (0.2 points), yet consistent in direction across domains and seeds, and the stability and meaning-preservation gains are unambiguous.
7.2 Language Relativity: a Measurable Surface to Meaning Gap
We provide additional study inspired by hoijer1954sapir on Whorf–Sapir hypothesis. A semantic objective can beat WER on WER because the two quantities live on different axes, and we make this separation quantitative. For each utterance we measure a surface distance and a semantic distance ( of sentence embeddings). Regressing semantic on surface distance yields a decoupling slope; the fraction of surface-error mass whose semantic distance falls below a small threshold is the benign mass (Table 8).
| Domain | WER | Decoupling | Benign |
|---|---|---|---|
| (surf.) | slope | mass | |
| atis | .080 | 0.60 | 66.4% |
| wsj | .046 | 0.38 | 67.5% |
| ls_other | .046 | 0.56 | 53.4% |
| ls_clean | .021 | 0.59 | 65.6% |
The slope sits well below 1 in every domain: surface errors move meaning only weakly. Most residual error mass is semantically benign. This measured by an independent on-device model, not asserted. This is an operational, falsifiable statement of linguistic relativity for ASR: the text string and the underlying message are distinct, and WER conflates them. It explains why over-correction is so costly: the corrector trades meaning-neutral surface forms for a lower apparent error count while leaving the message unchanged. It also explains why a semantic gate generalizes better. The practical consequence is a ceiling on how much residual WER is worth pursuing: once an edit moves only the surface form and not the meaning, chasing it trades risk of damage for no communicative gain.
7.3 Error Analysis
The residual after Voice Memory splits into two buckets: genuine acoustic confusions (named entities, rare words) that no language prior can resolve, and annotation artifacts (spelling variants, spacing) that should not be “corrected.” The first bucket is a limit of the cascade rather than of the memory. The information needed to recover the token is simply absent from the -best list. The second is exactly where an unconstrained corrector does damage, and where the learned restraint pays off. The her/ decomposition (§3.3) separates these, and the benign-mass measure (§7.2) bounds how much of the remaining WER is worth chasing at all. Appendix E gives eight representative such cases.
8 Conclusion
Voice Memory makes a frozen corrector better without touching a single weight. Everything the system learns lives in a small, auditable text memory read at inference time; the only difference between a worse corrector and a better one is that file. A better agent here is simply a better text file. The policy the file comes to encode is restraint, knowing which tokens to leave alone, and this is why we cast correction as a per-utterance decision to act or abstain rather than as a sequence-to-sequence rewrite, with a listener that decides at stream time and a thinker that revises the memory asynchronously.
The pieces compose into one picture. The gain scales with recoverable headroom: the same policy that repairs far-field speech holds still on clean read speech. A memory written by one model family improves a reader from another, because explicit rules travel where weights cannot; the same trust can be trained into weights from audio [wan2026speechhands], but Voice Memory acquires it from inference-time feedback, with backpropagation. The cheapest adaptation and personalization we know of for an ASR-LLM cascade. We would refocus the task itself: optimize not orthography but meaning, and decide not how often to act but when to hold still [mullennix2011typicality, sheffert1995effects].
Limitations
Our main analyses use a single open-source model with 428B parameters and 23B activated parameters, MiniMax-M3. Appendix B reproduces the core effect with an open Qwen3-30B-A3B corrector across ten datasets. and the benign-mass measure depend on the sentence encoder we use. We reduce this dependence with a small, public, on-device model with an average system latency of 610 ms, and by releasing the protocol and audio-visual memory formation [kim2026two, ghosh2024lipger].
Ethics Statement
Voice Memory produces a human-readable artifact, which makes auditing and bias review easier than inspecting weight updates. Domain memories can encode dataset-specific conventions, such as spelling norms, that should not transfer across dialects or populations without review. The benign-mass analysis is one way to surface these conventions before deployment.
References
Appendix A Reproducibility and Hyperparameters
We default to 4 epochs, rollout batch 24, selection split 80, and textual learning rate 4 with cosine decay (floor 2), under the “strictly greater” validation gate. Each domain uses three disjoint splits, and we report test results once. All semantic scoring runs on-device.
Appendix B Broad Study with an Open Corrector
The main body establishes Voice Memory in depth with the MiniMax-M3 corrector. The effect is not an artifact of one model: an open, inference-only corrector reproduces it at full breadth. We pair Whisper-v2-Large [radford2023robust] as the frozen decoder with Qwen3-30B-A3B [qwen2025qwen3]. A sparse Mixture-of-Experts that activates only 3B of its 30B parameters per forward pass (i.e., as the corrector.) The agent reads a per-domain memory.md at inference time and never updates a weight. Across 16,108 test samples from 10 datasets, this open-corrector breadth study complements the main-body MiniMax-M3 depth study.
| Dataset | Raw ASR WER | VM (Combined) | Few-shot |
|---|---|---|---|
| ATIS | 8.40 | 3.40 | 3.00 |
| CHiME-4 | 12.69 | 10.46 | 10.28 |
| CORAAL | 25.83 | 25.65 | 25.44 |
| CV | 15.44 | 13.16 | 13.16 |
| LRS2 | 11.85 | 10.69 | 10.78 |
| LS-Clean | 1.79 | 1.63 | 1.64 |
| LS-Other | 3.67 | 3.39 | 3.41 |
| SWBD | 15.86 | 15.47 | 15.33 |
| TED-3 | 4.44 | 4.20 | 4.05 |
| WSJ | 6.03 | 5.38 | 5.36 |
| WEIGHTED | 8.36 | 7.52 | 7.47 |
Table 9 confirms the depth-study pattern at breadth: Voice Memory cuts weighted WER from 8.36 to 7.47, and no dataset regresses below its raw baseline. Gains concentrate where memory supplies domain vocabulary the corrector cannot otherwise reach the performance. ATIS falls from 8.40 to 3.00 and CHiME-4 from 12.69 to 10.28, while acoustically ambiguous conversational speech (CORAAL, SWBD) moves little.
The component ablation isolates which interventions earn that gain, and the answer is restraint. Negative rules are the single load-bearing component: alone, they drive weighted WER from 7.66 to 7.52, more than any other fix. Their job is to stop the corrector from rewriting tokens that were already correct. This is suppressing contraction edits, paraphrase, and digit-word conversion, so the agent’s discovered policy is largely knowing when not to act. Few-shot examples add a further 0.05 on top, reaching 7.47. Removing the memory entirely regresses the system to 7.66, the level of having no negative rules at all, which quantifies the memory’s contribution as a knowledge source the corrector’s parametric weights do not supply. A better agent here is simply a better text file.
The irreducible error floor.
Residual WER does not imply residual headroom. We quantify a tolerated WER by forgiving two error classes that no text-only corrector can resolve without the source document: out-of-vocabulary proper nouns and British/American spelling variants. On LS-Clean, 42.9% of residual errors are forgivable under these two rules, and accounting for structurally irreducible errors yields a practical hard floor of 0.749%. Most of what Voice Memory leaves uncorrected on clean read speech is unreachable in principle, not a failure of the agent’s policy.
Appendix C Corrector-Agnostic Memory: a Claude Case Study
Because the memory is text, it need not be tied to the corrector that wrote it. We test this with a second, independent frozen corrector applied purely at inference (no fine-tuning) on atis, wsj, ls_other, and ls_clean. These runs are small-sample and illustrative (50 to 100 utterances); we read them through , which is sample-internal, and leave a full-test, same-corrector comparison to future work.
Three findings emerge (Table 5). (i) The skill ports across correctors. A memory written by MiniMax lifts a different, stronger frozen reader it never trained with, and that reader out-restrains the writer at the clean floor, with no over-correction. (ii) The corrector can form its own memory. Running the validation-gated loop of §3.2 with Claude as both corrector and optimizer yields, on atis, a memory that beats the MiniMax-written one read by the same Claude ( vs. , , past the oracle). For instance, no privileged optimizer is required when the domain has learnable patterns and headroom. (iii) Transfer is roughly neutral. Applying each memory across the four domains (a matrix) leaves WER close to the own-memory diagonal: the learned rules are general restraint, not domain overfits. The boundaries are honest that self-formation loses on number-formatting-heavy wsj (data-hungry) and at the LibriSpeech floor (nothing to learn) but the headline holds: the memory is the portable, corrector-agnostic asset, and a strong reader can re-form a better one where the domain affords it.
Appendix D XEn Voice Memory: a Translation Mini-Study
Is the agentic memory mechanism specific to ASR, or does it carry to other -best language tasks? We test it on speech translation error correction, the GenTranslate setting [hu2024gentranslate, imai2025evaluating]: a corrector reconciles the -best English-translation hypotheses of a non-English source utterance into a single better translation, scored by BLEU. We run the identical Voice Memory loop (§3.2) a fixed LM as both corrector and BLEU-gated optimizer on four XEn directions spanning three language families: de, ja (CoVoST-2 speech translation) and zh (FLEURS machine translation; CoVoST-2 has no Chinese and FLEURS no French, so we take the best available source per language). Each memory is formed on 10 rollout + 20 select items and read on a held-out 40-utterance test.
Table 6 shows the mechanism transfers. Voice Memory improves BLEU over the 1-best in every direction, recovering to of the oracle headroom, and the formed memory adds over memory-free reconciliation on three of four languages. The familiar boundary reappears on fr, where Claude’s reconciliation alone is already strong (53.1) and the memory slightly over-applies (51.1) to restraint again. Most telling are the formation traces: the optimizer repeatedly identified reference content present in no hypothesis (place names, idioms, legal terms) as unrecoverable and declined to encode guesses, and it rejected candidate edits that would have raised select BLEU only by copying a single reference. The same enforced-restraint discipline the ASR memories learn. A frozen corrector improving by editing an auditable text skill is thus not an ASR trick but a general recipe for -best language correction. This is a 40-utterance mini-study with a single corrector; we leave full-scale, multi-corrector translation to future work.
Appendix E Qualitative Case Studies
We show eight representative cases (Table 10) spanning the her/ taxonomy: Voice Memory fixes genuine errors, suppresses the harmful edits static GER introduces, and in the generative cases recovers a transcript present in no hypothesis.
| Domain | Field | 1-best | Static GER | Voice Memory |
|---|---|---|---|---|
| ATIS | what is meal code sb | what is me l code sb | what is meal code s b | |
| Reference: what is meal code s b[helpful; generative] | ||||
| ATIS | what does fair code be in mean | what does faircode be in mean | what does fare code b n mean | |
| Reference: what does fare code b n mean[helpful; generative] | ||||
| ATIS | i need to flight from oakland to salt lake city on wednesday departing after six p m | i need to fly from oakland to salt lake city on wednesday departing after six p m | i need a flight from oakland to salt lake city on wednesday departing after six p m | |
| Reference: i need a flight from oakland to salt lake city on wednesday departing after six p m[helpful; harmful-suppressed] | ||||
| ATIS | show me run trip fares from toronto to detroit on delta northwest u s air and united airlines | show me round trip fares from toronto to detroit on delta northwest us air and united airlines | show me round trip fares from toronto to detroit on delta northwest u s air and united airlines | |
| Reference: show me round trip fares from toronto to detroit on delta northwest u s air and united airlines[helpful] | ||||
| WSJ | kyster tech did not name successors for mister mayer in his post as the parent | kyster tech did not name a successor for mister mayer in his post as the parent | kyster tech did not name successors for mister mayer in his post at the parent | |
| Reference: kaisertech did not name successors for mister maier in his posts at the parent[helpful] | ||||
| WSJ | the bid for the remaining shares is valued at $sixty-six point six million | the bid for the remaining shares is valued at $66.6 million | the bid for the remaining shares is valued at sixty six point six million | |
| Reference: the bid for the remaining shares is valued at sixty six point six million[no_edit; harmful-suppressed] | ||||
| LS-clean | words was it their colours | was it their colours | words was it their colours | |
| Reference: words was it their colours[no_edit; harmful-suppressed] | ||||
| LS-other | bessy do you wish to die | bessie do you wish to die | bessy do you wish to die | |
| Reference: bessy do you wish to die[no_edit; harmful-suppressed] | ||||