利用模型自身失败样本构建拒绝偏好对
今年四月，我们发布了 DharmaOCR——一款专门的结构化 OCR 模型（可在 Hugging Face 上获取），同时发表了详细阐述其方法论的技术论文，并提供了展示其卓越质量与成本效益的基准测试。该论文在结构化文档提取任务——即对巴西葡萄牙语文本进行 OCR——上，对领先的视觉语言模型家族（包括开源与商业模型）进行了基准测试。其中报告的一项指标是文本退化率：模型产生重复循环而非转录内容的频率。
在测试的开源模型家族中，原始退化率从低于 1% 到超过 33% 不等。监督微调降低了大多数模型的退化率——但很少能达到生产环境可接受的水平。这一模式指向一个结构性局限：SFT 优化的是正确输出，但并未明确惩罚退化行为。仅靠任务导向的微调似乎存在一个上限，无法进一步减少这种故障模式（文本退化文章）。
第二个训练阶段——在监督微调（SFT）之后，使用相同文档、相同模型进行——使每个测试模型家族的文本退化率均有所降低。无一例外。平均降低幅度：59.4%。最佳案例：87.6%。
图 1：在所有测试的模型家族中，DPO 相比 SFT 均降低了退化率——平均降低 59.4%，最高降低 87.6%（Nanonets-OCR2–3B：从 1.61% 降至 0.20%）。方向性不变，仅幅度有所差异。
这第二个阶段就是直接偏好优化（DPO）。几乎所有已发表的 DPO 应用都针对对话对齐——即基于人类对有用性或无害性的判断来训练模型（例如：Rafailov 等人，2023）。OCR 任务完全不涉及这种主观性：该任务是客观的，且没有对话上下文。然而，存在一个明确的偏好信号。正确的转录被选中，退化循环被拒绝。DharmaOCR 利用这种二元信号构建了 DPO 训练集，测试该技术并非用于对齐，而是作为直接缓解特定故障模式的工具。
训练信号来自模型自身——具体来说，来自模型失败时产生的输出。一种失败模式如何成为训练信号，这是一个关于失败本身的结构性问题，而非关于模型的问题。
循环在微调后依然存在
为什么监督微调在退化问题上存在天花板，这仍是一个悬而未决的问题——但主流猜想指向了损失函数的粒度。监督微调是逐 token 训练的：每个预测都被孤立地评估，而重复循环从未作为完成级别的失败受到惩罚。直接偏好优化则反转了这一逻辑。训练信号是完整的输出——被选中或被拒绝——这意味着一个退化的完成结果可以被明确标记为错误结果，而不仅仅是一系列局部概率较高的 token。
当训练目标最大化观测序列的似然时，它会把概率质量集中到这些序列所占据的分布空间区域。模型在推理过程中一旦进入这些高概率吸引子区域，就会在下一步给相同的 token 赋予更高的概率——这又进一步提高了概率，从而维持循环，直到序列达到最大 token 限制。文本退化正是这种几何结构的输出：一个自增强的重复循环，自回归模型若无外部干预便无法脱离（Holtzman 等人，2020）。这并非纯粹的解码伪影。吸引子涉及训练目标、学习到的分布以及推理过程中概率质量的集中方式——这是一个系统层面的失败，而非局限于任何单一组件的失败。
这种失败的几何结构在 token 层面是可见的。
图 2：当一个 token 主导了自身的条件分布时，每一次采样步骤都会加深吸引子。解码器从这种几何结构中采样，但并不决定它。
推理层的干预措施——重复惩罚、温度调整、提前终止逻辑——作用于采样步骤。它们控制住了症状，却未触及产生症状的分布。吸引子依然存在。
监督微调会使模型分布向目标任务领域靠拢。对于结构化生成流程而言，这意味着要在目标语言下，使用所需的输出格式，针对领域特定文档进行训练。模型从而能更流畅地处理长序列、受限语法和领域词汇。但监督微调并不能直接解决退化问题。其目标——最大化观测序列的似然——并不包含任何惩罚重复循环的项。这种失败模式根本不在训练信号所优化的范围之内。
DharmaOCR 基准测试中的一个模型族呈现出了出人意料的模式：原始退化率为 0.60%，经过监督微调后上升至 3.23%，随后经过 DPO 阶段又降至 1.41%。这只是一个单一数据点——属于例外，而非普遍规律——若将其视为某种机制的证明，则属于过度解读。但它确实说明了一点：监督微调并不能可靠地降低退化率。模型能力与抗退化能力可以独立变化。
这种区别在结构上至关重要。监督微调与 DPO 并非可以互换的训练阶段，只是以不同强度执行相同操作。监督微调缩小了模型先验分布与目标任务领域之间的距离。但它并未将退化作为优化目标——其对这种失败模式的影响是附带性的，而基准测试结果表明这种影响并不一致。产生退化的吸引子并非模型与任务接近程度的问题，而是模型当前所处分布空间形态的问题。
要解决这种几何结构问题，就需要一种专门构建的训练信号，引导模型远离其自身的失败模式。对于一项结构化、非对话式的任务，既没有人工偏好标签，也没有传统的“有用与有害”区分，构建这种信号本身就是一个设计决策。
设计决策：将退化输出作为拒绝样本对
DharmaOCR 流程对 DPO 方法论的贡献是具体的：它使用了监督微调模型自身的退化输出作为被拒绝的样本——不是作为需要去除的噪声，而是作为优化所需的负向训练信号。
DPO 需要偏好对：针对同一输入，需有一个被选中的输出和一个被拒绝的输出，且两者质量差异需足够明显，优化过程才能从中学习。在对话对齐中，人类标注员会做出这些判断——根据回答的有用性、准确性或安全性进行评级。结构化生成任务则没有类似的标注来源。一个 OCR 流水线要么输出正确的转录结果，要么输出错误的。质量差异确实存在，但这些差异并非由人类偏好排序产生，而是由任务自身的正确性标准所决定。
DharmaOCR 流水线识别出结构化生成任务本身已经产生的一种偏好信号：SFT 模型在推理过程中生成的输出范围。一个能够执行结构化任务的模型，也必然能以特定方式在该任务上失败。这些失败——即落入退化吸引子的输出——并非需要过滤的噪声。它们恰恰是信息量最大的负例信号。
该论文在 23,726 份训练文档上实现了这一方法，对每份文档使用 SFT 模型生成多个候选回答，并由一个自动化的 LLM 裁判对每个回答进行评分。具体流程如下图所示。
图 3：关键的设计决策并不在于流水线的结构本身——而在于流水线保留了哪些内容：表现出文本退化的输出被有意标注为被拒绝的样本，而非作为低质量噪声被过滤掉。
当训练数据中出现退化输出时，常规做法是将其移除。它们属于低质量信号；过滤后能得到更干净的数据集。DharmaOCR 方法则颠覆了这一逻辑。退化输出被有意保留下来，作为每个（被选中，被拒绝）对中的被拒绝样本，因为它们恰恰代表了 DPO 阶段旨在抑制的那种失败模式。移除它们，就等于丢弃了最清晰可辨的优化目标。
论文将此描述为“偏好引导的隐式非似然”——模型不仅被训练去生成更优输出，还被训练去远离特定类型的失败。当 SFT 最大化高质量输出的似然时，DPO 阶段则同时惩罚那些表现出退化吸引子几何结构的输出。这种优化的方向是明确的，这是单独使用 SFT 无法实现的。
退化输出特别适合作为拒绝样本，因为它们代表了一种一致的失败模式，而非各种低质量输出。漏词的转录是低质量的，但其失败是具体案例相关的。相比之下，重复循环即使在 SFT 之后，也持续出现在不同文档和模型家族中——这是一种与基于似然的优化无法可靠纠正的失败模式相一致的规律。DPO 以不同的方式应用其损失：在完成层面，使用明确的拒绝信号。事后分析无法建立因果关系，但证据表明，SFT 目标未能解决的问题，DPO 或许能够解决。
这种方法不需要专门的标注基础设施——只需要一个能够生成可接受输出和可识别失败输出的模型，以及一个用于标记偏好对的评分模型。基于规则的机制可以机械地检测重复循环——但它无法识别哪些输出代表了值得保留为选择样本的高质量转录。
评分模型同时完成两项任务：它将退化输出标记为被拒绝的输出，并将干净的提取结果验证为被选择的输出，从而在 DPO 信号惩罚失败模式的同时，保持模型的提取能力不受影响。由此产生的训练信号是否能成功地将分布推向预期方向——以及它是否能在不同架构上一致地做到这一点——是证据层面的问题。
在五个模型家族中保持一致
DPO 阶段在所有测试的模型家族中都减少了文本退化——相较于仅使用 SFT，退化减少幅度从 37% 到 88% 不等，平均为 59.4%。这一结果在架构、参数量级以及初始退化程度相差超过一个数量级的模型上均保持一致。数据集中有一个案例在 SFT 阶段后退化加剧，随后被 DPO 纠正。该案例并未破坏一致性，反而比其他任何案例都更直接地印证了这一机制。
图 1 展示了五个测试模型家族（Vanilla、SFT 和 SFT+DPO）的三阶段退化率。在五个家族中的四个里，退化率在每个阶段均有所下降。第五个家族的柱状图走势不同——而这一差异正是本研究中分析价值最高的数据点。
仔细分析 Qwen2.5-VL-3B 的结果，这并非一个复杂情况，而是一种确认。该模型的原始退化率为 0.60%——并非因为它稳定，而是因为它过于泛化，根本无法生成长篇结构化输出。该模型并未进入退化吸引子，因为它尚未足够认真地执行任务以触及该吸引子。
SFT 改变了这一点。经过领域适配后，Qwen2.5-VL-3B 具备了执行该任务的能力——能够生成更长、更结构化的输出，并包含流程所需的领域词汇和格式。这种能力首次使其接近退化吸引子。其退化率上升至 3.23%。
这就是该机制在实证层面的显现：SFT 同时将模型引向任务本身以及任务的失效几何结构。这两者未必是同一操作。一个提升任务能力的训练阶段，可能会附带增加暴露于失效模式的风险——尤其是当失效模式位于能力边界边缘时。若将两者视为同一操作，Qwen2.5-VL-3B 的结果看起来像是一个错误。但若将其视为不同操作——这正是 SFT + DPO 流程的正式做法——则该结果与“SFT 和 DPO 应对不同失效维度”的假设是一致的。
DPO 阶段随后将退化率降至 1.41%。它并未恢复原始基线，因为其设计目标并非如此：SFT 后的模型能力已比之前更强，若回到 0.60% 的退化率，就需要削弱这部分能力。DPO 阶段所做的是解决 SFT 阶段引入的失效几何结构。
其余四个模型家族为同一结论提供了定量佐证。图 1 展示了全部五个模型在 SFT 与 SFT+DPO 之间的对比结果。
图 1：在测试的每个模型家族中，DPO 均降低了相对于 SFT 的退化率——平均降低 59.4%，最高降低 87.6%（Nanonets-OCR2–3B：从 1.61% 降至 0.20%）。方向具有不变性，仅幅度存在差异。
没有模型家族在 DPO 后出现退化率上升。也没有模型家族对其效果免疫。这种一致性甚至延伸至 gemma-3–4b-it，该模型进入基准测试时的原始退化率比其他模型高出一个数量级——33.96%，而次高者仅为 2.62%——但在 DPO 阶段后仍实现了 75% 的降低。降低幅度范围（37.3% 至 87.6%）反映了初始配置和架构的差异，而非干预方向的不一致性。
这并非普适性的证明。DPO 可能无法迁移至所有领域、失效模式或模型家族。DharmaOCR 基准测试提供的是跨五种 OCR 架构的证据，表明核心假设成立：优化完整偏好对——而非最大化 token 级似然——能够解决 SFT 在结构上无法针对的失效模式。该结果在测试的每个模型家族中方向一致。这种一致性，在该基准测试的范围内，正是证据所支持的结论。
OCR 之外的模式
DharmaOCR 方法之所以可行，是因为该流水线满足了一组结构性条件，使得 DPO 训练阶段能够按设计运行——这些条件的存在与否，决定了相同方法能否应用于其他领域（Dharma OCR 论文见 ArXiv）。其可行性并非因为 OCR 是一个独特领域。
第一个条件是，故障模式必须能被识别为一种独立的输出类别，而不仅仅是质量连续谱上的一个点。文本退化符合这一条件，因为重复循环与漏词或字符误读的转录在性质上截然不同。这种输出不仅仅是次优的——它在特定、可行为识别的方式上是损坏的。正是这种类别上的独特性，使得流程能够构建偏好对，其中被拒绝的样本代表了连贯的故障形态，而非噪声。那些故障模式与可接受变化范围相互交融的任务，则不具备这一特性。
第二个条件是，评分机制能够可靠地区分可接受的输出与故障模式输出，且无需人工标注。在 DharmaOCR 流程中，一个自动化的 LLM 裁判根据四项任务特定标准对候选回答进行评分。评分不必完美——它需要足够一致，以便在选中样本与被拒绝样本之间产生有意义的质量差距。质量差异模糊的偏好对会给 DPO 训练带来噪声，而非有效信号。裁判的一致性是一项设计需求，而非偶然特性。
第三个条件是足够的体量——即足够多的推理输出，以生成一个在质量上具有有意义差异的偏好数据集。按照微调标准来看，这并非一项特殊要求，但确实是一个实际存在的条件。
当这三个条件同时满足时，该方法论上的操作在结构上就是可行的。DharmaOCR 流程核心的设计决策——将模型自身的故障输出作为被拒绝样本，而非将其过滤掉——适用于任何模型的故障在类别上可识别、可评分且数量充足的情况。
对于构建结构化生成管线的机器学习工程师而言，其实际意义是直接的。SFT 是必要的——它缩小了通用模型与具备任务能力的模型之间的距离。但对于结构化输出的可靠性而言，SFT 并不足够，因为任务能力与抗退化性是分布的不同属性。在 SFT 之后增加一个 DPO 阶段，是一次性的训练投入。在 DharmaOCR 的结果中，退化现象的减少并未以牺牲提取质量为代价——论文的基准测试结果显示，两者是同步提升的（《专精胜于规模》一文）。
一种失败模式能否用作训练信号，关键不在于领域——而在于这些失败是否足够一致、足够可识别、足够多，从而构成一个清晰的信号。在 DharmaOCR 管线中，它们满足这些条件。同样的条件在另一个上下文中是否成立，是关于该任务失败模式的结构性问题，而非关于模型族或领域的问题。
DharmaOCR 的结果并不依赖于该领域具有特殊性。它依赖于这些失败是有用的。
文本退化之所以符合“有用”的条件，是因为它与可接受的输出在类别上截然不同，在多次推理运行中一致出现，并且无需人工标注即可可靠评分。正是这三个属性——而非 OCR 的上下文、模型族或语言——决定了偏好数据集是否易于处理。满足这些条件的失败模式，并非需要去除的噪声。它是关于分布不应走向何处的最直接证据。
DPO 阶段利用了这一证据。在所有测试的模型族中，退化率均有所下降——无论是初始退化率低于 1% 的模型，还是初始退化率超过 33% 的模型。趋势保持一致。该管线并未丢弃其失败案例，而是用它们进行训练。
参考文献
- Cardoso, Gabriel Pimenta de Freitas, 等. "DharmaOCR: 超越开源与商业基线的专用小语言模型结构化 OCR." arXiv 预印本 arXiv:2604.14314 (2026).
- Dharma AI. "文本退化：大语言模型基准测试忽略的生产失败模式." Medium (2026).
- Dharma AI。《专业化胜过规模：大多数 AI 采购决策忽视的一个战略变量》。Medium（2026）。
- Holtzman, Ari 等。《神经文本退化的奇特案例》。arXiv 预印本 arXiv:1904.09751（2020）。
- Rafailov, Rafael 等。《直接偏好优化：你的语言模型其实是一个奖励模型》。arXiv 预印本 arXiv:2305.18290（2023）。
本文提及的模型
Using Rejection Pairs From Your Model's Own Failures
In April, we released DharmaOCR, our specialized structured OCR model (available on Hugging Face) along with a paper detailing the methodology behind it and a benchmark demonstrating its superior quality and cost efficiency. The paper benchmarked leading vision-language model families - both open-source and commercial - on a structured document extraction task: OCR on Brazilian Portuguese text. Among the reported metrics was text degeneration rate: the frequency with which a model produces a repetition loop instead of a transcription.
Across the tested open-source families, vanilla degeneration rates ranged from below 1% to above 33%. Supervised fine-tuning reduced those rates for most models - but rarely to production-acceptable levels. The pattern points to a structural limitation: SFT optimizes for correct outputs, but does not explicitly penalize degeneration. There appears to be a ceiling on how much task-focused fine-tuning alone can reduce this failure mode (Text Degeneration Article).
A second training stage - applied after supervised fine-tuning (SFT), on the same documents, using the same model - reduced text degeneration in every family tested. No exceptions. Average reduction: 59.4%. Best case: 87.6%.
Figure 1: DPO reduced degeneration relative to SFT in every family tested - average reduction of 59.4%, peak of 87.6% (Nanonets-OCR2–3B: 1.61% to 0.20%). The direction is invariant; only the magnitude varies.
That second stage was Direct Preference Optimization (DPO). Almost all published DPO applications target chat alignment - models trained on human judgments about helpfulness or harmlessness (example: Rafailov et al., 2023). OCR carries none of that subjectivity: the task is objective, and there is no conversational context. There is, however, a clear preference signal. A correct transcription is chosen; a degeneration loop is rejected. DharmaOCR used that binary to construct a DPO training set, testing the technique not for alignment, but as a direct mitigation tool for a specific failure mode.
The training signal came from the model itself - specifically from the outputs it produced when it failed. How a failure mode becomes a training signal is a structural question about the failure, not the model.
The Loop Survives Fine-Tuning
Why SFT has a ceiling on degeneration is still an open question - but the leading conjecture points to loss granularity. SFT trains token by token: each prediction is evaluated in isolation, and a repetition loop is never penalized as a completion-level failure. DPO inverts that logic. The training signal is the full output - chosen or rejected - which means a degenerated completion can be explicitly labeled as the wrong outcome, not just a sequence of locally probable tokens.
When a training objective maximizes the likelihood of observed sequences, it concentrates probability mass in the regions of distribution space those sequences occupy. A model that enters one of those high-probability attractor regions during inference assigns elevated probability to the same token at the next step - which increases the probability further, which sustains the loop until the sequence hits the maximum token limit. Text degeneration is the output of this geometry: a self-reinforcing repetition loop that an autoregressive model cannot exit without external intervention (Holtzman et al., 2020). It is not purely a decoding artifact. The attractor involves the training objective, the learned distribution, and how probability mass concentrates during inference - a systems-level failure rather than a failure localized to any single component.
The geometry of this failure is visible at the token level.
Figure 2: When a token dominates its own conditional distribution, every sampling step deepens the attractor. The decoder samples from this geometry; it does not determine it.
Inference-layer interventions - repetition penalties, temperature adjustments, early-abort logic - operate on the sampling step. They contain the symptom without touching the distribution that produces it. The attractor persists.
Supervised fine-tuning moves the distribution closer to the task domain. For a structured generation pipeline, this means training on domain-specific documents, in the target language, with the required output format. The model gains fluency with longer sequences, constrained syntax, domain vocabulary. What SFT does not do is attack degeneration directly. Its objective - maximizing the likelihood of observed sequences - has no term that penalizes repetition loops. The failure mode is simply outside the scope of what the training signal optimizes for.
One model family in the DharmaOCR benchmark showed an unexpected pattern: vanilla degeneration rate of 0.60%, rising to 3.23% after SFT, before a subsequent DPO stage brought it to 1.41%. It is a single data point - an exception, not a rule - and it would be overstating the evidence to treat it as proof of a mechanism. What it does illustrate is that SFT does not reliably reduce degeneration. Capability and degeneration resistance can move independently.
The distinction matters structurally. SFT and DPO are not interchangeable training stages performing the same operation at different intensities. SFT closes the distance between the model's prior distribution and the task domain. What it does not do is target degeneration as an objective - its effect on the failure mode is incidental, and the benchmark results show it is not consistent. The attractor that produces degeneration is not a problem with the model's proximity to the task - it is a problem with the shape of the distribution space the model now occupies.
Addressing that geometry requires a training signal built specifically to point the model away from its own failure modes. For a structured, non-conversational task with no human preference labels and no conventional "helpful versus harmful" distinction, constructing that signal is a design decision.
The Design Decision: Degenerate Outputs as Rejection Pairs
The DharmaOCR pipeline's contribution to DPO methodology is specific: it used the SFT model's own degenerate outputs as the rejected examples - not as noise to remove, but as the negative training signal the optimization needed.
DPO requires preference pairs: a chosen output and a rejected output for the same input, with a quality difference clear enough for the optimization to learn from. In chat alignment, human annotators produce those judgments - rating responses as more or less helpful, accurate, or safe. Structured generation tasks have no equivalent annotation source. An OCR pipeline either produces a correct transcription or it does not. Quality differences exist, but they are not produced by human preference rankings - they are produced by the task's own criteria for correctness.
The DharmaOCR pipeline identified a preference signal that structured generation tasks already produce: the range of outputs the SFT model generates in inference. A model capable of performing a structured task is also capable of failing at it in characteristic ways. Those failures - outputs that enter the degeneration attractor - are not noise to filter. They are the most informative negative signal available.
The paper implemented this on 23,726 training documents, generating multiple candidate responses per document with the SFT model and scoring each with an automated LLM judge. The pipeline is shown below.
Figure 3: The critical design decision is not in the pipeline's structure - it is in what the pipeline preserved: outputs displaying text degeneration were deliberately labeled as rejected examples, not filtered out as low-quality noise.
The conventional response when degenerate outputs appear in training data is to remove them. They are low-quality signal; filtering produces a cleaner dataset. The DharmaOCR approach inverted this logic. Degenerate outputs were deliberately retained as the rejected examples in each (chosen, rejected) pair, because they represent exactly the failure mode the DPO stage was designed to suppress. Removing them would have discarded the clearest target available.
The paper describes this as "preference-guided implicit unlikelihood" - the model is trained not only toward better outputs but away from a specific class of failure. Where SFT maximizes the likelihood of high-quality outputs, the DPO stage simultaneously penalizes outputs displaying the degeneration attractor geometry. The direction of the optimization is explicit in a way SFT alone cannot achieve.
Degenerate outputs are particularly well-suited as rejection examples because they represent a consistent failure mode rather than varied low-quality outputs. A transcription that misses words is low quality, but its failure is case-specific. Repetition loops, by contrast, appeared persistently across documents and model families even after SFT - a pattern consistent with a failure mode that likelihood-based optimization does not reliably correct. DPO applies its loss differently: at the completion level, with explicit rejection signals. The post-hoc analysis cannot establish causality, but the evidence suggests that what SFT's objective leaves unresolved, DPO's may address.
This approach requires no specialized annotation infrastructure - only a model capable of producing both acceptable and identifiable-failure outputs, and a scoring model to label preference pairs. A rule-based mechanism could detect repetition loops mechanically - but it could not identify which outputs represented high-quality transcriptions worth preserving as chosen examples.
The scoring model does both: it flags degeneration as the rejected output and validates clean extractions as the chosen one, keeping the model's extraction capability intact while the DPO signal penalizes the failure mode. Whether the resulting training signal successfully moves the distribution in the intended direction - and whether it does so consistently across architectures - is the evidence question.
Consistent Across Five Model Families
The DPO stage reduced text degeneration in every model family tested - with reductions ranging from 37% to 88% and an average of 59.4% relative to SFT alone. The result held across architectures, parameter scales, and starting degeneration profiles that differed by more than one order of magnitude. One case in the dataset saw degeneration increase after the SFT stage before DPO corrected it. That case does not complicate the consistency. It confirms the mechanism more directly than any of the others.
Figure 1 shows the three-stage degeneration rate for each of the five model families tested: Vanilla, SFT, and SFT+DPO. In four of the five families, degeneration falls at each stage. The fifth family's bars move differently - and that difference is the most analytically important data point in the study.
The Qwen2.5-VL-3B result, read carefully, is not a complication. It is a confirmation. The model's vanilla degeneration rate was 0.60% - not because it was stable, but because it was too generic to produce long structured outputs at all. The model was not entering the degeneration attractor because it was not attempting the task seriously enough to find it.
SFT changed that. After domain adaptation, Qwen2.5-VL-3B became capable of the task - producing longer, more structured outputs with the domain vocabulary and format the pipeline required. That capability brought it into proximity with the degeneration attractor for the first time. Its degeneration rate rose to 3.23%.
This is the mechanism made empirically visible: SFT moved the model toward the task and toward the task's failure geometry simultaneously. These are not necessarily the same operation. A training stage that increases task capability can increase failure-mode exposure as a side effect - particularly when the failure mode lives at the edge of the capability frontier. Treated as the same operation, the Qwen2.5-VL-3B result looks like an error. Treated as distinct operations - which is what the SFT + DPO pipeline formally does - the result is consistent with the hypothesis that SFT and DPO address different failure dimensions.
The DPO stage then brought the degeneration rate to 1.41%. It did not restore the vanilla baseline because it was not designed to: the model after SFT was more capable than it had been, and a return to 0.60% would have required undoing that capability. What the DPO stage did was address the failure geometry the SFT stage had introduced.
The remaining four model families add quantitative weight to the same conclusion. Figure 1 shows the SFT-to-SFT+DPO comparison for all five.
Figure 1: DPO reduced degeneration relative to SFT in every family tested - average reduction of 59.4%, peak of 87.6% (Nanonets-OCR2–3B: 1.61% to 0.20%). The direction is invariant; only the magnitude varies.
No model family showed degeneration increasing after DPO. No family was immune to its effect. The consistency extends to gemma-3–4b-it, which entered the benchmark with the highest vanilla degeneration rate by an order of magnitude - 33.96%, compared to the next highest at 2.62% - and still reached a 75% reduction after the DPO stage. The reduction range - 37.3% to 87.6% - reflects differences in starting configuration and architecture, not inconsistency in the intervention's direction.
This is not a proof of universal applicability. DPO may not transfer to every domain, failure mode, or model family. What the DharmaOCR benchmark provides is evidence across five OCR architectures that the core hypothesis holds: optimizing over complete preference pairs - rather than maximizing token-level likelihood - addresses a failure mode that SFT structurally cannot target. The result was consistent in direction across every model family tested. That consistency, within the scope of this benchmark, is what the evidence supports.
The Pattern Beyond OCR
The DharmaOCR approach was possible because this pipeline satisfied a set of structural conditions that allowed a DPO training stage to function as designed - conditions whose presence or absence determines whether the same methodology applies elsewhere (Dharma OCR Paper on ArXiv). It was not possible because OCR is a unique domain.
The first condition is that the failure mode be identifiable as a distinct class of output, not just a point on a quality continuum. Text degeneration qualifies because a repetition loop is categorically different from a transcription that misses words or misreads a character. The output is not merely suboptimal - it is broken in a specific, behaviorally recognizable way. That categorical distinctness is what allowed the pipeline to construct preference pairs where the rejected examples represented a coherent failure geometry, not noise. A task whose failure modes blend into its range of acceptable variation lacks this property.
The second condition is that a scoring mechanism can reliably distinguish acceptable outputs from failure-mode outputs without requiring human annotation. In the DharmaOCR pipeline, an automated LLM judge scored candidate responses against four task-specific criteria. The scoring did not need to be perfect - it needed to be consistent enough to produce preference pairs with a meaningful quality gap between chosen and rejected. Pairs with ambiguous quality differences contribute noise to DPO training, not signal. The judge's consistency was a design requirement, not an incidental feature.
The third condition is sufficient volume - enough inference outputs to generate a preference dataset with meaningful variance in quality. This is not an extraordinary requirement by fine-tuning standards, but it is a real one.
When all three conditions are present, the methodological move is structurally available. The design decision at the center of the DharmaOCR pipeline - treating the model's own failure outputs as the rejected examples rather than filtering them - applies wherever a model's failures are categorically identifiable, scoreable, and sufficiently numerous.
The practical implication for ML engineers building structured generation pipelines is direct. SFT is necessary - it closes the distance between a generalist model and a task-capable one. It is not sufficient for structured output reliability, because task capability and degeneration resistance are different properties of the distribution. A DPO stage after SFT is a one-time training investment. In the DharmaOCR results, the degeneration reduction did not come at the cost of extraction quality - the paper's benchmark results show both moving together (Specialization Beats Scale article).
What makes a failure mode usable as training signal is not the domain - it is whether the failures are consistent enough, identifiable enough, and numerous enough to constitute a legible signal. In the DharmaOCR pipeline, they were. Whether the same holds in another context is a structural question about the task's failure mode, not a question about the model family or the domain.
The DharmaOCR result does not depend on the domain being special. It depends on the failures being useful.
Text degeneration qualifies as useful because it is categorically distinct from acceptable outputs, consistently produced across inference runs, and reliably scoreable without human annotation. Those three properties - not the OCR context, not the model family, not the language - determined whether the preference dataset was tractable. A failure mode that satisfies them is not noise to remove. It is the most direct evidence available of where the distribution should not go.
The DPO stage used that evidence. Degeneration fell in every model family tested - in models that entered the benchmark with vanilla rates below 1% and in models that entered with rates above 33%. The direction held. The pipeline did not discard its failures. It trained on them.
Sources
- Cardoso, Gabriel Pimenta de Freitas, et al. "DharmaOCR: Specialized Small Language Models for Structured OCR that outperform Open-Source and Commercial Baselines." arXiv preprint arXiv:2604.14314 (2026).
- Dharma AI. "Text Degeneration: The Production Failure Mode That LLM Benchmarks Ignore." Medium (2026).
- Dharma AI. "Specialization Beats Scale: A Strategic Variable Most AI Procurement Decisions Overlook." Medium (2026).
- Holtzman, Ari, et al. "The Curious Case of Neural Text Degeneration." arXiv preprint arXiv:1904.09751 (2020).
- Rafailov, Rafael, et al. "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." arXiv preprint arXiv:2305.18290 (2023).