# AutoIndex：面向检索的文档表示程序学习框架

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-07-21 08:00
- AIHOT 分数：58
- AIHOT 链接：https://aihot.virxact.com/items/cmrwoddqa00rlrobhymks00dv
- 原文链接：https://arxiv.org/abs/2607.18603

## AI 摘要

AutoIndex 将文档表示视为可执行程序进行优化，通过智能体诊断检索失败模式并合成候选程序，在固定 BM25 检索器下迭代改进索引质量。在 CRUMB 多任务基准上，该方法在所有 8 项任务中均超越静态全文 BM25 基线，平均 Recall@100 提升 +8.4%、nDCG@10 提升 +8.3%。

## 正文

Sam O’Nuallain

Nithya Rajkumar

Ramya Narayanasamy

Hanna Jiang

Shreyas Chaudhari

Andrew Drozdov

University of Massachusetts Amherst

Databricks Mosaic Research

samonuall@gmail.com

nithyaraj1506@gmail.com

rnarayanasam@umass.edu

hannajiangg@gmail.com

schaudhari@umass.edu

andrew.drozdov@databricks.com

Abstract

We present AutoIndex, a framework for learning representation programs: executable transformations that map raw documents into the representations exposed to a retrieval system. Rather than tuning retrievers, rerankers, or a small set of preprocessing hyperparameters, AutoIndex searches over programs that slice, enrich, normalize, reweight, or reorganize documents before indexing. At each iteration, AutoIndex performs validation-guided program search, in which agents diagnose failures of the current program and synthesize candidate updates, retaining only updates that improve retrieval quality under the resulting index. We evaluate AutoIndex on CRUMB, a benchmark of heterogeneous retrieval tasks, with BM25 held fixed across all experiments. The learned programs improve recall over a static full-document BM25 baseline on all 8 tasks, with average gains of +8.4% in Recall@100 and +8.3% in nDCG@10, and largest gains of +30.5% in Recall@100 and +43.6% in nDCG@10. These results suggest that document representation should not be treated as a fixed preprocessing choice made before retrieval begins, but as an explicit optimization target. Code to reproduce our results is available at https://github.com/auto-index/autoindex.

1 Introduction

Data representation is a central but often under-optimized part of information retrieval. Before documents can be searched, ranked, or used in retrieval-augmented generation pipelines, they must first be represented as indexable units: chunks of text, attached context, normalized fields, metadata, or other structures exposed to a retriever. These choices determine which lexical cues are preserved, which context is attached to each unit, and which document structure is visible during retrieval. Yet this representation step is often treated as static infrastructure rather than as an optimization target.

We argue that document representation should be optimized directly. Existing systems typically rely on fixed heuristics such as chunk size, overlap, normalization rules, and metadata templates. Retrieval-model research improves the matching function through sparse, dense, hybrid, and late-interaction architectures (Robertson and Walker, 1994; Kuzi et al., 2020; Karpukhin et al., 2020; Khattab and Zaharia, 2020), while Retrieval Augmented Generation (RAG) pipeline optimization methods search over combinations of retrievers, rerankers, prompts, and generation components (Zeng et al., 2026; Kartal et al., 2025). These approaches improve retrieval pipelines, but they generally treat corpus representation as manually specified preprocessing or a choice among predefined configurations, rather than directly learning executable transformations that improve retrieval under a fixed retriever.

AutoIndex addresses this gap by treating document representation as code optimization. Given a corpus, seed queries, and a fixed retriever, AutoIndex searches over executable document representation programs using retrieval performance as the objective. At each iteration, an analysis agent identifies failure modes under the current index, a code-generation agent proposes revised programs, and each candidate is executed, indexed, and selected using offline metrics such as Recall@100 and nDCG@10 on a validation set. By holding the retriever, ranking function, and indexing backend fixed, AutoIndex isolates the representation program as the optimization target.

We evaluate AutoIndex on the Complex Retrieval Unified Multi-task Benchmark (CRUMB; Killingback and Zamani 2025), using BM25 as a fixed retriever. AutoIndex improves recall over a static full-document BM25 baseline on 8 of 8 tasks, with average gains of +8.4% in Recall@100 and +8.3% in nDCG@10, and largest gains of +30.5% in Recall@100 and +43.6% in nDCG@10. These gains are achieved without retriever fine-tuning, embedding updates, or online feedback, and the learned program is applied at indexing time with no further LLM calls.

1.

We formulate retrieval indexing as code optimization over executable document representation programs, shifting attention from fixed preprocessing heuristics to retrieval-aware representation design.

2.

We introduce AutoIndex, an iterative framework that combines retrieval failure analysis, LLM-based program synthesis, sandboxed execution, and offline metric-based selection.

3.

We provide an empirical study on CRUMB showing that learned representation programs improve BM25 retrieval across heterogeneous tasks while keeping the underlying retriever fixed.

2 Related Work

Adaptive chunking and document representation.

Retrieval systems often tune document representation through preprocessing settings like chunk size, overlap, title inclusion, and metadata templates, which can yield large gains in retrieval and downstream LLM performance (Chen et al., 2024; Smith and Troynikov, 2024; Wang et al., 2024). However, exploring this space broadly is expensive, since each configuration requires rebuilding the index. AutoIndex targets this bottleneck but searches a broader space of executable programs that chunk, enrich, or reorganize documents, using failure analysis and metric-guided search rather than manual or grid-based tuning. Other work represents documents more dynamically: late chunking encodes full documents before pooling chunk embeddings (Gunther et al., 2024); kNN-LM-style systems retrieve at the token level (Khandelwal et al., 2020) or chunk level (Lan et al., 2023) to bypass manual chunking; and other methods use LLMs directly to intelligently chunk documents for semantic retrieval (Duarte et al., 2024; Zhao et al., 2025; Luo et al., 2024). AutoIndex shares this goal of dynamic chunking, but avoids invoking an LLM per document by instead using program search guided by failure analysis to explore the space more systematically.

Index enrichment.

Index-enrichment methods improve retrieval by augmenting documents before indexing. Doc2Query-style methods expand documents with predicted queries, improving lexical matching but introducing risks such as hallucinated or low-quality expansions that can inflate the index and hurt retrieval (Nogueira et al., 2019; Gospodinov et al., 2023). Recent LLM-based methods add summaries, canonicalizations, extracted facts, rationales, or learned rewrites before retrieval (Chen et al., 2025). Document Optimization fine-tunes a model to rewrite documents using black-box retrieval feedback (Uzan et al., 2026), while RL-Index augments documents with LLM-generated rationales optimized for retrieval (Lei et al., 2026). These approaches show that offline document transformations can improve retrieval, but they typically learn or specify a transformation model of a particular type. AutoIndex instead searches over executable representation programs while holding the retriever and ranking function fixed.

Agentic program optimization.

AutoIndex relates to LLM-based program-optimization systems, where models propose executable artifacts refined via feedback (Chen et al., 2021; Novikov et al., 2025; Agrawal et al., 2025; Jiang et al., 2025). Iterative reasoning frameworks such as ReAct, RLMs, and Parallel Thinking show that structured reasoning, tool use, and candidate trajectories can improve robustness (Yao et al., 2023; Zhang et al., 2026; Chang et al., 2026). RAG optimization systems search over pipeline components such as retrievers, rerankers, prompts, and generators (Zeng et al., 2026; Kartal et al., 2025). AutoIndex applies verifiable program search to a different object: the pre-indexing representation program that maps raw documents into the text units exposed to a fixed retriever.

Figure 1: AutoIndex optimization loop. Given a source corpus, validation queries, and a fixed retriever, AutoIndex builds an index from the current representation program, uses an Analysis Agent to diagnose retrieval failures, and uses a Code Agent to synthesize candidate updates. Candidates are evaluated by building new indexes and measuring validation retrieval quality. Updates that improve the objective become the next incumbent.

3 AutoIndex

AutoIndex learns executable document representation programs: programs that transform raw documents into indexable units searched by a fixed retriever. At each iteration, AutoIndex analyzes retrieval failures under the current index, synthesizes candidates, rebuilds the index induced by each, and selects programs according to a validation objective , as shown in Figure 1. In our experiments, the retriever is BM25. We next formalize the objective and describe the agentic program synthesis loop used to optimize it.

3.1 Formalization

We formalize AutoIndex as black-box code synthesis over executable document representation programs. Let denote the executable program code and denote the document-to-units mapping induced by running that code. Given a source document , the program maps it to indexable units, , where each unit retains the identifier of its source document. Applying to a corpus yields an indexed representation . A fixed retriever scores indexable units for a query . We aggregate unit-level scores to source-document scores using MaxP, , and rank documents by . Since the retriever, ranking rule, and indexing backend are held fixed, performance differences are attributable to the representation program .

Given validation queries , AutoIndex evaluates each program with an offline objective . We treat as a black-box objective: a candidate program can only be evaluated by executing on the corpus, rebuilding the index, and measuring retrieval quality. At iteration , AutoIndex produces a diagnostic summary of the current program’s behavior and synthesizes an updated program with an abstract update policy :

The search history is the running log of the optimization search up to iteration : it records every previously evaluated representation program together with its validation outcome, e.g., . This equation abstracts over the two-agent candidate generation and selection procedure described below: the Analysis Agent computes , the Code Agent proposes multiple candidate programs conditioned on and , each candidate is evaluated under , and validation selection determines the next incumbent. The best-performing iterate is evaluated once on held-out queries. AutoIndex is retriever-agnostic in principle; in this work, we instantiate with BM25.111We use bm25s; parameters and dependency versions are given in Appendix A.2.

3.2 Agentic Program Synthesis

AutoIndex uses two specialized agents to synthesize document representation programs, separating diagnostic reasoning from code generation (Appendix A.2). The Analysis Agent inspects retrieval behavior under the current representation program and produces a structured summary grounded in concrete examples. The Code Agent conditions on this summary and the search history to propose new executable representation programs. Together with validation selection, these agents instantiate the abstract update policy .

Analysis Agent.

The Analysis Agent is a tool-using policy that diagnoses the behavior of the current representation program. Given the current program , a stratified candidate query set , and access to a restricted tool set , the agent produces a natural-language summary :

The tool set is read-only: it allows the Analysis Agent to retrieve from the current index, inspect source documents, and view validation queries. The agent invokes these tools to ground its summary in concrete retrieval behavior rather than unguided introspection.222In our experiments, the Analysis Agent runs for up to 5 steps.

The candidate set is stratified into three diagnostic categories: (i) Anchors, queries for which the initial program retrieves a gold document in the top-, but the current program does not; (ii) Recall Violations, queries for which the current program does not retrieve a gold document in the top-; and (iii) Small-Margin Positives, queries for which the current program retrieves a gold document in the top-, but not in the top-.333We set and sample five queries per category. Since Anchors are a subset of Recall Violations under the current program, categories are assigned in priority order: Anchors are selected first and excluded from Recall Violations.

Code Agent.

The Code Agent is a policy that proposes new executable representation programs conditioned on the analysis summary and the search history. Given the current program , the analysis summary , and the search history , the agent jointly generates candidate programs:

Each candidate is executed on the corpus to produce a new index and evaluated on validation queries. We define its improvement over the current incumbent as , and retain candidates with improvement at least in the round-level set

In our experiments, is validation .444Additional implementation details, including candidate generation and synthesis, are provided in Appendix A.2.

Program Selection.

After each proposal round, AutoIndex selects the next incumbent program according to validation performance. The incumbent program at the start of each iteration is always the best program found so far according to validation . If , the incumbent remains unchanged. If , its sole member becomes the next incumbent. If , we attempt synthesis: the LLM is asked to write a program that combines the transformations implemented by the programs in . The synthesized candidate is adopted only if it outperforms the best individual candidate in ; otherwise, AutoIndex falls back to the best single candidate. The search runs for 5 iterations; generated code must pass syntax validation, and each candidate is evaluated with an execution timeout of 15 minutes. Final prompts are included in Appendix A.4; BM25 parameters, dependency versions, and the prompt refinement process are documented in Appendix A.2.

4 Experimental Setup

Dataset and splits.

We evaluate AutoIndex on CRUMB, a benchmark of eight complex retrieval tasks designed to stress compositional queries, long documents, and heterogeneous evidence requirements (Killingback and Zamani, 2025). These properties make CRUMB a natural testbed for learned representation programs, since effective indexing must support both direct lookup and reasoning-intensive retrieval within a single corpus. For each CRUMB split, we partition the available queries into validation and held-out evaluation sets at a 1:2 ratio. Exact split sizes and query IDs are provided in Appendix A.5. We use the following split abbreviations in tables and figures: CT = ClinicalTrial, CR = CodeRetrieval, LQA = LegalQA, PR = PaperRetrieval, SOE = SetOpEntity, SE = StackExchange, TR = TheoremRetrieval, and TOT = TipOfTongue.

Metrics and baselines.

We use the bm25s library Lucene implementation of BM25 (Lù (2024)). We report Recall@100 and nDCG@10, with passage-level scores aggregated to documents using MaxP over 10,000 retrieved candidate chunks per query (Appendix A.2). Our main baseline is BM25 over the full-document markdown corpus provided for each split, where documents have been converted to a uniform markdown format with relevant headings preserved. We also compare against CRUMB’s passage corpus baseline, which chunks each markdown document into 512-BERT-token passages while preserving the relevant markdown titles.

Experimental design.

Across all experiments, the retriever and indexing backend are held fixed, so performance differences reflect changes in the learned representation program. Our primary setting uses the full AutoIndex loop with search history enabled: the Code Agent receives both the Analysis Agent’s diagnostic summary and the search history. We evaluate two code-generation backbones, Claude Sonnet 4.6 with seeds and qwen3-coder with seeds. Each run uses 5 optimization iterations and jointly proposes candidate programs per iteration; candidates are retained when their validation improvement satisfies , where is validation Recall@100. The best validation checkpoint is then evaluated once on the held-out evaluation queries.

5 Experimental Results

媒体内容 · 前往原文查看

Recall@100 CT LQA SOE SE TOT CR PR TR AVG

# Queries 84 4569 314 79 100 2510 53 51 —

BM25 (Full-Doc) 20.7 55.1 25.7 67.1 25.0 4.7 33.7 8.5 30.1

Passage Corpus 8.4 22.4 15.1 49.0 5.8 — — — —

AutoIndex 32.6

vs. Full-Doc

vs. Passage — — — —

Table 1: Recall@100 for AutoIndex (qwen3-coder, search history enabled, 5 iterations) on held-out CRUMB test splits, against both the BM25 full-document baseline and CRUMB’s passage-corpus baseline. AutoIndex values are mean std over 3 seeds. is relative to the baseline named in each row. Per-split is computed from unrounded scores. Bold entries in the vs. Full-Doc row indicate a relative gain of at least . CRUMB does not provide a passage corpus for CodeRetrieval, PaperRetrieval, or TheoremRetrieval, so those columns and the passage-relative AVG are marked "—". There is a counterpart using Claude Sonnet 4.6 in Table 4.

媒体内容 · 前往原文查看

nDCG@10 CT LQA SOE SE TOT CR PR TR AVG

# Queries 84 4569 314 79 100 2510 53 51 —

BM25 (Full-Doc) 52.2 16.4 12.2 21.9 12.0 4.4 67.3 0.5 23.4

Passage Corpus 42.3 4.5 11.9 11.8 2.5 — — — —

AutoIndex 25.3

vs. Full-Doc

vs. Passage — — — —

Table 2: nDCG@10 for AutoIndex (qwen3-coder, search history enabled, 5 iterations) on held-out CRUMB test splits, against both the BM25 full-document baseline and CRUMB’s passage-corpus baseline. Same parameters as Table 1. Bold entries in the vs. Full-Doc row indicate a relative gain of at least .

Main results.

AutoIndex improves Recall@100 across all CRUMB tasks. Tables 1 and 2 report held-out performance for qwen3-coder when search history is enabled, against both the BM25 full-document baseline and CRUMB’s passage-corpus baseline. The largest gains over the full-document baseline appear on TheoremRetrieval, SetOpEntity, and LegalQA, where BM25 vocabulary mismatch leaves substantial headroom for representation changes. nDCG@10 often improves alongside Recall@100 despite selection being driven solely by validation Recall@100, suggesting that AutoIndex improves retrieved evidence quality rather than merely expanding the index.

Comparison to uniform chunking.

CRUMB’s passage corpus applies a uniform chunking strategy across splits. As shown in the vs. Passage rows of Tables 1 and 2, AutoIndex outperforms this baseline on all reported splits, suggesting that learned representation programs can outperform domain-agnostic chunking by adapting to each split’s document structure and vocabulary.

Preliminary dense retrieval result.

To test whether learned representation programs transfer beyond BM25, we run a single dense-retrieval experiment on StackExchange using Qwen3-Embedding-0.6B. Reusing the learned AutoIndex representation improves held-out Recall@100 from 0.7391 to 0.8741, a relative gain of . Broader dense, hybrid, and reranking evaluations remain future work.

媒体内容 · 前往原文查看

Ablation condition

Split Full AutoIndex - 5 iter. 1 iter. w/o history w/o analysis

ClinicalTrial

CodeRetrieval

LegalQA

PaperRetrieval

SetOpEntity

StackExchange

TheoremRetrieval

TipOfTongue

Positive splits 8/8 3/8 5/8 6/8

Table 3: Ablation results on held-out CRUMB test splits using the qwen3-coder backbone. Values report Recall@100 relative to the BM25 full-document baseline, averaged over runs per condition. Full AutoIndex is the unablated pipeline with search history enabled and 5 iterations; 1 iter. is the same pipeline limited to a single iteration; w/o history withholds the search history from the Code Agent; w/o analysis removes the Analysis Agent, leaving the Code Agent with only aggregate metric feedback.

6 Framework Analysis and Discussion

6.1 Role of Iteration, Search History, and Analysis

Table 3 isolates three design choices in AutoIndex: iterative search, search history, and example-grounded analysis. The single-iteration condition improves only 3 of 8 splits, suggesting that AutoIndex’s gains generally do not come from a single prompt-level rewrite. Instead, useful representation programs often require repeated analysis, proposal, evaluation, and incumbent selection. Removing search history is neutral on aggregate (5 of 8 splits positive) but produces large per-split swings, helping CodeRetrieval (+5.9%) and StackExchange (+4.7%) while regressions stay small. This suggests the search history acts as a soft prior on candidate diversity, stabilizing within-run trajectories rather than preventing destructive edits on aggregate. Finally, removing the Analysis Agent shrinks effect magnitudes: 6 of 8 splits stay positive but with much smaller gains than the full pipeline, indicating that grounded failure analysis is a substantial source of signal.

Overall, the ablations suggest that AutoIndex works best when candidate programs are both grounded and constrained: grounded in concrete retrieval failures surfaced by the Analysis Agent, and constrained by the outcomes of previous proposals recorded in the search history.

6.2 Search Dynamics across Iterations

Figure 2 shows that improvements emerge through different search trajectories across splits. StackExchange improves sharply after an early adopted program and then continues to refine; SetOpEntity accumulates gains across multiple rounds; and ClinicalTrial shows small candidate improvements that are repeatedly rejected by the selection rule. These trajectories indicate that AutoIndex is not simply a one-shot prompt transformation, but an iterative search process in which candidate programs are proposed, evaluated, and selectively incorporated.

The single-iteration ablation in Table 3 reinforces this view: limiting AutoIndex to one iteration produces gains on only a small number of splits, whereas the full loop benefits from repeated analysis, proposal, and incumbent selection. This suggests that several improvements require multiple rounds of search before the system identifies a representation program that generalizes beyond the validation queries.

Figure 2: Iteration dynamics for three representative CRUMB splits under qwen3-coder. Curves show nDCG@10 relative to the BM25 baseline. Filled markers indicate adopted candidates and open markers indicate rejected candidates. AutoIndex exhibits different search patterns across splits, including early gains, gradual accumulation, and near-threshold rejected improvements.

Figure 3: Worked AutoIndex example on a LaTeX-heavy StackExchange article. The Analysis Agent flags repeated inline LaTeX; the Code Agent emits a threshold-gated strip_latex step (chunk: 1,847 1,102 tokens); top-3 retrieval flips from off-topic math references to relevant pricing articles, improving both Recall@100 and nDCG@10.

6.3 Learned Representation Behaviors

The learned programs reveal recurring behaviors across splits rather than one-off preprocessing tricks. On TipOfTongue, AutoIndex identifies a mismatch between concrete scene descriptions in user queries and abstract Wikipedia plot summaries, leading the Code Agent to reweight Plot and Cast sections while preserving full-document context. This improves retrieval by increasing the term frequency of query-relevant narrative content without aggressively discarding other evidence.

Figure 3 shows a second behavior that appears across LaTeX-heavy datasets. The Analysis Agent observes that repeated LaTeX markup can inflate chunk length and dilute the natural-language terms useful for BM25 matching. The Code Agent responds with targeted transformations that strip or normalize high-frequency LaTeX syntax, reducing non-informative token mass and allowing each indexed chunk to contain a higher proportion of semantic content. This behavior is not a generic cleanup rule applied blindly; it emerges when the agent finds evidence that markup is acting as retrieval noise in the current corpus.

Together, these examples mirror the broader ablation results: useful edits arise from corpus-specific failure analysis, while search history helps avoid destructive transformations such as overly aggressive filtering which harmed earlier runs. Additional case studies and generated programs are provided in Appendix A.3.2 and A.6.

6.4 Limitations and Future Work

AutoIndex currently optimizes primarily for Recall@100 with a fixed BM25 retriever, limited iteration budget, and small number of seeds. This leaves open how reliably gains converge, how to balance recall against nDCG, latency, index size, and preprocessing cost, and how learned programs interact with dense, hybrid, learned sparse, late-interaction, or reranking-based systems. Future work should also study generalization beyond per-corpus optimization: program transfer, where a learned representation program is applied directly to unseen datasets; adaptive programs, where the program inspects a new corpus or performs lightweight calibration before indexing; and optimizer transfer, where the AutoIndex procedure is initially tuned then frozen and evaluated across many datasets and domains.

7 Conclusion

We introduced AutoIndex, a framework for learning executable document representation programs for retrieval. Rather than tuning the retriever, AutoIndex optimizes the corpus representation itself: the chunks, context, normalization, and transformations exposed to a fixed retrieval system. On CRUMB, AutoIndex improves BM25 retrieval across heterogeneous tasks without retriever training, embedding updates, or online feedback. These results suggest that indexing should be treated as an optimization target in its own right, and that program synthesis is a promising mechanism for adapting corpus representations to retrieval objectives.

References

L. A. Agrawal, S. Tan, D. Soylu, N. Ziems, R. Khare, K. Opsahl-Ong, A. Singhvi, H. Shandilya, M. J. Ryan, M. Jiang, C. Potts, K. Sen, A. G. Dimakis, I. Stoica, D. Klein, M. Zaharia, and O. Khattab (2025) GEPA: reflective prompt evolution can outperform reinforcement learning. External Links: 2507.19457, Link Cited by: §2.

J. D. Chang, A. Drozdov, S. Toshniwal, O. Oertell, A. Trott, J. Portes, A. Gupta, P. Koppol, A. Baheti, S. Kulinski, I. Zhou, I. Dea, K. Opsahl-Ong, S. Favreau-Lessard, S. Owen, J. J. G. Ortiz, A. Singhvi, X. Andrade, C. Wang, K. K. Sreenivasan, S. Havens, J. Liu, P. J. Deniro, W. Sun, M. Bendersky, and J. Frankle (2026) KARL: knowledge agents via reinforcement learning. ArXiv abs/2603.05218. External Links: Link Cited by: §2.

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba (2021) Evaluating large language models trained on code. External Links: 2107.03374, Link Cited by: §2.

P. B. Chen, T. Wolfson, M. Cafarella, and D. Roth (2025) EnrichIndex: using llms to enrich retrieval indices offline. External Links: 2504.03598, Link Cited by: §2.

T. Chen, H. Wang, S. Chen, W. Yu, K. Ma, X. Zhao, H. Zhang, and D. Yu (2024) Dense x retrieval: what retrieval granularity should we use?. External Links: 2312.06648, Link Cited by: §2.

A. V. Duarte, J. Marques, M. Graça, M. F. Freire, L. Li, and A. L. Oliveira (2024) LumberChunker: long-form narrative document segmentation. ArXiv abs/2406.17526. External Links: Link Cited by: §2.

M. Gospodinov, S. MacAvaney, and C. Macdonald (2023) Doc2Query–: when less is more. In European Conference on Information Retrieval, External Links: Link Cited by: §2.

M. Gunther, I. Mohr, B. Wang, and H. Xiao (2024) Late chunking: contextual chunk embeddings using long-context embedding models. ArXiv abs/2409.04701. External Links: Link Cited by: §2.

Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y. Wu (2025) AIDE: ai-driven exploration in the space of code. External Links: 2502.13138, Link Cited by: §2.

V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020) Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y. He, and Y. Liu (Eds.), Online, pp. 6769–6781. External Links: Link, Document Cited by: §1.

M. Y. Kartal, S. K. Kose, K. Sevinç, and B. Aktas (2025) RAGSmith: a framework for finding the optimal composition of retrieval-augmented generation methods across datasets. External Links: 2511.01386, Link Cited by: §1, §2.

U. Khandelwal, O. Levy, D. Jurafsky, L. Zettlemoyer, and M. Lewis (2020) Generalization through memorization: nearest neighbor language models. External Links: 1911.00172, Link Cited by: §2.

O. Khattab and M. Zaharia (2020) ColBERT: efficient and effective passage search via contextualized late interaction over bert. External Links: 2004.12832, Link Cited by: §1.

J. Killingback and H. Zamani (2025) Benchmarking information retrieval models on complex retrieval tasks. External Links: 2509.07253, Link Cited by: §1, §4.

S. Kuzi, M. Zhang, C. Li, M. Bendersky, and M. Najork (2020) Leveraging semantic and lexical matching to improve the recall of document retrieval systems: a hybrid approach. ArXiv abs/2010.01195. External Links: Link Cited by: §1.

T. Lan, D. Cai, Y. Wang, H. Huang, and X. Mao (2023) Copy is all you need. External Links: 2307.06962, Link Cited by: §2.

Y. Lei, N. Lipka, Z. Qi, U. Sahu, K. Goswami, F. Dernoncourt, R. A. Rossi, and Y. Wang (2026) RL-index: reinforcement learning for retrieval index reasoning. External Links: Link Cited by: §2.

X. H. Lù (2024) BM25S: orders of magnitude faster lexical search via eager sparse scoring. External Links: 2407.03618, Link Cited by: §4.

K. Luo, Z. Liu, S. Xiao, and K. Liu (2024) BGE landmark embedding: a chunking-free embedding method for retrieval augmented long-context large language models. External Links: 2402.11573, Link Cited by: §2.

R. Nogueira, W. Yang, J. J. Lin, and K. Cho (2019) Document expansion by query prediction. ArXiv abs/1904.08375. External Links: Link Cited by: §2.

A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. R. Ruiz, A. Mehrabian, M. P. Kumar, A. See, S. Chaudhuri, G. Holland, A. Davies, S. Nowozin, P. Kohli, and M. Balog (2025) AlphaEvolve: a coding agent for scientific and algorithmic discovery. External Links: 2506.13131, Link Cited by: §2.

S. E. Robertson and S. Walker (1994) Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, External Links: Link Cited by: §1.

S. E. Robertson, H. Zaragoza, and M. J. Taylor (2004) Simple bm25 extension to multiple weighted fields. In International Conference on Information and Knowledge Management, External Links: Link Cited by: §A.3.1.

B. Smith and A. Troynikov (2024) Evaluating chunking strategies for retrieval. Technical report Chroma. External Links: Link Cited by: §2.

O. Uzan, R. Polonsky, D. Kiela, and C. Potts (2026) Document optimization for black-box retrieval via reinforcement learning. ArXiv abs/2604.05087. External Links: Link Cited by: §2.

X. Wang, Z. Wang, X. Gao, F. Zhang, Y. Wu, Z. Xu, T. Shi, Z. Wang, S. Li, Q. Qian, R. Yin, C. Lv, X. Zheng, and X. Huang (2024) Searching for best practices in retrieval-augmented generation. External Links: 2407.01219, Link Cited by: §2.

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. External Links: 2210.03629, Link Cited by: §2.

X. Zeng, Y. Liu, Y. Luo, and J. Zhen (2026) AutoRAGTuner: a declarative framework for automatic optimization of rag pipelines. External Links: 2605.02967, Link Cited by: §1, §2.

A. L. Zhang, T. Kraska, and O. Khattab (2026) Recursive language models. External Links: 2512.24601, Link Cited by: §2.

J. Zhao, Z. Ji, Z. Fan, H. Wang, S. Niu, B. Tang, F. Xiong, and Z. Li (2025) MoC: mixtures of text chunking learners for retrieval-augmented generation system. External Links: 2503.09600, Link Cited by: §2.

Appendix A Appendix

A.1 Claude Sonnet 4.6 Results

Table 4 reports the Claude Sonnet 4.6 counterpart to the qwen3-coder results in Tables 1 and 2. The setting is otherwise identical: search history is enabled, the loop runs for five iterations, and the best validation checkpoint is evaluated once on the held-out CRUMB test splits. Values are averaged over two seeds, with computed relative to the BM25 full-document baseline. Claude Sonnet 4.6 improves Recall@100 on 7 of 8 splits and yields similar qualitative patterns to qwen3-coder, with the largest gains on TheoremRetrieval, SetOpEntity, and LegalQA.

媒体内容 · 前往原文查看

Metric CT CR LQA PR SOE SE TR TOT AVG

# Queries 84 2510 4569 53 314 79 51 100 —

BM25 R@100 20.7 4.7 55.1 33.7 25.7 67.1 8.5 25.0 30.1

AutoIndex R@100 32.1

R@100

BM25 nDCG@10 52.2 4.4 16.4 67.3 12.2 21.9 0.5 12.0 23.4

AutoIndex nDCG@10 24.9

nDCG@10

Table 4: AutoIndex using Claude Sonnet 4.6 with search history enabled and five iterations on the held-out CRUMB test splits. Scores are multiplied by 100 and reported as mean standard deviation over two seeds; denotes a single seed. AVG is the unweighted macro-average across datasets, and the aggregate is computed from the corresponding macro-averaged scores. Bold values indicate relative gains exceeding . Qwen3-Coder results are reported in the main text (Tables 1 and 2).

Figure 4: Recall@100 relative to the BM25 full-document baseline across CRUMB splits for both Code Agent backbones. Bars show relative improvement; error bars show standard deviation. AVG is the unweighted macro-average and is shown without error bars.

A.2 Design Decisions and Implementation Notes

Optimization target.

We adopt Recall@100 as the primary optimization signal rather than nDCG@10 because recall is the dominant quality signal for agentic search and downstream RAG, where a reader or reasoning step can recover from imperfect ranking but cannot recover from missing evidence. Recall@10 was also considered but left too little headroom on the hardest splits for any candidate to clear the acceptance threshold.

Validation/evaluation split ratio.

Earlier experiments used much smaller validation pools, which weakened the optimization signal: on splits like PaperRetrieval, nearly all validation queries already had their gold document in the top 100 under a reasonable baseline, leaving no headroom for the acceptance threshold to fire and stalling the agent. The 1:2 validation/evaluation ratio gives the analysis agent enough validation queries to surface actionable failure patterns while still preserving a representative held-out evaluation set. Evaluating exclusively on the held-out set also reflects realistic deployment, where the preprocessing code must generalize to unseen queries against the same corpus.

Choice of BM25.

We use BM25 as the primary retriever because it provides a strong, widely used, and transparent testbed for studying document representation. Its sensitivity to segmentation, normalization, and term reweighting makes it well suited for isolating the effect of learned representation programs, while efficient indexing allows AutoIndex to evaluate many candidates within a practical search budget.555For larger corpora, candidate programs could first be evaluated approximately on sampled subsets, with promising candidates subsequently validated on the full corpus. We leave such budget-aware evaluation strategies to future work. Although our main experiments use BM25, the programs operate before retrieval and are not inherently tied to its scoring function. Transformations that improve contextual coverage, structure, or noise removal may transfer to other retrievers, while BM25-specific term reweighting may transfer less directly.

We use bm25s v0.2.14 with and MaxP aggregation from chunks to source documents, following the CRUMB evaluation protocol. Text is lowercased and tokenized using bm25s.tokenize defaults: regex-based word tokenization, English stopword removal, and no stemming.

Why two agents.

In early experiments, a single coding agent given only aggregate Recall@100/nDCG@10 deltas had to guess what was going wrong in the corpus, producing low-signal hypotheses biased toward generic preprocessing tricks. Splitting the roles also keeps the code agent’s context clean of the long document and query excerpts that the analysis agent must consume, which is particularly important for weaker backbone models whose performance degrades with context length.

Analysis agent tools.

The analysis agent is given the tool set . The bm25_retrieve(query, top_k) tool is mostly used by the analysis agent to search the current index using a candidate query, giving it the top chunks retrieved using the current preprocessing program. The read_file(file_path, max_chars) is then used to allow the agent to selectively read the original documents associated with retrieved chunks from the bm25_retrieve tool. The analysis agent can also use grep_search(pattern, file_path, max_results) to selectively search for specific terms in documents and queries for targeted analysis, as without this tool the analysis agent could easily overload its context window by reading many documents in search of a specific term or pattern. All of these tools together allow the analysis agent to take a candidate query, retrieve the chunks that match with that query from the current index, and read the original documents from which these chunks came from in order to reason about what preprocessing factors contributed to failure and success. All tools are read-only, and are restricted to the current split’s validation queries and document corpus.

Curated candidate queries.

Restricting the analysis agent to a curated, balanced slice of validation queries dramatically reduces the time it spends searching for relevant patterns and prevents fixation on idiosyncratic queries that do not generalize, which we observed when the agent was given unrestricted access to the full validation pool.

Prompt iteration.

We developed the analysis and code agent prompts by inspecting their outputs across early runs and refining the prompts to correct recurring failure modes: lack of hypothesis diversity, over-anchoring on suggested ideas embedded in the system prompt (which we removed in favor of more general guidance), over-fitting to domain context, and inefficient tool use that caused context overflow. Final prompts are included in Appendix A.4.

A.3 Case Studies

Computational cost.

Table 5 summarizes per-run token usage and LLM latency for the two backbones. Sonnet 4.6 runs consume more tokens per run and per call than qwen3-coder, and its analysis and code calls are both slower, yielding a longer LLM-only wall-clock time per run.

媒体内容 · 前往原文查看

Metric Sonnet-4.6 Qwen3-coder

Total tokens/run 412,739 344,081

analysis tokens 267,947 193,013

code tokens 125,031 107,386

completion tokens 40,378 27,221

Tokens/call 8,864 5,832

LLM s/call

analysis 4.16 1.77

code 29.96 17.70

combined 15.48 9.67

LLM-only wall/run (s) 754 488

Table 5: Token usage and LLM latency statistics per AutoIndex run for each backbone, averaged over runs. Analysis, code, and completion tokens are the mean per-run token totals attributed to the Analysis Agent, Code Agent, and generated completions respectively. Tokens/call and LLM s/call report per-request averages, with the combined figure pooling analysis and code calls; LLM-only wall/run excludes time spent on preprocessing execution, evaluation, and indexing.

A.3.1 More information on the TipOfTheTongue Case Study

The target documents for this task are Wikipedia pages corresponding to the movies and TV shows described in queries.

The code agent’s strategy was based on the analysis agent’s findings.

Reasoning explicitly about BM25’s term-frequency saturation and length normalization, the code agent considered extracting the plot section as a separate chunk, but instead chose to preserve a single full-document representation so that plot-related terms remained accompanied by identifying context from the rest of the article. It therefore designed an in-chunk reweighting scheme: keep one chunk per document, but repeat the plot section three times and the cast section twice, increasing the term-frequency contribution of query-relevant narrative content while retaining the full article context. This can be viewed as a form of text-level field reweighting within an unmodified single-field BM25 index. It resembles the motivation behind field-weighted retrieval methods such as BM25F (Robertson et al., 2004), but differs technically because the weighting is implemented through the indexed representation rather than through field-specific scoring and normalization. The implementation extracted the relevant sections from the raw markup before cleaning, applied the multiplicative weighting, and emitted one chunk per document.

A.3.2 StackExchange Case Study

This case study is from the StackExchange split, where queries are community questions taken from a variety of StackExchange forums. The target documents are relevant web pages taken from answers in the forums.

In this iteration the Analysis Agent began by triaging a small batch of validation queries whose gold documents had fallen out of the top-100, noticing a common pattern: each query asked about a surface-level real-world phenomenon (a company’s pricing strategy, a kitchen observation, a financial puzzle) while the gold documents were abstract conceptual reference articles. To probe one such gold document the agent progressively read larger character windows of its raw text, and observed that the article was filled with repeated inline mathematical notation fragments — the same backtick-wrapped LaTeX commands appearing dozens of times alongside the genuine prose. Since math questions are common on StackExchange, many documents contain latex syntax. The Analysis Agent then framed this observation in terms of BM25 internals, reasoning that the repeated markup was introducing retrieval noise and reducing the relative prominence of the content terms most useful for matching the query.

Using the summary generated by the Analysis Agent, the Code Agent came up with a hypothesis that stripping latex commands would increase Recall@100. Crucially, the Code Agent consulted the run’s search history and recalled that an earlier iteration had attempted a broad boilerplate stripper and caused regressions by removing too much; this prior failure directly shaped the new hypothesis to be a narrow, pattern-targeted cleaner that activates only when a document crosses a heavy-LaTeX threshold, leaving prose-only documents and documents where math is meaningful untouched. The resulting code yielded multiple recovered queries with zero regressions. Rather than applying generic preprocessing, AutoIndex diagnosed corpus-specific noise and designed a surgical, theory-grounded intervention (final implementation from this run available in Listing 1, Appendix A.6).

A.4 Prompts

A.4.1 Code Agent Prompt

BM25

bm25s

Recall@100

primary

nDCG@10

secondary

hypothesis

gains

R@100

while

losing

nDCG@10

Recall@100

nDCG@10

R@100

but

can

dilute

nDCG@10

BM25

retrieval

retriever

BM25

via

bm25s

VAL_QUERY_COUNT

EVAL_QUERY_COUNT

VAL_ONE_QUERY_PCT

__file__

doc_id

exactly

match

source

Document

doc_id

chunk_id

values

e

g

f

doc_id

CRITICAL

chunk

doc_id

must

be

one

original

doc_id

doc_id

doc_ids

doc_id

doc_id

doc_id

verbatim

into

chunk

doc_id

doc_id

doc_id

BM25

Considerations

BM25

Clarification on MaxP and additional chunks.

The prompt states that “additional chunks can in principle only help” because, under MaxP, a useful new chunk can improve a document’s score without replacing its existing representation. This is a practical heuristic rather than a strict guarantee: added chunks may introduce competing results or alter BM25 collection statistics. We retain the original wording to reproduce the prompt used in our experiments.

A.4.2 Analysis Agent Prompt

BM25

BM25

indexes

fields

indexed

BM25

Recall@100

primary

nDCG@10

secondary

change

improves

Recall@100

but

regresses

nDCG@10

Recall@100

win

when

nDCG@10

at

worst

flat

Recall@100

nDCG@10

BM25

BM25

VAL_QUERY_COUNT

validation

queries

EVAL_QUERY_COUNT

VAL_ONE_QUERY_PCT

bm25_retrieve

retrieve

top

results

query

use

read_file

filter_id

BM25

A.5 Query Splits

Full query IDs available here in files section: https://github.com/auto-index/autoindex/blob/main/query_splits.json.

Split Validation queries Evaluation queries

clinical_trial 41 84

code_retrieval 1255 2510

legal_qa 2284 4569

paper_retrieval 26 53

set_operation_entity_retrieval 156 314

stack_exchange 39 79

theorem_retrieval 25 51

tip_of_the_tongue 50 100

A.6 Case Study Generated Code Implementations

媒体内容 · 前往原文查看

Listing 1: Preprocessing code from case study 1, generated for StackExchange split of CRUMB.

⬇

1importsys,pathlib

2sys.path.insert(0,str(pathlib.Path(__file__).parents[2]/"evaluation"))

3

4fromtypingimportList

5fromschemaimportDocument,Chunk

6frombaseimportBasePreprocessor

7importre

8

9

10

11STOPWORDS={

12’i’,’me’,’my’,’myself’,’we’,’our’,’ours’,’ourselves’,’you’,’your’,

13’yours’,’yourself’,’yourselves’,’he’,’him’,’his’,’himself’,’she’,

14’her’,’hers’,’herself’,’it’,’its’,’itself’,’they’,’them’,’their’,

15’theirs’,’themselves’,’what’,’which’,’who’,’whom’,’this’,’that’,

16’these’,’those’,’am’,’is’,’are’,’was’,’were’,’be’,’been’,’being’,

17’have’,’has’,’had’,’having’,’do’,’does’,’did’,’doing’,’a’,’an’,

18’the’,’and’,’but’,’if’,’or’,’because’,’as’,’until’,’while’,’of’,

19’at’,’by’,’for’,’with’,’about’,’against’,’between’,’into’,’through’,

20’during’,’before’,’after’,’above’,’below’,’to’,’from’,’up’,’down’,

21’in’,’out’,’on’,’off’,’over’,’under’,’again’,’further’,’then’,

22’once’,’here’,’there’,’when’,’where’,’why’,’how’,’all’,’both’,

23’each’,’few’,’more’,’most’,’other’,’some’,’such’,’no’,’nor’,’not’,

24’only’,’own’,’same’,’so’,’than’,’too’,’very’,’s’,’t’,’can’,’will’,

25’just’,’don’,’should’,’now’,’d’,’ll’,’m’,’o’,’re’,’ve’,’y’,’ain’,

26’aren’,’couldn’,’didn’,’doesn’,’hadn’,’hasn’,’haven’,’isn’,’ma’,

27’mightn’,’mustn’,’needn’,’shan’,’shouldn’,’wasn’,’weren’,’won’,

28’wouldn’,’also’,’may’,’would’,’could’,’one’,’two’,’three’,’four’,

29’five’,’six’,’seven’,’eight’,’nine’,’ten’,’however’,’thus’,’hence’,

30’therefore’,’although’,’though’,’since’,’whether’,’while’,’whereas’,

31’upon’,’within’,’without’,’along’,’following’,’across’,’behind’,

32’beyond’,’plus’,’except’,’but’,’up’,’around’,’including’,’among’,

33}

34

35

36defstrip_latex(text:str)->str:

37"""

38RemoveLaTeX/mathnotationfromtexttoreducetokennoise.

39Targets:

40-Backtick-wrappedLaTeXfragments:‘\command{...}‘or‘\command‘

41-Displaymathenvironments:\displaystyle,\frac,\sum,\int,etc.

42-Greeklettercommands:\alpha,\beta,etc.

43-OthercommonLaTeXcommands

44-Inlinemathdelimiters:$...$and$$...$$

45-CurlybracesleftoverfromLaTeX

46"""

47

48text=re.sub(r’‘[^‘]*\\[^‘]*‘’,’’,text)

49

50

51text=re.sub(r’\$\$.*?\$\$’,’’,text,flags=re.DOTALL)

52

53

54text=re.sub(r’\$[^$\n]{0,200}\$’,’’,text)

55

56

57text=re.sub(r’\\begin\{[^}]*\}.*?\\end\{[^}]*\}’,’’,text,flags=re.DOTALL)

58

59

60text=re.sub(r’\\[a-zA-Z]+\{[^}]*\}’,’’,text)

61

62

63text=re.sub(r’\\[a-zA-Z]+’,’’,text)

64

65

66text=re.sub(r’[{}]’,’’,text)

67

68

69text=re.sub(r’{2,}’,’’,text)

70

71returntext.strip()

72

73

74defsentence_density_score(sentence:str)->float:

75"""

76Scoreasentencebythecountofuniquenon-stopwordtokens.

77Higher=moreinformation-dense.

78"""

79tokens=re.findall(r’[a-zA-Z]+’,sentence.lower())

80unique_content={tfortintokensiftnotinSTOPWORDSandlen(t)>2}

81returnlen(unique_content)

82

83

84defsplit_into_sentences(text:str)->List[str]:

85"""

86Splittextintosentencesusingsimpleregex-basedsplitting.

87"""

88sentences=re.split(r’(?<=[.!?])\s+(?=[A-Z])’,text)

89result=[]

90forsentinsentences:

91parts=re.split(r’\n\s*\n’,sent)

92result.extend(parts)

93cleaned=[]

94forsinresult:

95s=s.strip()

96ifsandlen(s.split())>=5:

97cleaned.append(s)

98returncleaned

99

100

101defextract_title(doc:’Document’)->str:

102"""

103Extractthemostinformativetitlefromthedocument.

104"""

105ifdoc.metadataanddoc.metadata.get("title"):

106returndoc.metadata["title"].strip()

107

108text=doc.text.strip()

109ifnottext:

110return""

111

112lines=text.splitlines()

113

114

115forlineinlines[:20]:

116line=line.strip()

117ifnotline:

118continue

119ifre.match(r’^#{1,3}\s+(.+)’,line):

120title=re.sub(r’^#{1,3}\s+’,’’,line).strip()

121iftitle:

122returntitle

123

124

125forlineinlines[:10]:

126line=line.strip()

127ifnotline:

128continue

129words=line.split()

130if1<=len(words)<=15:

131returnline

132

133

134forlineinlines:

135line=line.strip()

136ifline:

137words=line.split()

138return"".join(words[:15])

139

140return""

141

142

143defhas_heavy_latex(text:str)->bool:

144"""

145DetectifadocumenthasheavyLaTeX/mathnotationpollution.

146ReturnsTrueifthedocumentcontainssignificantLaTeXcontent.

147"""

148

149latex_patterns=[

150r’‘[^‘]*\\[^‘]*‘’,

151r’\$[^$]{1,100}\$’,

152r’\\[a-zA-Z]+’,

153r’\\begin\{’,

154]

155total_matches=0

156forpatterninlatex_patterns:

157matches=re.findall(pattern,text)

158total_matches+=len(matches)

159

160

161returntotal_matches>10

162

163

164classPreprocessor(BasePreprocessor):

165name="analysis_code_agent"

166description="Full-textchunk(withLaTeXstrippedformath-heavydocs)+openingchunk+title-repetitionchunk+top-3densitysentenceschunk."

167

168OPENING_WORD_LIMIT=200

169TITLE_REPEAT_COUNT=5

170=3

171

172defpreprocess(self,docs:List[Document])->List[Chunk]:

173chunks=[]

174fordocindocs:

175doc_id=doc.doc_id

176text=doc.text

177

178

179latex_heavy=has_heavy_latex(text)

180

181

182

183

184

185iflatex_heavy:

186chunk0_text=strip_latex(text)

187else:

188chunk0_text=text

189

190chunks.append(Chunk(

191chunk_id=f"{doc_id}_0",

192doc_id=doc_id,

193text=chunk0_text,

194metadata=doc.metadata,

195))

196

197

198

199words=chunk0_text.split()

200iflen(words)>self.OPENING_WORD_LIMIT:

201opening_text="".join(words[:self.OPENING_WORD_LIMIT])

202chunks.append(Chunk(

203chunk_id=f"{doc_id}_1",

204doc_id=doc_id,

205text=opening_text,

206metadata=doc.metadata,

207))

208

209

210title=extract_title(doc)

211iftitle:

212

213iflatex_heavy:

214title=strip_latex(title)

215title_chunk_text="".join([title]*self.TITLE_REPEAT_COUNT)

216chunks.append(Chunk(

217chunk_id=f"{doc_id}_2",

218doc_id=doc_id,

219text=title_chunk_text,

220metadata=doc.metadata,

221))

222

223

224

225sentences=split_into_sentences(chunk0_text)

226iflen(sentences)>self.:

227scored=[(sentence_density_score(s),i,s)fori,sinenumerate(sentences)]

228scored.sort(key=lambdax:(-x[0],x[1]))

229top_sentences=scored[:self.]

230top_sentences.sort(key=lambdax:x[1])

231density_text="".join(sfor_,_,sintop_sentences)

232chunks.append(Chunk(

233chunk_id=f"{doc_id}_3",

234doc_id=doc_id,

235text=density_text,

236metadata=doc.metadata,

237))

238

239returnchunks

媒体内容 · 前往原文查看

Listing 2: Preprocessing code from case study 2, generated for TipOfTongue split of CRUMB.

⬇

1importsys,pathlib

2sys.path.insert(0,str(pathlib.Path(__file__).parents[2]/"evaluation"))

3fromtypingimportList

4fromschemaimportDocument,Chunk

5frombaseimportBasePreprocessor

6

7importre

8

9

10defclean_wiki_markup(text:str)->str:

11"""

12CleanWikipediamarkuptoextractplaintext.

13"""

14

15text=re.sub(r’<ref[^>]*>.*?</ref>’,’’,text,flags=re.DOTALL|re.IGNORECASE)

16

17text=re.sub(r’<ref[^/]*/>’,’’,text,flags=re.IGNORECASE)

18

19

20text=re.sub(r’\[\[(Category|File|Image|Media):[^\]]*\]\]’,’’,text,flags=re.IGNORECASE)

21

22

23text=re.sub(r’\[\[(?:[^\|\]]*\|)([^\]]+)\]\]’,r’\1’,text)

24

25

26text=re.sub(r’\[\[([^\]]+)\]\]’,r’\1’,text)

27

28

29prev=None

30whileprev!=text:

31prev=text

32

33defreplace_template(m):

34inner=m.group(1)

35parts=inner.split(’|’)

36values=[]

37forpartinparts[1:]:

38part=part.strip()

39if’=’inpart:

40key,_,val=part.partition(’=’)

41val=val.strip()

42ifvalandnotre.match(r’^https?://’,val)andnotre.match(r’^\d+$’,val):

43values.append(val)

44else:

45ifpartandnotre.match(r’^https?://’,part):

46values.append(part)

47return’’.join(values)ifvalueselse’’

48

49text=re.sub(r’\{\{([^{}]+)\}\}’,replace_template,text)

50

51

52text=re.sub(r’\{\{|\}\}’,’’,text)

53

54

55text=re.sub(r’={2,}([^=]+)={2,}’,r’\1’,text)

56

57

58text=re.sub(r’<[^>]+>’,’’,text)

59

60

61text=re.sub(r’\{\|.*?\|\}’,’’,text,flags=re.DOTALL)

62text=re.sub(r’^\s*[|!].*$’,’’,text,flags=re.MULTILINE)

63

64

65text=re.sub(r’https?://\S+’,’’,text)

66

67

68text=re.sub(r’[|]{2,}’,’’,text)

69text=re.sub(r’\[|\]’,’’,text)

70

71

72text=re.sub(r’\s+’,’’,text).strip()

73

74returntext

75

76

77defextract_section(text:str,section_names:List[str])->str:

78"""

79Extractasectionfromthecleanedtextbylookingforsectionheaderpatterns.

80Returnsthesectioncontentoremptystringifnotfound.

81Worksontheraw(pre-cleaned)texttocapturesectionboundaries.

82"""

83

84names_pattern=’|’.join(re.escape(name)fornameinsection_names)

85

86pattern=re.compile(

87r’={2,}\s*(?:’+names_pattern+r’)\s*={2,}\s*(.*?)(?=={2,}[^=]|$)’,

88re.IGNORECASE|re.DOTALL

89)

90match=pattern.search(text)

91ifmatch:

92returnmatch.group(1).strip()

93return’’

94

95

96defextract_sections_weighted(raw_text:str)->str:

97"""

98Buildaweighteddocument:

99-Plotsection:3xrepetition

100-Cast/Characterssection:2xrepetition

101-Everythingelse:1x

102

103AllwithinasinglechunktopreserveBM25lengthnormalizationbenefits.

104"""

105

106plot_raw=extract_section(raw_text,[’Plot’,’Synopsis’,’Story’,’Plotsummary’])

107

108cast_raw=extract_section(raw_text,[’Cast’,’Characters’,’Castandcharacters’,’Voicecast’])

109

110

111plot_clean=clean_wiki_markup(plot_raw)ifplot_rawelse’’

112cast_clean=clean_wiki_markup(cast_raw)ifcast_rawelse’’

113

114

115full_clean=clean_wiki_markup(raw_text)

116

117

118parts=[]

119

120

121parts.append(full_clean)

122

123

124ifplot_clean:

125parts.append(plot_clean)

126parts.append(plot_clean)

127

128

129ifcast_clean:

130parts.append(cast_clean)

131

132return’’.join(parts)

133

134

135classPreprocessor(BasePreprocessor):

136name="section_weighted_repetition"

137description=(

138"Applysection-weightedtextduplicationwithinasinglechunk:"

139"Plotsectionrepeated3x,Castsectionrepeated2x,restkeptonce."

140"Usesin-chunkrepetitiontoboostTFforplot/casttermswhile"

141"preservingBM25length-normalizationbenefitsofasinglelongdocument."

142)

143

144defpreprocess(self,docs:List[Document])->List[Chunk]:

145chunks=[]

146fordocindocs:

147weighted_text=extract_sections_weighted(doc.text)

148

149chunks.append(

150Chunk(

151chunk_id=f"{doc.doc_id}_0",

152doc_id=doc.doc_id,

153text=weighted_text,

154)

155)

156

157returnchunks