DataOrchestra：
Learning to Orchestrate Per-Example Curation of Pretraining Data
Happy to introduce DataOrchestra by @Z_Huang_02 and GAIR NLP, and feel thrilled to participate!! Most pretraining pipelines ask: Which processing recipe should ...
AI 导读
DataOrchestra 是 GAIR NLP 推出的开源框架，将预训练数据清洗转为逐样本决策：1.7B 轻量编排器对每个数据块决定 Drop、Untouch 或 Clean，并只为需清洗块选择必要操作。基于执行反馈生成 300K 训练对，在 11 项基准上从 0.5B 到 7B 规模一致优于固定与单独处理策略，优势随模型增大而扩大。约 35% 数据块跳过清洗，节省约 37% 清洗算力。
DataOrchestra：
Learning to Orchestrate Per-Example Curation of Pretraining Data
Happy to introduce DataOrchestra by @Z_Huang_02 and GAIR NLP, and feel thrilled to participate!! Most pretraining pipelines ask: Which processing recipe should ...
来源：Dongxi 东锡 NLP· x.com