# MatMMExtract：面向材料科学的大规模多模态数据集MatSciFig

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-06-29 08:00
- AIHOT 分数：50
- AIHOT 链接：https://aihot.virxact.com/items/cmr2428xl04llsl8zr940lcmt
- 原文链接：https://arxiv.org/abs/2606.29667

## AI 摘要

MatMMExtract 是一个端到端开源管道，将复合图表分解为独立子面板，并利用大语言模型基于材料科学分类法生成结构化标注。应用于 14,810 篇开放获取文章，从 180,571 张图中生成 MatSciFig 数据集，包含 391,606 个面板级图像-文本对，每对配有子标题、两级可视化类别（19 个大类、100+ 子类）和科学摘要。引入 MaterialScope 检测数据集（2,811 张人工标注图），微调 YOLO12-m 检测器达到 mAP_50 0.9227。六种基准语言模型中，Gemini 3.1 Flash Lite 在标注生成上取得最佳成本-质量平衡，82% 输出良好，模型幻觉率 4.8%。基于 MatSciFig 的检索基线在 R@1 上比零样本 CLIP 提升 4.4 倍，所有资源已向社区开放。

## 正文

The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.