MatMMExtract:面向材料科学的大规模多模态数据集MatSciFig
阅读原文· arxiv.orgMatMMExtract 是一个端到端开源管道,将复合图表分解为独立子面板,并利用大语言模型基于材料科学分类法生成结构化标注。应用于 14,810 篇开放获取文章,从 180,571 张图中生成 MatSciFig 数据集,包含 391,606 个面板级图像-文本对,每对配有子标题、两级可视化类别(19 个大类、100+ 子类)和科学摘要。引入 MaterialScope 检测数据集(2,811 张人工标注图),微调 YOLO12-m 检测器达到 mAP_50 0.9227。六种基准语言模型中,Gemini 3.1 Flash Lite 在标注生成上取得最佳成本-质量平衡,82% 输出良好,模型幻觉率 4.8%。基于 MatSciFig 的检索基线在 R@1 上比零样本 CLIP 提升 4.4 倍,所有资源已向社区开放。
The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.