EmpiriGraph-Psy:心理学摘要实证关系图抽取数据集与LLM流程
阅读原文· arxiv.org现有科学关系抽取基准主要面向计算机科学,缺乏心理学等变量导向实证领域的任务。本文提出变量中心实证图抽取任务,将科学摘要映射为以归一化变量为节点、边表示实证与层级关系的类型化图。构建EmpiriGraph-Psy基准,包含210篇经领域标注者标注的心理学摘要。评估表明,分阶段图构建管道(分步进行变量抽取、归一化、层级构建、证据选择、关系抽取和边验证)显著优于直接抽取,最佳配置macro-F1达0.74。错误分析显示,调节关系和概念层级仍是最大难点。
Existing scientific relation extraction benchmarks mainly target domains such as computer science, where entities are tasks, methods, datasets, materials, or metrics. This leaves a gap in variable-oriented empirical fields such as psychology, where findings are expressed as relations among constructs, measurements, interventions, and outcomes. We introduce variable-centered empirical graph extraction, the task of mapping scientific abstracts to typed graphs whose nodes are normalized variables and whose edges represent empirical and hierarchical relations. To support this task, we construct EmpiriGraph-Psy, a benchmark of 210 psychology abstracts annotated by domain-trained annotators with normalized variables, concept hierarchies, empirical relation types, and validation states. We evaluate frontier and open-weight LLMs using both direct extraction and a staged graph-construction pipeline that separates variable extraction, normalization, hierarchy construction, evidence selection, relation extraction, and edge validation. The staged pipeline substantially outperforms direct extraction, with the best configuration achieving a macro-F1 of 0.74. Error analysis shows that moderation relations and concept hierarchies remain the most challenging cases, highlighting the difficulty of extracting higher-order empirical claims and implicit abstraction structure from scientific abstracts.