Ψ-Bench:评估说服性对话中的人格敏感影响力
阅读原文· arxiv.orgΨ-Bench 是一个新提出的基准测试,旨在评估大语言模型在说服性对话中主动影响用户的能力。它设计了三个真实的交互场景,并通过对话历史为模拟客户赋予具体的人格特征。该基准对 10 个前沿大语言模型进行了评估,发现即使是最先进的模型在说服力方面仍有较大提升空间。研究同时发现,在提供客户档案后,模型性能平均提升 18.24%,凸显了用户特定信息对于实现有效说服的重要性。该工作指出,人格敏感影响力是评估与发展更主动的个性化 AI 智能体的一个有挑战性且实用的方向。
Personalization is a crucial capability of modern language agents. However, current research primarily positions personalized agents as passive responders to user preferences, limiting their ability to interact with users and provide suggestions or guidance proactively. To systematically evaluate such proactive personalization in realistic interactions, we propose Ψ-Bench, a benchmark for assessing LLMs' ability to influence realistic users through conversation. We design three real-world interaction scenarios that involve persuasion in Ψ-Bench, and endow simulated clients with personal characteristics through explicit user profiles derived from dialogue histories. We evaluate 10 frontier LLMs on Ψ-Bench and find that while most models can produce coherent and reasonable arguments, even state-of-the-art models still leave considerable room for improvement in persuasion. We also find that providing access to client profiles yields an average performance gain of 18.24\%, highlighting the importance of user-specific information for effective persuasion. Overall, our work highlights persona-sensitive influencing as a challenging yet practical direction for evaluating and developing more proactive personalized LLM agents. Codes are available at: https://github.com/Hanpx20/Psi-Bench.