We increasingly use LLMs as prixies, for real people in cross-country surveys. But do they simulate every country equally well, or just the rich, well-connected ones?
New in Computational Linguistics, from @TsinghuaNLP with Fudan University, Nankai University, and Shanghai Jiao Tong University. It introduces Representational Equality: a concept and metric for whether a model simulates diverse populations evenly, tested across 59 countries and 2,420 subpopulations on 9 open models.
1⃣️ Equality is measurable, and it's not the same as accuracy. Simulation accuracy is 1−JSD against real World Values Survey distributions; equality is the cross-country spread, EqCV = σ/µ. GLM-4-9B is the most accurate (0.739) yet far from the most even (EqCV 0.0486), while ChatGLM3-6B leads on both equality (0.0286) and the joint AE score (0.096). A higher average does not mean fairer coverage.
2⃣️ The inequality is systematic and structural. Models simulate Australia, North America and Europe more accurately than the Middle East (Egypt, Jordan, Iraq). Country accuracy correlates positively with GDP per capita, Internet use, the Global Innovation Index and governance quality, and culturally with lower power distance and higher individualism and indulgence.
3⃣️ One failure mode: models pull underrepresented countries toward the US. On "how much do you trust your family," ~60% of Americans but over 95% of Iraqis answer "trust completely", yet GLM-4-9B predicts ~40% for both, erasing Iraq's near-unanimous pattern. Across most questions its simulated Iraqi distribution sits closer to the empirical US one than to Iraq's.
4⃣️ What helps, and what doesn't. Adding retrieved context is the most robust fix, improving both accuracy and equality (GLM-4-9B EqCV 0.0486→0.0255 with BM25). Native-language prompting helps some models but often just relocates inequality to a new dominant group. Continued pretraining lifts targeted languages unevenly, and preference alignment shows no systematic gains, with human-annotated data preserving accuracy better than AI-annotated.
Paper: https://doi.org/10.1162/COLI.a.648
#AI #THUNLP #OpenBMB #LLM #NLP #SocialSimulation #AIFairness #ComputationalLinguistics