Ethan Mollick@emollick · XAI 摘要
LLM 在评判定性工作时仍缺乏一致性，作品呈现方式的细微变化会影响结果。 更好的运用和方法（多次评判并随机排序等）肯定有所帮助，但锯齿状前沿（jagged frontier）仍然真实存在。
LLMs are still not consistent judges of qualitative work， and small changes to how that work is presented affect outcomes.
Better harnessing and methods （multiple judging runs with randomized orders， etc） would certainly help， but the jagged frontier is very much still real.
Does an LLM keep the same judgment when you swap the answer order? New LLM Position Bias Benchmark! Judge models compare two lightly edited versions of the same...