Good measurement work on whether retrieved agent skills actually help.

They report that agent skills that lift your aggregate score can be hurting every task they touch.

The usual way of checking compares tasks where a skill was retrieved against tasks where none was.

Those are different tasks, so the comparison mixes the effect of retrieval with the effect of which tasks trigger it.

The fix presented in the paper is a matched comparison. Retrieval-Invoked Actual-Use Effect runs the same task twice, once with skills enabled and once disabled, and counts only tasks where the agent actually retrieved something.

Across 17 LLMs on coding and math, models frequently show positive aggregate retrieval lift alongside a negative same-task effect.

On MBPP+, several models that look beneficial system-wide are hurting themselves on exactly the tasks where retrieval fired.

Anyone maintaining a skills directory can run this against their own stack today.

Paper: https://arxiv.org/abs/2609.00549

Chat with Paper: https://academy.dair.ai/papers/skill-following-evaluating-actual-skill-use-in-retrieval-enabled-llm-agents-2609.00549