Very relevant Microsoft paper on agent reliability.

Succeeding once and being reliable are not the same thing.

The best agent solved 91% of business tasks at least once but only 25% every time, so measure repeats.

The failures are also hard to spot from the outside.

4 out of 5 failed runs ended politely and called a tool that writes to the database.

The agent said the job was done, but the records said otherwise.

So Microsoft built ThinkingBox, a sandbox that runs an agent against real tools, a simulated customer, and a live backend, then checks the database afterward instead of reading the reply.

---

– arxiv. org/abs/2608.19741

Title: "One Success Isn't Reliability: Thinkingbox, a Sandbox and Benchmark for Agents in Stateful Business Workflows"