This is one of the most useful writeups I have seen on keeping an LLM judge effective in production.

(bookmark it)

Netflix runs judges over hundreds of thousands of show-level recommendation explanations per week, served to millions of members on mobile.

They describe the judge as a lifecycle with four phases rather than an artifact that you only validate once.

Birth defines multiple evaluation criteria and builds curated benchmarks with human labels and rationales.

Training refines the judge's rubric through Reasoning-Aligned Rubric Tuning, using a meta-judge over reasoning output as the learning signal.

Deployment puts one judge in two roles, quality gating and reflective generation.

Monitoring runs continuous human-in-the-loop alignment that detects drift and triggers re-tuning behind a review gate.

A five-week A/B test over tens of millions of members shifted viewing toward previously unwatched content and increased successful browse-to-play sessions against a no-explanation control, with no quality-related takedowns.

Paper: https://arxiv.org/abs/2608.18300

Track more trending AI papers in our academy: https://academy.dair.ai/