Netflix has explained the system behind those short "because you watched" lines: an AI writes them, another AI grades them, and humans audit weekly.

An AI judge is usually validated once and then trusted forever.

New Netflix paper argues a judge running in production has a lifecycle and needs maintaining like any other model.

At Netflix, one model writes the short lines telling members why a title was recommended, and another scores every one before it is shown.

Netflix splits that work into four stages: building labelled examples, tuning the judge, running it as a gate, and watching it for drift.

Checking whether the judge agrees with human labels is not enough.

It also has to reject a bad explanation for the same reason a person would, so tuning runs on written rationales rather than pass-fail marks.

A weekly human review then sets the bar by how far the raters disagree among themselves.

In a 5-week test against no explanation at all, members shifted slightly toward titles they had not watched and more often ended a browse by playing something.

– arxiv. org/abs/2608.18300

Title: "The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations"