very notable trajectory comparison writeup here buried in the RLM paper from @a1zhang and @lateinteraction.

an open secret of "frontier" model training is that even without training on test， you can basically cheat by training on test lookalikes， enabling you to goalseek almost any benchmark number you want.

however when they are released open weights， 99% of the time the norm is that you do not get the datasets/rlenvs that would easily show you if someone was training on Temu Tbench， so there is plausible deniability. Alex and Omar discuss applying standard NLP distance metrics on hidden trajectories. There's no ultimate solution here， but they have some prelim explorations. It happens to support the finding that RLMs can generalize to unseen tasks that share latent structure observed in training.