New Microsoft paper argues that transformers generalize better when they learn compact internal states， not just next tokens.

The problem is that normal transformers can look back at every earlier token， so they do not have to squeeze the past into a clean summary. token prediction alone can reward shortcuts that do not become coherent world models.

That can work beautifully on familiar data and still fail when the model has to plan， detour， reason， or carry a hidden structure forward.

NextLat fixes this by adding a training task where the model must predict its next hidden state， not just the next word.

A hidden state is the model's private summary of what it has seen， so predicting the next one pushes the model to learn how situations change over time.