Banger compression paper from NVIDIA.

(bookmark it)

Bigger MoE models keep winning on quality, but serving them at interactive latency is still hard.

NVIDIA compresses the hybrid MoE Nemotron-3-Super into Puzzle-75B-A9B and roughly doubles interactive server throughput while holding quality.

Pay attention to the joint structural search. Heterogeneous MoE pruning, active-parameter budget, and Mamba pruning get optimized together rather than one at a time, wrapped in an iterative pipeline with distillation, RL, quantization, and a Multi-Token Prediction head.

Why does it matter?

On a single 8xB200 node it hits about 2x the parent's server throughput at matched user-throughput, and 1M-token concurrency on a single H100 climbs from 1 request to 8. Accuracy holds across reasoning, coding, long-context, and agentic benchmarks.

Cheaper serving with agentic capability intact changes what you can afford to run with these models.

Paper: https://arxiv.org/abs/2607.04371

Learn to build effective AI agents in our academy: https://academy.dair.ai/