An 8 GB gaming laptop can run a 35B model at 39.3 tokens a second, faster than Codex in production, if the software stops guessing which parts of the model to keep on the GPU.

That's FreeToken, New research from Berkeley and University of Texas, a new engine for running big open models on ordinary machines.

Every other local engine picks which experts sit on the GPU at load time and leaves them there, so when the model routes elsewhere, that expert gets copied over PCIe while the CPU sits idle.

FreeToken lets the GPU cache follow whatever the router just asked for, cutting missed experts from llama.cpp's 62% to 16%.

The rest it splits two ways: some copied to the GPU, the others run in place in system RAM, in a ratio computed from your machine's real PCIe and memory speeds, so neither side waits on the other.