It stops being all about the vibes when the API bill hits you. Many companies are now discovering that AI can indeed be pricey. One habit that might be driving up your AI bill is comparing models by $X per 1M tokens. A lower number should mean lower costs, right? Well, not really.

$X per 1M tokens is incomparable

Each frontier lab has its own tokenizer, which determines how many tokens a body of text is split into. For example, all text in this post so far would’ve been split into 160 tokens for gpt-4o, but that same input would cost you 200 tokens for gpt-4 (1106-preview, generated with tiktokenizer.vercel.app). Even within one frontier lab, OpenAI in this case, model pricing per token is incomparable. Comparing numbers between different labs, especially when they’re constantly tweaking proprietary tokenizers, introduces an error that is hard to measure reliably. Anthropic has recently modified its tokenizer, which resulted in Claude splitting the same text into 30% more tokens. Ceteris paribus, this would be equivalent to a rather steep price hike; however, there is another important factor to take into account.

Extreme variance of token efficiency

Even if we ignore the influence of the tokenizer, the other important factor is how much one more token is actually worth. I don’t mean the price of the token, but how much you actually achieve with it. If you’re using AI for serious work, chances are that most of your token consumption is spent on “thinking”, which is often hidden or obscured but billed at the same rate as visible output tokens. This technique can greatly improve output quality; however, the length of that so-called “chain of thought” can become the main factor influencing your overall cost of AI usage — and this can vary wildly.

I’ve picked some of the best current AI models from American frontier labs as well as the best offerings from Chinese labs (which are often pitched as almost as good as American models but for 1/x the cost, often x > 10) and put them in a table below. I’ve also included each model’s score in the Artificial Analysis benchmark, which gives AI models tasks to complete. The goal of AA’s researchers was partly to measure model capabilities and partly to measure how much they were billed for each completed task.

Notice that even though GPT-5.5 is nominally more expensive than Claude Opus 4.8, it completes the benchmark at almost half the cost per task compared with Anthropic’s model. GLM-5.2 is much cheaper per token than both GPT (3.57×/5.68×) and Claude (3.57×/6.82×); however, its cost per task is not proportionally lower, suggesting that it’s less token-efficient than frontier models from the West.

One model that perplexes me is Sonnet 5, since it seems to perform worse than Opus 4.8 while also requiring a higher cost per task due to much lower token efficiency. If someone using it could explain to me what the purpose of this model is, I would be glad to listen. (Conspiracy theory: maybe it’s some sort of psy-op by Anthropic to have a lower sticker cost to coax people into using a less token-efficient model that will ultimately raise their bills?)

DeepSeek V4 Pro seems like the strongest cost-efficiency outlier. Although it scores clearly lower on the intelligence benchmark, its cost per task is extremely low. Fable 5 (Mythos with a security muzzle) seems to show a modest improvement with a price hike of more than 3× compared to GPT-5.5.

Overall, I think this table shows that price per million tokens isn’t a meaningful cost indicator. If you don’t consider the actual cost per task, you will make worse model-selection decisions and be left with inferior performance for a higher price.