Update (29 July 2026): We have run Kimi K3 through the same setup, served with SGLang. At 1.4TB of weights, K3 does not fit within the memory budget of the 8×B200 node used for GLM-5.2 (1.5TB of total HBM leaves no headroom for KV cache). This run therefore used an 8×B300 node, which brings 288GB of HBM per GPU instead of 192GB, or 2.3TB per node. That averages out to around 20% higher hardware cost than the 8×B200 setup, depending on your rental provider.

In our runs, K3 served 16 concurrent sessions (GLM-5.2 managed 24). Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users), and median task time is about 50% longer (38 vs 26 minutes). That makes K3 roughly 8 times slower than our Claude Code baseline. However, K3 makes up for it in quality, resolving 86.4% of tasks, 24 percentage points above both GLM-5.2 and Opus 4.8 (62.5% for both). One caveat is that our benchmark tasks from SWEBench Pro may have been in K3's training data, so treat that resolution rate with a grain of salt. Only the graphs in this article have been updated with the new K3 data.

01

Why your token bill
blew up this year

The warning signs were everywhere the first half of this year: GitHub Copilot switched to token-based billing [1], Uber burned through its annual AI tools budget in just four months [2][3], and developers reported projected cost jumps from $29 to $750 a month [4]. We even invented words for it: "tokenmaxxing", "token panic", "token budgets".

The median employee [5] currently spends ~$140 per year on AI API usage. Sounds harmless, until you look at the tail: the 90th percentile nears ~$7,300 per year, and the 99th approaches ~$90,000. For many organisations agentic AI adoption is only just beginning, so expect these numbers to change in the year ahead.

Why is the bill so volatile? Because of how we use AI agents: Currently, over 70% of ARR across major model providers [4] comes from coding use cases. The more of your workflow you hand to agents, the higher your token bill becomes.

Many organizations start their AI coding agent adoption using one of the major frontier model providers like OpenAI or Anthropic through API access. A few months in, more and more organizations are at least considering the alternatives for a number of reasons: token pricing models become more costly, users are upgrading to newer and more expensive AI models and agent adoption in the organization starts to boom. There are sensible alternatives out there that are certainly worth a closer look: switching to API routers; using cheaper model tiers for routine work while only escalating the hard 20% to a frontier model and others.

However, if your token consumption becomes significant or if you're working with sensitive data, you might want to abandon the API approach altogether and setup your own inference stack. When ownership of hardware is on the table, you'll need to understand what renting or owning a GPU actually gives you in return. When does it make sense to make that switch? Read on as we give you the tools to decide for yourself.

02

You pay for the pool 24/7,
your team uses it 8 hours a day

So you've decided to investigate whether buying or leasing your own GPUs makes sense. Once configured correctly, it will give you an API endpoint you can plug into your coding tools, much like Anthropic, OpenAI or other model providers offer.

In contrast to the frontier APIs, your own endpoint has a ceiling on output tokens per second but you only reach it when the system is fully loaded. One developer working alone leaves most of that ceiling unused, but the same hardware serving forty-eight agent sessions in parallel is a different story. Your ceiling is determined by the model, the GPU(s), and the serving configuration.

媒体内容 · 前往原文查看

Unlike an API usage model where you are charged per token consumed, running your own setup signifies you pay for the underlying infrastructure and the cost of keeping it running. In all but a few cases that means you're also paying for it even when you're not generating tokens. Whether that trade-off works for you, is decided by two questions. The first is how much of the pool are you using. Let’s tackle that one first.

You size for the peak and pay for it 24/7. Utilisation, not headcount, is what makes or breaks the cost case.

Real demand is not constant, but spiky. For many organizations, tokens for coding tasks will be near-zero overnight, ramp up through the morning, dip visibly at lunch, and peak again mid-afternoon. This means you buy hardware for the peak load, but you pay for it 24/7 (unless you rent ‘GPU spot instances’, but these are harder to rely on in this case). The shape of your peak in usage matters as much as the volume: a single-timezone team concentrates token demand into narrower spikes than the same headcount spread across timezones: same tokens, higher peak usage, but spread out longer. Published data on enterprise inference workloads serving internal developer tools reports average GPU utilization of 15–22% [6][7], though even a well-run deployment currently rarely exceeds 25–35% (and that is being generous). The graph below shows the real, actual token usage over 2 days a couple of weeks ago from our friends over at TechWolf[14], reflecting exactly what we describe above.

媒体内容 · 前往原文查看

One emerging trend may work in your favor here: As organizations mature in their agent driven workflows, more and more sessions are starting to come from automated agents rather than people (scheduled jobs, agents autonomously picking up a bug report at 3am, etc.). This allows for work that doesn't adhere to office hours. If scheduled with intent, these automated sessions can soak up overnight capacity you are paying for anyway.

Now let’s tackle the second question, how many developers can share the pool before it feels slow?

03

What happens when
48 developers share a box?

A token pool occupied by just one developer feels wonderful - tokens stream in faster than you can read. The question is what happens at five, ten or twenty concurrent developer sessions, because AI coding agents are greedy, bursty clients that not only consume tokens, but also browse the web, execute files or compile code.

One of your main concerns when moving away from a frontier model provider API should be developer experience. When looking at benchmark results you will often see tokens-per-second used to gauge how well a model performs on a given GPU setup. While it is an easy to understand metric, we find it less useful in this context. After all, you can burn thousands of tokens with no satisfiable result, or you can switch to a smaller model that ends up using 10 times more tokens.

The better unit of value delivery is a task successfully completed, and while that is harder to quantify in general, that's what we'll often return to in the analysis below. You will want to understand how long a task takes start to finish and how fast tokens stream while the agent is waiting on the model.

The better unit of value delivery is a task successfully completed.

To make task completion measurable we use a subset of tasks from SWEBench Pro by ScaleLabs, a popular set of longer-horizon code engineering tasks. While it is most often used to pit individual models against each other, our team is using it specifically to create a more lifelike and challenging enough workload as we compare the cost and performance of different model and hardware combinations under real use cases that would make sense for your organization.

A couple of graphs will help us further evaluate the performance of the model & hardware combinations we have chosen. First, we run our subset of coding tasks using a given model on a specific hardware setup at increasing levels of user concurrency and evaluate how this impacts the time to complete individual tasks. This will help to find that sweet spot of how many concurrent users you can have active for your chosen setup before tasks simply start taking too long to complete.

媒体内容 · 前往原文查看

It is convenient to see this information together. So often the trade-off between how much the hardware is in use vs. the developer experience in terms of speed is shown in one graph that allows for convenient decision making. We’ll use the graph below to show how different model / hardware combinations approach the maximum token output they can generate as user concurrency increases, while keeping an eye on how long we can keep our developers wait for a task to be resolved.

媒体内容 · 前往原文查看

Understanding these values will set you up to better understand the differences in performance and developer experience across the options that may be a match for your organization’s needs.

Over the last weeks our team was hard at work to do ~100 runs of a selection of 64 real coding tasks from SWEBench Pro across different and appropriate hardware configurations to calculate precisely the right metrics (median time per tasks, tok/s/u and many more). Guided by open-weight quality rankings from Artificial Analysis [9], we paired each hardware tier with a leading model that fits its memory budget:

• Qwen3.6-35B-A3B-FP8 on a single H200 served with vLLM
• DeepSeek-V4-Flash on 4×H200 served with vLLM
• GLM-5.2 on the 8×B200 rack served with SGLang.
• Qwen3.6-35B-A3B-FP8 on NVIDIA's DGX Spark served with vLLM

In the next sections we’ll dive into the actual results and round it up with a cost analysis comparing buying your own GPUs to renting them to buying tokens from either the frontier model providers as well as open weights API providers. We feel these are the kinds of questions many organizations are pondering at the moment and we wanted to give you the tools to answer them.

04

From a Spark on a desk to an 8×B200 node: which box to buy

If you’re interested in renting or buying hardware, several options are available. Below we introduce four popular contenders that we’ll come back to throughout this post. Each of those also comes with a best-in-class open weights model that nicely fits the hardware.

You’ll see that we’re leaning heavily into NVIDIA here, which has a lot to do with what was easily available to us. There are definitely similar alternatives from other suppliers, which we’ll get to in later posts. For now, consider these as reference choices rather than specific endorsements.

• Let’s start with the cheapest option, an NVIDIA DGX Spark. This is the box for a team that wants to try self-hosting without talking to procurement: it sits on a desk and costs less than a conference trip. It can fit Qwen3.6-35B-A3B (referred to as "Qwen3.6"). That’s a surprisingly big model for such a machine - We’ll see later in the article what it delivers in terms of developer experience.
• Next up is the NVIDIA H200, a powerful, single accelerator. Perhaps surprisingly, It actually does not buy you a lot more in terms of the models you can host compared to a DGX Spark. This is mostly determined by the available memory (128GB for the DGX Spark vs. 141GB for the H200). However, what you do get is a bigger token pool which translates in throughput speed for higher concurrencies.
• If you want to host more powerful models, but don’t want to fully break the bank, NVIDIA HGX H200 with 4xH200s is a solid option. With 4xH200s, you can host bigger models such as DeepSeek-V4-Flash, which promise a very good compromise in terms of speed, concurrency and quality.
• Taking it up a notch, the HGX B200 with 8xB200’s is a powerful reference system, as it lets you serve state-of-the-art open source models like GLM-5.2 that currently come closest to the frontier models in terms of quality. This is your option if quality is the main concern.

At the time of writing this article, there is a lot of buzz about Kimi’s K3 model, as early benchmarks (which will need to be confirmed) indicate it would be able to compete with Anthropic’s Fable model, but at much lower cost. For now, our team is awaiting the open weights release for this model later this month to evaluate what it would take for this 2.8T to run on your own infrastructure. More on this soon.

Next up, we’ll put these contenders to the test in terms of developer experience, cost, and quality of the models they can fit. Let’s do this.

05

How big are those pools?

So what about having a DGX Spark sitting on your desk within reach? Well, unsurprisingly, it does not come with a big token pool. In our tests, we could not push it beyond 1-2 concurrent users and were confronted with an avalanche of time-out errors and the likes. If you’re happy with Qwen3.6, it can be a fit for you, but don’t expect it to scale in speed, concurrency or model quality. It shows that - despite Linkedin headlines - we're not there yet that we can simply download frontier intelligence on an affordable desktop box. Simpler things like code completion? That is less of an issue.

媒体内容 · 前往原文查看

This is where paying extra for that professional H200 pays off, for the same model it can generate at least 30 times more tokens/s for Qwen3.6. Actually, don’t be surprised that Qwen3.6 generates so many tokens, even on a single H200 - it’s still quite a small (and thus less capable) model. We can also see here that DeepSeek-V4-Flash on the HGX H200 comes with an even bigger token pool (of course on 4xH200s) and only reaches that potential at high concurrency. In contrast, for GLM-5.2 on the HGX B200, the token pool is smaller despite it running on the most powerful hardware in our tests. Indeed, 16 concurrent sessions seems to more or less saturate the GPUs and the total pool is only about 175 tokens/s for GLM-5.2. Bigger, more powerful models come with smaller token pools, and the hardware we selected doesn’t fully compensate for that.

Probably you noticed something surprising and uncomfortable - some lines crash down instead of tapering off to a maximal value. We’ll zoom in on that after we look into the developer experience.

06

How happy are my developers with the speed?

You may not want to max out your token pool, as individual developers may perceive your solution as a snail compared to their trusted Codex or Claude companion. With the latter as a baseline we can see that GLM-5.2 struggles already from 8 concurrent sessions onwards despite the impressive HGX B200 hardware. Just as with the relatively small pool size, we see here that serving a frontier model at scale is very challenging. On the other end of the spectrum, the DGX Spark is slow even for 1 user, and simply unusable for multiple users.

Both DeepSeek-V4-Flash and Qwen3.6 are holding up longer, up to 32 users and then take a dive. Note that DeepSeek-V4-Flash is close to Qwen3.6-35B-A3B but runs on more expensive hardware - trading off that investment for higher quality while maintaining speed under load. Of course, for both, compared to GLM-5.2, the price you pay is the lower quality, more on that later.

媒体内容 · 前往原文查看

So ballpark, you can expect to host around the following number of concurrent users on each of these setups before developers start knocking your door down complaining about the speed:

Model	Comfortable concurrency
Qwen3.6 (DGX Spark)	just 1 — and still slow, almost 3× slower than Claude Code
Qwen3.6 (H200)	easily 32 and at 48 you are 2X slower than Claude Code
DeepSeek-V4-Flash	easily 32 and at 48 you are 2X slower than Claude Code
GLM-5.2	8 is pushing it — already 3× slower than Claude Code

And now for the elephant in the room: both in this plot and the next ones, you see Qwen3.6 on H200 and DeepSeek-V4-Flash on the HGX H200 take a nose dive after concurrency 48 and 64. That’s surprising and not something you expect intuitively, as theoretically the lines should converge to the maximal output of the GPUs. So why do we see a collapse so early on? The reason has to do with the specifics of the inference engine (vLLM), its default parameters and how it balances compute between prefill and decode requests, as well as important resources such as the effective KV-cache size and its usage.

Additionally, you are perhaps disappointed by the concurrency level provided by GLM-5.2. One culprit could be the usage of speculative decoding which speeds up token generation for low concurrency but can hurt performance for higher ones.

But don’t despair too quickly, if you optimize the inference engine (e.g. vLLM) for your workloads, you can deal better with such issues and get even more out of the system - we’ll explain and discuss this in depth in an upcoming blogpost.

Back to the plots. Be aware that at these concurrencies, keeping your devs happy, you still leave some tokens on the table. We can combine the info of previous plots in one to bring out this gap for each of the systems. For instance, for DeepSeek-V4-Flash on HGX H200 at c=32, with happy devs, you only use 1381 tokens/s, while the pool for c=64 is 1717 tokens/s, or 24% more.

媒体内容 · 前往原文查看

Be aware of this trade-off, while you can adapt it in real life settings, don’t simply expect you will get the full pool out of a system as it’s often advertised.

07

How much will this all cost me?

When it comes down to how to invest in your AI coding agents, your choices are not limited to sticking to your model API provider of choice or outright buying the hardware. There’s also the option of renting your GPUs, with suppliers ranging from Hyperscalers to NeoScalers, AI Factories or other parties. Each come with different prices, but also different guarantees, services and integrations.

As always, model quality matters a lot. To compare fairly, we also reran our SWEBench Pro task set against the official APIs of the different model providers and kept track of the token costs. For buying, or renting, hardware, the cost is determined by how long it takes to run such a set of tasks. As discussed, this depends on the concurrency. For each of the systems, we take the optimal concurrency from a cost perspective, the one that uses the GPU optimally. Note this may be a point that impacts user experience.

We did not include DGX Spark in this comparison as it really is a different beast in terms of pricing and performance. Qwen3.6 buy/rent costs are for 1 H200, for DeepSeek-V4-Flash it is the HGX 4xH200 and for GLM-5.2 the HGX 8xB200. We depreciate those costs over 5 years, but do not include additional power or maintenance costs as those depend a lot on organisation specifics.

System (Model/HW/Walltime)	API Cost	Buy Cost	Buy Cost (30% usage)	Rented Cost
Qwen3.6 (1xH200, 28 min)	$57.67*	$0.43	$1.43	$1.62
DeepSeek-V4-Flash (4xH200, 31 min)	$2.13	$1.90	$6.33	$7.19
GLM-5.2 (8xB200, 101 min)	$92	$13.84	$46.12	$71.23
Anthropic API (Opus 4.8)	$98	N/A	N/A	N/A

*Alibaba list price (no caching for the model)

*Hardware purchase and rental prices are publicly advertised rates as of July 2026 [10][11][12][13].

Same workload and three completely different answers. Most relevant fun takeaways from our calculations: a 4×H200 has to stay 89% busy (day and night, all five years) before owning it beats DeepSeek's $2.13 invoice, while the B200 rack only needs 15% utilisation to beat the frontier APIs, and at rental rates it already sells frontier-quality tasks at $1.11 apiece (rented cost divided by 64 tasks), cheaper than either API charges.

If you’re under that utilisation threshold of e.g. 15%, renting GPUs can be a better and more affordable option as it allows you to turn off usage during off peak hours. Is it? Well, surprisingly, it strongly depends on the model type. For Qwen3.6, renting a GPU is a factor 35 cheaper than going API. We also see a cost reduction for GLM 5.2, although a more moderate one of 23%. However, for DeepSeek, the numbers are the opposite. Without any optimization specifically for your workload, it’s actually a lot more expensive than DeepSeek’s own API.

What matters when taking the cost decision is less of the open vs closed debate, and more what running your specific workloads cost. For coding agents, up to 98% of input tokens are cached, and this is where the DeepSeek-V4-Flash API pricing beats even the smaller Qwen3.6 model by a landslide.

There's no single winner. Priced at real-world utilisation, buying only pays off if you keep the box busy — a 4×H200 needs to stay

loaded, the B200 rack just

. Below that, renting or a cheap open-weights API often wins. Buy GPUs for what an invoice can’t price — control, privacy, latency — not for cost alone.

08

But I like the closed frontier models from the labs?

And you are not alone. Yet here you are, reading about moving away from them. Truth be told, while open-weight models are always lagging a couple months behind the frontier models from Anthropic, OpenAI or Google, the best of them have become very usable in 2026. Moreover, the gap between open-weight and closed models is closing as we speak with new releases such as GLM 5.2 and Kimi K3. For those, like GLM-5.2, you’ll need at least an HGX B200 system. Don’t expect the same performance when going with a H200 or even 4xH200s.

The gap between open-weight and closed models is closing as we speak.

For the tasks we ran, we reproduced the prevailing insights by model quality evaluators such as Artificial Analysis [9]. For completeness sake, here are the resolution rates (pooled over all runs and concurrencies) for the models in question with Claude Code as harness on our sample of 64 tasks from SWE Bench Pro.

Model	Resolution Rate
Kimi K3	86.4%
GLM-5.2	62.5%
Anthropic Opus 4.8 (API)	62.5%
DeepSeek-V4-Flash	39.1%
Qwen3.6	35.4%

Remember though that such benchmarks are only a proxy for how good a model will work for your use case - and you will have to test that for yourself. Luckily, just for the quality of the model, you can judge that with free or cheap APIs in a limited test.

09

So how many developers fit on a GPU?

Between 0 and 64. Or at least according to the configurations we looked at — but the fact that it’s a range is the point we’re making here. If you’re serious about investing in your own hardware, here are some key takeaways that can be relevant.

• Your DGX Spark is really only for single use. It will fit one developer on a good day with a small model and a moderate token pool. If you’re looking for scale, this is not your best bet. If you’re looking for individual developer experience on moderate tasks, it’s a great workhorse.
• However, a single H200 running Qwen3.6 can easily serve 32 concurrent sessions, and 4×H200s with DeepSeek-V4-Flash can do the same at higher quality. Stack eight B200s together and your team is ready to serve a near-frontier model like GLM-5.2 — but suddenly we’re back down to eight developers before task execution starts absolutely crawling.
• Renting GPUs can be a real cost saver, but it highly depends on the model. In our tests we saw up to a factor 35 cost saving (!) in the case of Qwen, but higher costs than the API for DeepSeek. This shows how having the right workload to model and hardware mapping is key and that switching from one infrastructure to the other is not just a decision that gives you a flat cost saving.
• If your workload exceeds a certain threshold, buying your own hardware might become interesting. It depends which combination of quality, speed, and cost your organization actually needs. If your workload could keep a rented B200 rack even 15% busy, you’re — surprisingly enough — already en route to beat frontier API pricing on near-frontier quality. If it doesn’t, don’t self-host just for the sake of escaping your token bill, as an idling GPU rack costs more per task than your current API invoice. There might be other reasons such as sensitivity of data.

Your mileage may vary of course. Hence, measure your own workload. Get a better view on your peaks. And keep on doing it as your organization matures into using AI coding agents. While your current numbers may point towards API usage for now, in a few months’ time this may be much different. In the meantime we’ll be out here building tools and insights to help you with that puzzle.

If you want to squeeze more devs onto the same hardware, that’s exactly what our next posts are about: tuning the inference engine to fit your workload, sometimes doubling what one box can do. Stay tuned.

Contributors

Steven L., Wouter V.d.B., Michaël M., Ioana F., Jan F., Christian M., Maxim C., Arathy U., Bohdan D., Sepideh P., Baptist V., Jeroen B., Robbert V.C., Mathias F., Hendrik M., Jannick V.

References

Numbers cited above trace to these sources.

1. GitHub Blog (Mario Rodriguez), “GitHub Copilot is moving to usage-based billing” (April 2026) github.blog
2. Forbes (Janakiram MSV), “Uber Burns Its 2026 AI Budget In Four Months On Claude Code” (May 2026) forbes.com
3. Bloomberg (Natalie Lung), “Uber Caps Employee Spending on AI Tools Like Claude Code to Manage Costs” (June 2026) bloomberg.com
4. SemiAnalysis (Crystal Huang, Joey Brookhart, Dylan Patel), “TokenBudgeting: Our Conversations with Enterprises on Token Spend” (June 2026) newsletter.semianalysis.com
5. Ramp Economics Lab (Ara Kharazian), “How much does it cost to be AI-pilled?” — Ramp AI Index (June 2026) ramp.com
6. Spheron (Mitrasish), “AI Inference Cost Economics in 2026: GPU FinOps Playbook” (April 2026) spheron.network
7. VentureBeat, “5% GPU utilization: The $401 billion AI infrastructure problem enterprises can’t keep ignoring” (May 2026) venturebeat.com
8. Hebbia (Jake Skinner), “The Hidden Economics of LLM Inference” (April 2026) hebbia.com
9. Artificial Analysis, model comparison: Claude Opus 4.8, DeepSeek V4-Flash, Qwen3.6-35B-A3B, GLM-5.2 (accessed July 2026) artificialanalysis.ai
10. TRG Datacenters, “NVIDIA H200 Price Guide” (accessed July 2026) trgdatacenters.com
11. GetDeploying, “H200 Cloud Pricing: Compare 34+ Providers” (accessed July 2026) getdeploying.com
12. Tech-Insider, “NVIDIA Blackwell GPU Pricing 2026: B200, B300 & DGX Costs” (July 2026) tech-insider.org
13. Spheron, “NVIDIA B200 Cloud Pricing 2026: Per-Hour Rental Cost Across Providers” (June 2026) spheron.network
14. TechWolf, internal developer token-usage data shared with aistack (July 2026) techwolf.com

Appendix: Our methodology

Core to our philosophy is that the details of the real-world workload matter when thinking about your AI stack - both software and hardware. To measure this, we set up a system that mimics as much as possible the behavior of coding agents. The parts of the system that can be configured are on the one hand the inference engine:

• The underlying hardware. We tested
  • DGX Spark (Lenovo ThinkStation PGX) sitting on our desks
  • a single H200 hosted in our on-premise datacenter (with 24 CPU cores, 64GB memory)
  • a HGX 4xH200 connected via NVLink hosted in our on-premise datacenter
  • a HGX 8xB200, hosted by a dedicated AI cloud infra provider
• The different models as discussed
  • Qwen3.6-35B-A3B on DGX Spark with vLLM settings from https://recipes.vllm.ai (using MTP speculative decoding)
  • Qwen3.6-35B-A3B on H200 with vLLM settings from https://recipes.vllm.ai (using MTP speculative decoding)
  • DeepSeek-V4-Flash hosted with tensor parallelism 4 and the vLLM settings from https://recipes.vllm.ai (we adapted the 8xH200 recipe, using MTP speculative decoding)
  • GLM-5.2 hosted with tensor parallelism 8 and the SGLang settings recommended by https://modal.com/blog/introducing-auto-endpoints (using EAGLE speculative decoding)

On the other hand, we need to mimic the workloads, we can configure

• The number of workers running agentic coding tasks in parallel against the endpoint. Those are different docker containers, each getting a coding task from the in the Harbor task format[reference]. Those workers are hosted on the infra of a sandbox provider and run on commodity CPUs.
• Each task run consists of three phases.
  • a setup to ensure prerequisites are installed
  • the actual code generation loop which drives requests to inference endpoint
  • a final evaluation to check if the generated code passes tests
• We opted to not remove the first and second phase as they act as a basic resemblance to the work a developer does outside of coding - like grabbing a coffee in between two tasks. However, we can separate them out when required for different analyses. E.g. task time in our plots is reported on the actual code generation only. The inclusion of the setup and evaluation lower the effective concurrency of the endpoint as at any time some agents are in such a phase and not sending requests.
• As the fixed task set we sampled 64 tasks from SWEBench Pro by ScaleLabs. We use our own platform to orchestrate this, but have reproduced numbers obtained with Harbor to ensure correctness.

On both inference engine and workers we capture many metrics that allow for a deep analysis of how the workload interacts with the underlying hardware and software inference stack. These allow us to optimize the different settings further, to fit more devs on the same GPU. We discuss this in a following blogpost.

To measure tokens/s, we look at sustained throughput in the bulk of the taskset, e.g. excluding the end of the queue when only few straggling tasks remain. Note there are individual short peaks that go a lot higher (especially for GLM-5.2), but for the workloads under tests, these levels are not consistent for longer times.

There is a variance in observations, while it is small for tokens/s, task times and resolution rate (we average it over all runs) it is especially outspoken in the occurrence of the collapse of the throughput of Qwen3.6 and DeepSeek-V4-Flash at higher concurrences. This has to do with the stochasticity of the incoming requests when using so many workers. We have opted to pick the lowest concurrency level at which it happens, as this indicates the setup is no longer reliable for production purposes.

Read next

Soon

Proofline to reality

Agentic tools spend much of their run waiting on tool calls — and that idle GPU time is where the money leaks.

Coming soon

Soon

How to fit more devs on a GPU

Yesterday’s rack assumed steady jobs. Today’s bursts leave silicon stranded — and the bill still arrives.

Coming soon