elvis@omarsar0 · 2天前64http://x.com/i/article/2071684582336782336
# FW Serverless 2.0: The Routing Pattern
GLM 5.2 has kept open-weight models in the conversation and has everyone wondering how to start leveraging these open models in production. Once you move open models into production, the first thing that breaks under load is not output quality. It is whether the request is served at all. When traffic across the shared fleet exceeds available capacity, Fireworks can reject the request before generation and return a 503 Service Overloaded. The traditional fix has been to buy capacity ahead of time, either reserved GPUs or an enterprise contract sized to your peak. That leaves two bad options. Over-provision for traffic you rarely see, or guess low and eat failures when a spike arrives.
Fireworks Serverless 2.0 (@FireworksAI_HQ) turns that standing capacity decision into a per-request routing decision. Each call can select the serving tier that handles it, so reliability becomes a runtime control instead of a procurement decision. The pattern below keeps live traffic available during congestion without reserving GPUs up front.
## The three serving tiers
Serverless 2.0 gives you three serving tiers behind one API and one endpoint.
Fig. 1. Three synchronous serving paths share one API surface and one fleet. Priority is selected with service_tier, while Fast uses a Fast model ID. Source: Fireworks Serverless 2.0 announcement.
- Standard for everyday traffic. This is your default for production calls. It runs on elastic shared infrastructure and is the most cost-efficient path. Under high platform load, Standard requests are the first to be queued or rejected.
- Priority for reliability under load. Reach for it when a dropped request has real cost, like an interactive session or a long agent run. It gets stronger admission during congestion and is shed last, at a higher per-request price than Standard.
- Fast for latency-sensitive generation. Use it when wall-clock generation time is the bottleneck, such as agent loops, coding workflows, and interactive apps. Fast uses the same model family through an optimized serving path for higher generated-token throughput, not a smarter model or a different reasoning tier.
Same API surface, no capacity reservation. You choose one serving behavior per request. Leave the default model on Standard, add `service_tier="priority"` for stronger admission during congestion, or switch to a Fast model ID for higher generated-token throughput. Priority and Fast solve different problems and are not stackable on one request. Take a concrete case. A chatbot runs fine on Standard until a launch drives a traffic spike and Standard starts returning 503s. Instead of provisioning GPUs or putting users behind a queue, you add `service_tier="priority"` on that endpoint, keep serving through the spike, and switch back to Standard once it passes.
## When to switch tiers
You do not pick Standard or Priority up front. You default to Standard all day, and the moment a request gets shed under congestion (a 503 Service Overloaded, not a rate-limit 429), you flip to Priority for the next 30 minutes, then drift back.
Fig. 2: The escalation policy. Default to Standard, flip to Priority for a 30-minute window on a 503 Service Overloaded, then drift back to Standard once the window expires.
The premium is a control-plane tradeoff, not a new architecture. Priority costs more than Standard for the requests that use it, so the point is to promote only the traffic where a failed request has user-visible or workflow-visible cost. Interactive endpoints and long agent runs get the escalation path. Batch jobs should use Standard, the Batch API, or Background serving when retries and queueing are acceptable. Use Priority only when a 503 would waste expensive multi-step work.
## The code
The code below is illustrative — written to demonstrate the documented Serverless 2.0 pattern, not an official Fireworks code sample. The `service_tier="priority"` field and the 503 Service Overloaded signal are from the Fireworks docs. The control loop, including the 30-minute window and `priority_until` bookkeeping, is our recommended implementation.
The important part is the scope of the fallback. Escalate on 503 because that indicates serving capacity pressure. Do not use the same branch for 429 rate limits, auth errors, invalid requests, or application exceptions. Those are different failure modes and should not silently move traffic into a higher-priced tier.
## Guardrails to set
- Track priority_until, escalation count, and 503 rate in metrics so you can see when Priority is masking sustained load.
- Keep the escalation window bounded. A 30-minute window is enough to ride through a spike without leaving the service permanently promoted.
- Apply the policy per workload or per route. User-facing paths can be promoted to Priority on 503. Evals, offline jobs, and other async batch workloads should use Standard or Background unless a failed request wastes expensive progress.
- Alert if Priority remains active for multiple windows in a row. That is a capacity or traffic-shaping signal, not just a transient failover.
## What Priority costs
Use the Serverless pricing docs as the source of truth. In the current pricing table, Kimi K2.7 Code Priority is listed at 1.5x the Standard row, while Kimi K2.7 Code Fast is listed as a separate Fast model ID at 2x Standard. Pricing varies by model, so always keep the docs as the reference.
The operational point is simple. If a worker needs Priority for a 30-minute congestion window, that +50% per-token premium can still be a useful tradeoff when the alternative is failed multi-step work. For broader cost framing, refer to this article, which reports open-worker plus advisor setups running 19% to 67% cheaper than Opus-as-worker across its benchmark table.
## Which tier for which workload
The pattern matters in the three places AI devs actually ship.
Fig. 3. Routing by workload type. Batch and offline work routes to Standard or Background when retries are acceptable. Fast remains for latency-sensitive generation when wall-clock time is the bottleneck.
- User-facing chat and agents. Interactive traffic is latency-sensitive and bursty. Keep it on Standard and let the first 503 during a spike (a launch, a viral post) auto-escalate to Priority, so users get answers instead of errors and you are not babysitting a dashboard.
- Long agent runs. A single agentic task fans out into dozens of dependent calls, and one shed request mid-chain can sink the whole run. Escalating to Priority after the first 503 protects the expensive, multi-step work where a retry is not free.
- Batch and offline jobs. Evals, synthetic data, bulk embeddings, nightly summarization, report generation, offline analysis, and data enrichment usually care more about throughput and completion cost than instant response time. Keep these on Standard or Background when retries and queueing are acceptable. Use Priority only when a 503 would waste expensive multi-step work. Leave Fast for latency-sensitive generation paths where wall-clock time is the bottleneck.
Because the switch is per call, you run these paths off one codebase. Live endpoints can default to Standard with the escalation guard, long-running workflows can promote to Priority when 503s threaten completion, and async workers can stay on Standard, Batch, or Background. No separate clusters, no separate SDKs.
## Reliability without the cluster
Serverless 2.0 gives teams more room before they need dedicated capacity. Start on Standard, add Priority when overload behavior matters, switch to Fast when wall-clock latency matters, and reserve capacity when you need hard guarantees.
## Links
- Sign up
- Docs
- Serverless 2.0 announcement (tiers, the service_tier parameter, and 503 behavior)
- Coding-model pricing comparison
译Fireworks AI 推出 Serverless 2.0，通过同一 API 端点下的三种服务层级解决共享集群高负载时的 503 Service Overloaded 问题。Standard 为默认经济型；Priority 在拥塞时提供更强准入，价格更高；Fast 通过优化路径提升生成 token 吞吐量，适用于低延迟场景。推荐默认使用 Standard，遇到 503 时临时切换 Priority 30 分钟，随后自动回退。Priority 和 Fast 不可叠加。
宝玉@dotey