Serverless GPU Inference Compared: Baseten, Modal, RunPod, Together and Fireworks

Self-hosting a frontier open-weight model stopped being a weekend project some time ago. GLM-5.3-Flash carries 320B total parameters with 18B active, and its FP8 weights come to approximately 328 GB. That fits in no single accelerator you can buy off a shelf, so serving it is a multi-node problem before you write any application code: interconnect, a parallelism strategy, a scheduler, and someone on call when a node drops out. Even the smaller end of the frontier is heavy — Qwen3.8-Flash in NVFP4 lands at approximately 135 GB, which is why a well-known community deployment of it runs on two 128 GB DGX Spark units.

Managed and serverless GPU platforms exist so you can skip all of that. The trouble is that "serverless GPU" is marketing vocabulary covering at least three genuinely different product categories, with different billing models, failure modes and operational burdens. Teams routinely buy one category while budgeting for another, and the gap shows up as a surprise invoice or a latency regression nobody can explain.

One caution first: published rates, instance availability and free-tier terms in this market change frequently, sometimes monthly. Nothing below quotes a provider's current price, because any figure written today is likely wrong by the time you read it. Treat this as a framework, and compute the numbers against current published pricing at the moment you decide.

Three products, one name

Category 1: token APIs for open-weight models

You send a request to an HTTP endpoint, get tokens back, and pay per million input and output tokens. You never see a GPU, never choose a serving engine, never think about batch size. Someone else runs a large shared fleet and amortises it across every customer.

This is the cheapest and simplest option for standard, popular open models, and the floor has fallen hard. As one illustrative datapoint of first-party pricing at the time of writing, Qwen3.8-Flash-Next is offered on a first-party cloud API at $0.16 per 1M input tokens and $0.47 per 1M output tokens. At that order of magnitude, a service processing tens of millions of tokens a month spends less on inference than on the engineer who would configure a GPU.

The constraints are real. You get the model as the provider chose to serve it — their quantization, context limit, sampling defaults and rate limits. You cannot deploy a fine-tune unless the platform supports it. Your data crosses a boundary you do not control. And you inherit the provider's capacity: when a popular model saturates, your latency is their queue depth.

Category 2: managed model deployment and serverless endpoints

You bring a model — a checkpoint from a public hub, your own fine-tune, sometimes an arbitrary container — and the platform turns it into an autoscaling endpoint. It provisions workers, pulls weights, runs a serving engine, load-balances, scales replicas under load and, on most such platforms, scales to zero when idle. You pay for compute time rather than tokens.

This category most deserves the name "serverless GPU": control over the model and the serving configuration without operating a cluster. It costs more per token than a shared API at low volume, because you rent capacity instead of sharing it, and it introduces the cold-start problem below.

Category 3: raw GPU rental and container platforms

You rent GPUs by the hour or second, on-demand or from a spot or community pool, and run whatever you like. Some providers add a thin serverless container layer that blurs the line with category 2, but the mental model differs: you own the stack — serving engine, tuning, model loading, autoscaling and monitoring.

Maximum control, maximum operational burden, typically the lowest raw cost per GPU-hour. Right when you have unusual requirements or enough sustained volume that the engineering time pays for itself; wrong when your team's scarcest resource is attention.

The most common expensive mistake is buying category 2 or 3 for a workload category 1 would have served — paying for dedicated capacity to run a stock open model at low, spiky volume, and idling most of it.

Where the providers sit

These are durable positioning notes, not feature lists, because feature lists here go stale within a quarter.

Baseten

A model deployment platform: you define how a model is packaged and served, and the platform handles autoscaling, endpoints and observability. Its centre of gravity is production inference performance — making a given model serve faster and more reliably on given hardware, rather than breadth of general-purpose compute. Its publicly visible inference research and technical writing is a useful signal, consistently concerning serving-engine behaviour, decoding techniques and throughput under concurrency. Vendors write about what they invest in.

Best for: teams shipping inference as a product surface who want engineering depth on serving performance without running infrastructure. Wrong choice when: you want raw GPUs for training or batch work, or your model is a stock open one a shared API would serve for less.

Modal

General-purpose serverless compute with first-class GPU support and a Python-native developer experience: you decorate functions, declare the image and hardware in code, and deploy. Inference is one workload among many, alongside batch jobs, data pipelines and fine-tuning runs. That generality is the point: if inference sits inside a larger pipeline, keeping all of it in one programming model is worth a great deal. It also means the platform is not narrowly tuned for one job.

Best for: Python-heavy teams with mixed workloads and custom pipelines, where inference is a step rather than the whole product. Wrong choice when: you want a turnkey endpoint for a popular model with no code, or infrastructure-level control over machines.

RunPod

GPU rental with a serverless container layer on top, positioned close to the infrastructure and oriented toward cost and community. You get GPUs, including community-supplied capacity, and bring your own stack. It is the most "here are the machines" of the five. The trade is explicit: less platform, lower cost, more operational work on you, with reliability varying by capacity pool — a choice to make deliberately rather than by default.

Best for: cost-sensitive teams comfortable operating their own serving stack; experimentation, fine-tuning, workloads where GPU-hour price dominates. Wrong choice when: you need strong tenancy and compliance guarantees for regulated data, or nobody wants to own an inference stack at 3am.

Together

Hosted APIs across a broad catalogue of open models, plus dedicated deployments and fine-tuning. It spans categories 1 and 2: start on the shared API, move to dedicated capacity for the same model when volume or isolation justifies it, fine-tune in place. That continuity removes the usual cliff between prototype and production.

Best for: teams wanting a wide open-model catalogue on day one and a path to dedicated capacity and custom weights later. Wrong choice when: you need an arbitrary container or a model outside the catalogue.

Fireworks

Hosted inference for open models with an emphasis on serving performance, alongside dedicated deployment options and support for customised weights. The positioning overlaps with Together — shared API plus dedicated capacity — with the emphasis placed on speed and efficiency of the serving path.

Best for: latency-sensitive production traffic on open models where you want someone else's optimised serving stack. Wrong choice when: the workload is batch or offline and cost per token matters more than tail latency, or you need heavy custom code beside the model.

Comparison

The cost structure, which is the whole decision

Per-token versus per-GPU-second

Per-token billing is consumption-based: you pay for exactly the work you cause, and idle time costs nothing. Per-GPU-second billing is capacity-based: you pay for the machine whether or not a request is in flight.

Consumption pricing wins when utilization is low or unpredictable. Capacity pricing wins when it is high and steady, because you then buy the underlying hardware close to cost instead of buying it with a margin attached to every token. Everything below refines that sentence.

The cold-start problem

Scale-to-zero is the headline feature of serverless GPU platforms and the source of most disappointment. With no traffic, replicas shut down and billing stops. When a request arrives, a replica must start — and for a GPU workload that means scheduling onto a machine, pulling a container image, reading weights from storage, and moving them into accelerator memory.

The last step dominates, and its cost scales with model size. A few gigabytes of weights is a manageable wait. A model in the hundreds of gigabytes is not a scale-to-zero candidate at all: those weights must cross a network or disk and then an interconnect boundary before the first token appears, and no platform cleverness makes that instantaneous. Caching, faster loading paths and snapshotting help considerably, but the scaling relationship with size does not disappear.

The standard mitigation is keep-warm: hold a minimum number of replicas running so a request always finds a loaded model. It works, and it reintroduces exactly the cost scale-to-zero removed. A "serverless" deployment with a minimum replica count above zero is a reserved GPU with extra steps, and you should budget it as one.

The rule: scale-to-zero suits small models, spiky traffic, and workloads tolerant of a slow first request. Large models serving interactive traffic need warm capacity, which is capacity pricing again.

Utilization is the whole game

Dedicated capacity beats per-token pricing above a utilization threshold and loses below it. Compute the threshold yourself, with current published rates:

monthly_api_cost = (tokens_in / 1e6) * rate_in + (tokens_out / 1e6) * rate_out

monthly_dedicated_cost = replicas * gpu_hourly_rate * hours_per_month

utilization = tokens_out / (sustained_tokens_per_sec * 3600 * hours_per_month * replicas)

Set the two costs equal and solve for the token volume where they match. Below it the shared API is cheaper; above it dedicated capacity is, provided you keep it busy — which is what the utilization line measures.

Two things wreck this in practice. First, sustained_tokens_per_sec is not a spec-sheet number: it depends on your model, quantization, context lengths and concurrency, so measure it on the hardware you would rent. Second, utilization is set by peak-shaped traffic, not average — provision for a peak four times your mean and effective utilization is a quarter of what the capacity could deliver.

Traffic shape beats traffic volume

Two services processing identical monthly token counts can have opposite correct answers.

Spiky interactive traffic — business-hours usage, bursts around events — has low average utilization however large the totals. Consumption pricing absorbs spikes for free; dedicated capacity makes you pay for the troughs.

Steady interactive traffic is where dedicated capacity wins soonest, because keep-warm replicas are actually doing work.

Batch and offline work — evaluation runs, bulk enrichment, nightly scoring — has the most favourable shape, because you control when it runs: saturate a rented GPU deliberately, run at high concurrency, tolerate queues, use interruptible capacity. Batch justifies dedicated capacity at far lower total volume than interactive traffic does.

Build versus buy

Self-hosting, on your own hardware or long-lived reserved instances, actually wins in four situations:

• Sustained high utilization. Keep accelerators genuinely busy around the clock and owning them beats any per-token rate, because you remove a margin rather than a cost.
• Data residency, air-gap or contractual constraints. If prompts cannot leave a jurisdiction or a network, the question is settled before economics enter.
• Hard latency floors. Round-trips to a provider region add time you cannot optimise away.
• Heavy customisation. Custom kernels, unusual quantization, your own draft model for speculative decoding — anything a managed platform will not expose.

It does not win for variable traffic, small teams, or standard models. The hidden cost is not hardware; it is the engineering attention to keep a serving stack healthy, patched and performant while the ecosystem moves underneath it. Price that as a fraction of an engineer's month and redo the comparison.

Selection criteria that are not price

For anyone serving prompts containing customer data, source code or anything regulated, these belong alongside cost, and they are often decided by someone other than the engineer choosing the platform. Gather the answers early.

Data handling and retention. How long are request and response payloads kept, is retention configurable down to zero, and is there a documented deletion path? Take this from the provider's published policy, not a summary.

Training use. Whether prompts and completions may be used to train or improve models, and whether that differs between free, shared and paid tiers. It frequently differs between a shared API and a dedicated deployment.

Regional and residency controls. Which regions can serve traffic, whether inference can be pinned to one, and whether that pinning covers logs, metrics and support access too.

Tenancy isolation. On a shared endpoint your requests are batched alongside other customers' on the same accelerator. Understand the isolation boundary — process, container, node, cluster — and whether dedicated deployment changes it. This is one of the strongest arguments for dedicated capacity independent of cost.

Compliance posture. Which attestations the provider holds, which services they actually cover — scope is often narrower than the marketing page suggests — and whether a data processing agreement and subprocessor list exist.

Egress and networking. Private networking, IP allowlisting, whether the endpoint is reachable without traversing the public internet, and what egress costs.

Lock-in through deployment format. Every managed platform has its own packaging format, configuration schema and SDK. The more serving logic lives there, the more expensive leaving becomes. Treat proprietary deployment definitions as debt you accepted deliberately.

Portability, deliberately preserved

Two practices keep your options open at almost no cost.

Prefer an OpenAI-compatible API surface. Nearly every provider here exposes one, as does every serious open-source serving engine. If your application speaks that dialect and the base URL, model name and key are configuration rather than code, switching providers — or failing over during an incident — is an environment-variable change. Isolate provider-specific extensions behind your own interface so the non-portable parts stay few and visible.

Keep the serving stack reproducible. Pin the engine version, model revision and quantization; keep launch configuration in version control; reconstruct the deployment from that repository alone. If you can stand the same model up on a rented GPU in an afternoon, you always have a credible alternative — and that is your leverage in a pricing conversation.

Decision checklist

Work through these in order and stop as soon as one decides the question.

1. Is the model a standard open-weight one available on shared APIs? If so, start there and measure. Do not provision GPUs to find out whether you need GPUs.
2. Do data handling, residency or tenancy requirements exclude shared endpoints? If so, the shortlist narrows to dedicated deployment or self-hosting.
3. What is the traffic shape? Spiky and interactive favours consumption pricing; steady or batch favours capacity pricing.
4. How large is the model? Hundreds of gigabytes of weights rules out meaningful scale-to-zero for interactive use. Budget keep-warm capacity.
5. Compute the break-even with current published rates and your own measured sustained throughput, not a vendor benchmark.
6. Add the engineering cost for anything self-hosted or in category 3. It is usually the largest line item and the one most often omitted.
7. Check the exit before committing: portable API surface, reproducible serving configuration.

And once more, because it is the most important caveat here: rates, instance types, free tiers and even product boundaries move constantly in this market. Every provider reprices, adds hardware and reshapes its offering on a timescale of months. Use this as the structure of the decision, then read each shortlisted provider's current published pricing before you sign anything.