Cost Comparison of GPU Cloud Providers for Agent Eval Workloads

An agent eval run has a distinctive shape: a sandbox spins up, the agent executes a task for anywhere from seconds to a few minutes, the sandbox tears down, and the whole cycle repeats dozens or hundreds of times in parallel across a test suite. That shape has almost nothing in common with the workload GPU cloud pricing was built around: a single job running continuously for days, consuming a GPU at a steady rate until training finishes. The per-hour rate on a provider's pricing page assumes something closer to the second pattern, so applying it to the first produces a number that looks precise but measures the wrong thing. Three distinct provider tiers have grown up around this market, hyperscalers such as AWS, Azure, and GCP, dedicated GPU clouds such as CoreWeave, Lambda, and RunPod, and decentralized marketplaces such as Vast.ai and io.net, and the tier a team picks changes almost every cost variable that follows from there. The rest of this piece works through those variables one at a time, starting with the one that punishes short jobs first: how each provider actually counts the clock.

Billing granularity inflates the cost of short eval runs on hyperscalers

Before egress fees, before storage, before any of the less visible line items, the billing clock itself already treats short-lived eval runs unfairly on certain providers. AWS and Azure both enforce a one-hour minimum billing period, while GCP, Lambda, RunPod, and Vast.ai bill in per-minute or per-second increments, and CoreWeave bills by the hour. That distinction sounds like an administrative detail until it's run through an actual eval scenario. A sandbox that runs for four minutes and tears down is charged as a full hour on AWS or Azure, so the effective per-compute-minute rate is a multiple of the advertised per-hour rate.

What if a team isn't running one short eval, but dozens of them in parallel, all torn down within minutes of spinning up? The one-hour billing floor doesn't just tax a single run, it multiplies across every concurrent sandbox, so the gap between billed time and used time widens with parallelism rather than staying fixed. io.net's own pricing comparison lays this out directly: Vast.ai and RunPod both bill per second, while AWS and Azure hold to the one-hour floor, and that same table is where the wide spread in H100 on-demand rates across providers first becomes visible. Billing granularity, in other words, is not a footnote to the rate card. For bursty eval workloads, it can widen the gap between the sticker price and the actual bill more than any other single factor.

The raw rate spread across provider tiers before hidden fees

Before layering in the granularity effect, it helps to see where the raw, advertised rates actually sit across the three tiers, because the spread is wide enough to matter on its own. For H100 80GB on-demand instances, io.net's guide puts AWS at $6.88 per hour on a p5.xlarge, Azure at $6.98 per hour on its ND H100 instance, and GCP at $5.62 per hour on an a3-highgpu instance. Dedicated GPU clouds sit meaningfully lower: CoreWeave at $2.99 per hour, Lambda at $2.49 per hour, and RunPod at $2.69 per hour, with marketplace and decentralized providers like Vast.ai and io.net clustering lower still. CloudZero's April 2026 comparison, updated September 2026, confirms the pattern using GetDeploying's index across a wide range of providers and 98 GPU models: the median on-demand hyperscaler H100 rate carries a substantial premium over the dedicated-cloud median for identical silicon, a gap that widens further on newer Blackwell-generation hardware.

The pattern holds below H100 as well. A100 80GB and RTX 4090 pricing follows the same tier structure, with specialized and marketplace providers undercutting hyperscalers by a wide margin, and one detail stands out for eval-specific planning: consumer-grade GPUs like the RTX 4090 are not available on any hyperscaler at all. For teams running smaller eval models that don't need a full H100's VRAM, that absence narrows the hyperscaler column of the comparison before price even enters the conversation. None of this spread should be read as a verdict yet. A wide gap between $6.98 an hour on Azure and $2.49 an hour on Lambda looks decisive on its face, but the number only describes the sticker price of one GPU-hour. Whether that gap survives contact with actual billing behavior, egress charges, and idle-time costs is a separate question, and it's the one the next section takes on directly.

The hidden fees that inflate actual bills well above the rate card

Rate cards describe compute. Actual GPU cloud bills consistently run meaningfully above the compute-only rate because of fee categories that most comparison guides omit: data egress, persistent storage, spot interruption re-run costs, and minimum commitment penalties.

Data egress charges apply when data moves out of a provider's network, and hyperscalers charge for this by the gigabyte in a way most dedicated clouds do not match. io.net's guide notes that decentralized networks like io.net itself typically charge nothing for the first terabyte moved each month, with a modest per-gigabyte charge beyond that threshold, which matters directly for eval pipelines that move large model weights or datasets in and out of storage repeatedly. Lambda takes a related approach with a zero-egress policy, a detail that never appears on a comparison of hourly compute rates but functions as a real discount for any eval pipeline that handles large volumes of data.

Storage compounds the problem in a way that's specific to bursty workloads. Persistent storage on hyperscalers bills continuously, whether or not a GPU is actively running, so a team that spins sandboxes up and down throughout the day still pays for storage during every idle gap in between. That billing shape fits sustained training jobs reasonably well, since compute and storage costs move together for the life of the job. It fits agent eval work poorly, because eval workloads spend a large share of their time in exactly the idle state that storage billing doesn't discount.

Commitment structure adds a fourth constraint. Reserved-instance pricing gets hyperscalers to their best published rates, but it requires locking into one- to three-year terms. Eval compute needs rarely hold still that long. Model development cycles shift, eval suites grow and shrink, and a team locked into a multi-year reservation loses the flexibility to scale down when a project's compute needs shrink, a tradeoff that doesn't appear next to the headline rate but shapes the total cost as much as anything on this list.

Provisioning speed, throughput efficiency, and the effective cost per eval

Cost per completed evaluation depends on two separate things: what a GPU-hour costs, and how much useful work happens inside that hour. Provisioning latency and throughput efficiency both belong in the second category, and both function as cost variables even though they're usually discussed as performance metrics.

Provisioning latency matters because every minute a GPU instance spends starting up is a minute billed with no eval progress to show for it. A provider that takes several minutes to provision an instance is burning billable time before the first eval token gets generated. RunPod's own materials describe its FlashBoot feature delivering cold starts under 200 milliseconds for serverless deployments, a difference that, multiplied across hundreds of short eval runs, changes how much of each billed hour actually goes toward useful compute.

Throughput efficiency works alongside provisioning speed to determine how far a given hour of billed compute actually goes. A provider offering higher GPU density, lower-latency networking such as InfiniBand instead of standard Ethernet, or stronger quantization support can generate more tokens per billable hour on identical hardware, which pulls its effective cost per completed eval below what the raw hourly rate would suggest. CoreWeave's InfiniBand networking is a useful illustration of how this cuts both ways: for single-GPU workloads it makes CoreWeave comparatively expensive per GPU-hour, since the networking investment isn't doing much work, but at large multi-GPU scale, where interconnect speed determines how quickly a distributed job finishes, that same networking can make CoreWeave cost-competitive despite the higher sticker price.

One more layer of billing behavior sits below the GPU provider entirely, at the sandbox or runtime level where the actual eval execution happens. Some platforms bill only for active CPU usage during a run, while others bill for the sandbox's entire lifespan, including idle stretches between tool calls. An eval workload with long think-time gaps, where the agent pauses between actions while a model call resolves, can rack up substantial idle-time charges under the second billing model even if the actual compute used is modest. This is a variable that a provider-level rate comparison misses completely, since it operates one layer below the GPU rate card, inside whatever sandbox or execution layer sits on top of the infrastructure.

Matching provider tier to eval workload shape: where each tier wins and loses

No single provider tier dominates across all agent eval workload shapes; the right choice depends on run duration, parallelism, data movement, compliance requirements, and whether the team needs GPUs at all for the eval model being used.

Hyperscalers make sense under a specific condition: when a team's data, compliance tooling, and existing infrastructure already sit inside AWS, Azure, or GCP, and when the eval workload runs long and steady enough that the one-hour billing floor doesn't inflate the bill much. That's a fit for sustained, infrequent eval jobs. It's a poor fit for the short, bursty, high-parallelism pattern that characterizes most agent eval work.

Dedicated GPU clouds fit the shape of most production eval pipelines more closely. CoreWeave, Lambda, and RunPod all offer per-minute or per-second billing, avoiding the one-hour floor entirely, and the Cloud GPU pricing comparison for 2026 puts hyperscaler median H100 pricing at 88% more expensive than the dedicated-cloud median. Availability adds to the case: Jarvislabs' comparison and RunPod's own materials both point to better access to H100 and H200 hardware on dedicated clouds without the waitlists that sometimes accompany hyperscaler capacity. For a team running a continuous eval pipeline against a model in active development, that combination of billing granularity and availability cuts total cost more than the raw hourly rate does.

Marketplace and decentralized networks such as Vast.ai and io.net are at the low end of the price spectrum, with per-second billing on top of already low rates. The tradeoff is reliability. Because capacity on these networks comes from a distributed pool of underlying hosts, performance and uptime can vary from one instance to the next, which suits cost-sensitive experimentation and early-stage R&D better than a production eval pipeline where reproducible results matter run over run. A team testing a new eval methodology or running exploratory benchmarks has different tolerance for that variability than a team validating a model before a release gate.

For teams running smaller eval models that fit within consumer-grade VRAM limits, GPUs like the RTX 4090, available on RunPod and Vast.ai but not on any hyperscaler, represent a cost-efficient option that a standard rate comparison misses entirely. Any team assuming it needs H100 or H200 capacity by default should check whether the eval model in question actually requires that much VRAM before ruling out the cheaper hardware tier.

Compliance and isolation requirements that constrain provider choice independent of price

For eval workloads touching regulated data, the provider decision often gets made before cost enters the discussion. Healthcare data governed by HIPAA, EU personal data under GDPR, or any system classified as high-risk under the EU AI Act each carry requirements that eliminate a meaningful share of the provider landscape outright.

One might argue that existing compliance frameworks already cover this ground, since GDPR, HIPAA, and SOC 2 have been around for years and apply broadly to cloud infrastructure. But those frameworks were built around human users interacting with systems under some form of oversight, not around autonomous agents that generate and execute code without a person reviewing it first. That mismatch leaves a real gap that teams running agent evals have to close at the infrastructure level themselves, since the compliance frameworks weren't written with this execution pattern in mind.

The financial stakes attached to getting this wrong are concrete rather than abstract. HIPAA penalties can run into large annual totals, and a provider unable to sign a Business Associate Agreement represents an immediate financial exposure for any team handling protected health information, not a hypothetical one to worry about later. Among dedicated GPU clouds, RunPod offers SOC 2 and HIPAA compliance through its Secure Cloud tier, running on T3 and T4 data centers, while Northflank takes a different architectural route through Bring Your Own Cloud, which keeps data inside the customer's own environment rather than a shared provider infrastructure. Both approaches solve for the same underlying requirement through different means, and which one fits depends on how a team's existing infrastructure is already organized.

Isolation is a related but distinct requirement, separate from network-level compliance certifications. The sandbox running an agent's code needs full separation from the host system and from any adjacent workloads, so a misbehaving or compromised agent can't reach beyond its own sandbox. Standard containers share a kernel with the host, which doesn't meet that isolation bar on its own. This constraint applies independently of the compliance certifications discussed above: a provider can meet them and still lack the isolation architecture agent code execution requires. A provider can carry SOC 2 or HIPAA compliance at the network level and still need a specific sandboxing architecture to satisfy the isolation requirement that agent code execution introduces.

Building a total-cost model that accounts for all the variables, not just the rate card

Everything covered so far points toward the same conclusion: a credible cost comparison for agent eval infrastructure has to model at least five variables beyond the advertised hourly rate. Those are billing granularity measured against actual run duration, egress volume and the per-gigabyte fees attached to it, storage costs accumulated during idle periods, the probability of spot interruptions and the re-run cost that follows, and how much provisioning latency wastes in billable time before an eval even starts.

The starting point for building that model is workload shape, not provider shopping. What is the average eval run's duration? How many runs execute in parallel at peak load? How much data moves per run, and whether the eval model requires H100/H200 VRAM or fits on smaller hardware? Those four questions, answered honestly, do more to narrow the provider field than any rate comparison alone, because they determine which of the cost mechanisms described in this piece actually apply at meaningful scale.

A team running long, steady eval jobs with modest parallelism and no regulated data might reasonably tolerate the one-hour billing floor on a hyperscaler, especially if its existing tooling already lives there. A team running hundreds of four-minute sandboxes in parallel against a small model is looking at a completely different cost profile, one where billing granularity, provisioning speed, and idle-time charges at the sandbox layer matter more than whichever provider posts the lowest number on a comparison chart. The rate card is the beginning of that analysis. It was never built to be the end of it.