When to Use Kubernetes for AI Workloads vs a Purpose-Built Agent Runtime

Kubernetes has become the default platform for AI workloads because it already solves problems engineers were going to hit anyway with ML pipelines and model serving. Teams do not choose Kubernetes for AI because it was purpose-built for transformers or GPU clusters. They choose it because the platform underneath their existing services already does three things well, and extending that platform costs less than building a parallel one.

The first is packaging. Anyone who has debugged a model that behaves differently in staging than it did on a laptop knows how much engineering time goes toward closing that gap, and containers close most of it by construction.

The second is the extension model. DRA, which governs how GPUs and other accelerators get claimed and scheduled, reached general availability in Kubernetes v1.34. That a resource-allocation mechanism this central to AI workloads could mature inside the existing API server, rather than requiring a forked control plane, says something about how much headroom the extension model has.

The third is governance that already exists. Platform and security teams govern infrastructure through tooling they have already deployed: admission controllers, OPA or Kyverno policy engines, network policies, and API server audit logging. None of it requires custom integration to see a new AI workload land in the cluster. For many teams, Kubernetes is the infrastructure that already exists, along with the expertise and tooling built around it, and an organization rarely abandons that kind of sunk investment without a structural reason.

What Kubernetes was designed to run

Kubernetes was designed to run software with a predictable shape: a process starts, serves requests, and terminates or restarts under failure. Agentic workloads break that shape in ways that are structural, following directly from what an agent requires and what Kubernetes was built to manage. The honest answer sits across five dimensions.

Compute differs first. Web services use CPU in fairly predictable amounts, while agent workloads are often GPU or accelerator-driven, arriving in bursts tied to inference calls. A web service holds a handful of scoped secrets, but an agent holds API keys for model providers plus whatever tools and MCP servers it has been granted access to. And access control follows the same pattern: a web service has fixed, narrow permissions, but an agent that can act on infrastructure needs infrastructure-shaped permissions. Its blast radius looks nothing like a typical microservice's.

Kubernetes' own SIG Apps has acknowledged this gap directly. At the scale most agent platforms are now targeting, it becomes an operational burden that grows roughly linearly with the number of agents running concurrently.

Cold-start latency makes the mismatch concrete. That same second breaks the continuity of an agent interaction the moment it is invoked after sitting idle, because the orchestrating system expects a ready environment, not a one-second pause it has to explain to a waiting user or a waiting chain of tool calls.

The workloads where Kubernetes is still the right answer

None of this means Kubernetes is the wrong infrastructure for AI in general. It earns its role squarely in three workload patterns, and engineers running these should not be talked out of it.

GPU-heavy distributed model serving and training is the clearest case. KubeRay paired with vLLM is the dominant pattern for LLM inference, and KubeFlow's Training Operator handles distributed PyTorchJob, TFJob, XGBoostJob, MPIJob, PaddleJob, and JAXJob workloads at real scale. Kueue handles job queueing for batch and ML workloads, while Volcano and DRA round out the GPU scheduling ecosystem. Topology-aware placement and GPU bin-packing, mixing serving and training on the same cluster with intelligent resource sharing, are precisely the problems the Kubernetes scheduler and its extensions were built to solve, and nothing in the agentic-workload argument above changes that.

Multi-team platform governance is the second case.

The third case is organizational: teams with mature Kubernetes expertise and GitOps pipelines can extend that investment to agent workloads instead of standing up an entirely separate operational surface. The community's own response to the abstraction gap, the Agent Sandbox project from SIG Apps, has reached a stable v1beta1 API as a declarative Sandbox CRD built entirely on Kubernetes primitives. For teams already committed to the Kubernetes ecosystem, this is the direction the community is actively building toward, even while parts of it are still in development.

Where Kubernetes Primitives Break Down

Two properties predict, more reliably than anything else, whether a given workload will strain against Kubernetes: whether it must persist state across sessions, and whether it must resume in under a second. Agent execution tends to fail both tests at once.

Agents operate as stateful workspaces. They hold conversational context, partially completed tool calls, and execution history that needs to survive crashes, timeouts, and redeploys without losing the thread of what the agent was doing. Kubernetes has no native primitive built for exactly this pattern. Teams solve this today by reaching for workflow engines layered on top: Temporal treats agent steps as persistent, fault-tolerant durable executions so tasks running for hours or days survive infrastructure events, and LangGraph is built around durable execution with human-in-the-loop control, memory, streaming, and failure recovery.

Provisioning latency is the second failure mode. Agent Sandbox's answer is SandboxWarmPool: pre-provisioned Sandbox pods that can be claimed immediately, which effectively eliminates cold starts inside the cluster. It works, but you have to maintain a pool of idle resources sitting around waiting to be claimed, and that pool carries its own cost and its own operational overhead to keep sized correctly.

A deeper, structural issue produces both symptoms. None of this makes Kubernetes incapable of running agents. It makes the cost of running them well, reliably, and at scale considerably higher than the cost of running a stateless web service, and that cost is the whole argument for looking at purpose-built alternatives.

Why untrusted, AI-generated code needs a different isolation model than containers provide

Standard containers share the host kernel, and that shared kernel is precisely the boundary that matters when the code running inside the container was written by a model. A container isolates processes well enough for most multi-tenant SaaS workloads. But it was never designed to hold the line against code whose intent nobody on the team has reviewed.

The practical isolation hierarchy makes the stakes legible. Firecracker microVMs move to hardware-enforced isolation, suited for serverless and untrusted code execution, with a boot time around 125 milliseconds. Kata Containers also deliver hardware-enforced isolation through a VMM, and they suit regulated industries and zero-trust environments, with boot times between roughly 150 and 300 milliseconds depending on configuration. Kubernetes can reach the stronger end of this hierarchy: the Agent Sandbox CRD natively supports both gVisor and Kata Containers, giving it the kernel and network isolation that multi-tenant, untrusted execution actually requires.

The real gap is what surrounds the isolation technology that gets used underneath. A purpose-built runtime wraps the isolation layer in provisioning, lifecycle management, and networking designed around the agent pattern from the start, rather than assembled after the fact from general-purpose primitives that were built for something else. Agent Sandbox addresses this with isolated environments offering graphical desktop capabilities, including VNC, browsers, and IDEs, and an AIO Sandbox preinstalled with VNC, VSCode, Jupyter, and a terminal. Isolation decisions made at this layer do not stay confined to engineering.

How enterprise compliance requirements shape the infrastructure choice

Compliance frameworks were written with human actors in mind, so when an agent becomes the entity taking action, they break in structurally predictable ways. GDPR assumes a contracted data processor. HIPAA assumes an identifiable workforce member. SOC 2 relies on access controls tied to a specific user. None of those assumptions hold cleanly when the acting party is a model executing tool calls on someone else's behalf. Shared accounts between agents break audit trails and make it impossible to attribute a given action to a given actor, so each AI agent needs its own identity if traceability and controlled access are going to mean anything in practice.

HIPAA carries direct financial exposure, with penalties substantial enough that choosing an inference environment that is actually HIPAA-eligible becomes a concrete business requirement. The EU AI Act adds a further layer of categorization on top of that. Agents used in HR and recruitment, credit scoring, healthcare diagnosis, critical infrastructure, education, or law enforcement land in the high-risk category, which triggers conformity assessment, post-market monitoring, and human oversight requirements that have to show up concretely in the runtime's audit and logging capabilities.

Kubernetes gives platform teams real governance primitives, admission controllers, OPA or Kyverno, network policies, API server audit logging, but addressing the agent-identity problem with them requires deliberate configuration. Daytona is built to meet this bar directly: SOC 2, HIPAA, and GDPR compliance are built into the platform rather than sold as premium add-ons layered on top later. For organizations with strict data-residency requirements that favor self-hosting, running agents inside their own Kubernetes cluster keeps data inside their own network, though teams handling untrusted, AI-generated code may still want a dedicated secure runtime like Daytona layered in for isolated agent execution kept separate from production infrastructure.

A concrete decision framework: which workload pattern goes where

The infrastructure choice follows from four workload properties, and if you map a given workload across these four dimensions, it reliably predicts which pattern fits.

The first property is execution lifetime. Long-running sessions that must survive crashes, resume from checkpoints, and hold context across hours or days push in the other direction: Kubernetes requires significant glue, Temporal, LangGraph, custom StatefulSet wiring, while a purpose-built agent runtime treats long-running state as a first-class primitive.

The second property is code trust level. AI-generated code executing at runtime, code the platform did not write and cannot fully audit beforehand, needs strong hardware-enforced isolation, and the runtime underneath it has to be built to provision and manage that isolation at scale.

The third property is provisioning latency tolerance. Daytona provisions sandboxes in under 90 milliseconds, and that figure functions as a hard engineering requirement for agent responsiveness at scale, not a benchmark to chase down later.

The fourth property is organizational context. Teams that need to move from prototype to production without building and operating Kubernetes agent infrastructure, or that need SOC 2, HIPAA, and GDPR compliance without hand-rolling custom policy configuration, are better served by a purpose-built agent runtime that removes that operational surface.

What a purpose-built agent runtime provides beyond Kubernetes

A purpose-built agent runtime is not a stripped-down Kubernetes. It represents a different design point, optimized for the exact properties that Kubernetes can only approximate by assembling several separate tools.

Sub-second provisioning is built in. The design assumes agents will be invoked on demand and must be ready before the orchestrating system times out waiting for them. At scale, the gap between 90 milliseconds and one second stops being a rounding error: it governs whether an agent platform can serve many concurrent users smoothly or has to queue and stagger invocations to keep up.

Statefulness is handled as a first-class primitive rather than something reconstructed by an external workflow engine. Daytona sandboxes are stateful by design and can be configured to run indefinitely, since auto-stop defaults to 15 minutes of inactivity and has to be explicitly disabled for longer-running work. Agents pick up long-running work exactly where they left off, without needing Temporal or LangGraph to reconstruct state from a checkpoint, eliminating the workflow-engine-as-crutch pattern that Kubernetes-based agent deployments typically need for reliable long-running execution.

Isolation for untrusted code is designed in. Daytona's customers, including LangChain, Turing, Writer, SambaNova, and Prosus, are LLM providers, agent builders, and enterprises that need to execute code they did not write by hand. So that is the use-case profile where the Kubernetes framework laid out above runs into its structural limits.

Applying the framework: four workload patterns and matching infrastructure

The decision framework earns its keep when applied to the patterns engineers actually run into, and four cover most of the cases where the choice is genuinely in question.

KServe with canary rollouts and autoscaling, or KubeRay for vLLM-backed serving, handles this well, and GPU time-slicing and MIG partitioning are already solved problems inside the Kubernetes ecosystem.

Autonomous agents with tool use and code execution look very different. Daytona fits this pattern directly, with sub-90ms provisioning, stateful sandboxes, hardware-enforced isolation, SOC 2, HIPAA, and GDPR compliance, and Docker-native compatibility.

Computer-use agents operating a graphical environment, controlling a browser or desktop, clicking, navigating, interacting with UI elements, carry a risk that a compromised environment could reach sensitive internal tools or exfiltrate data through the UI itself. Isolation is the variable that matters most here. Kubernetes Agent Sandbox offers VNC and browser environments with gVisor or Kata Containers isolation, while purpose-built runtimes with microVM-level isolation provide the strongest guarantee for this specific pattern. Which one fits depends largely on whether the team is already operating on Kubernetes at the platform level; for teams that are not, the overhead of standing up the Agent Sandbox stack just for computer-use workloads tips the decision toward a purpose-built runtime instead.

The ecosystem behind all of this keeps moving. DRA reaching GA, Agent Sandbox reaching a stable API, and CNCF's growing attention to AI workloads all suggest the abstractions available to engineers will keep improving on both sides of this framework. What will not change nearly as fast are the underlying properties of the workload itself, execution lifetime, code trust, latency tolerance, and organizational context, and those four properties are what should be driving the infrastructure decision regardless of which tools exist to implement it next year.