Parallel Agent Evaluation at Scale With Ray and SLURM
Distributed agent evaluation requires dynamic scheduling, not fixed batch allocation.
Agent evaluation no longer fits on one machine because the arithmetic demands it. The thing being scored is a full execution trajectory, not a single reply: every tool call, every argument passed, every error the agent catches or misses, every step toward (or away from) the final outcome. A response can read as correct while the path that produced it was broken the entire way through, so an evaluator that only checks the final answer is measuring the wrong thing. That compounding matters because end-to-end success falls off fast as trajectories get longer. A per-step accuracy rate that sounds fine on paper can turn into a sharply lower end-to-end pass rate once you chain enough steps together, and the only way to get a trustworthy read on that curve is to run a lot of trajectories in parallel rather than one after another. Benchmarks built for this moment make the point concrete. OSWorld 2.0 covers 108 end-to-end desktop workflows with a median human completion time of around 1.6 hours each, and SWE-Bench carries similar weight in software tasks. Running either suite one task at a time turns the eval from a pipeline into a multi-day wait.
How Ray and SLURM divide the scheduling problem differently
Ray and SLURM solve parallelism, but they start from opposite assumptions about when compute gets claimed and how work gets shaped. Ray treats the cluster as a dynamic task graph: tasks and actors get dispatched the moment work shows up, not declared ahead of time as a fixed batch, and the scheduler treats CPUs and GPUs as one shared pool it can mix on the fly. That design lets Ray push millions of tasks per second at sub-millisecond scheduling latency, roughly an order of magnitude faster than another engine built around fixed-batch partitioning on AI-shaped workloads, because the scheduler is built for fine-grained, heterogeneous dispatch. Two primitives carry most of that weight: stateless remote Tasks, which Ray spreads across workers automatically, and stateful Actors, named remote objects that hold state across method calls and can restart from a checkpoint stored in Ray's object store. Those two primitives are what make Ray a natural fit for agent work, where state and timing both shift mid-run.
SLURM runs on a fundamentally different contract. A SLURM job allocation starts all its nodes at once or none of them start at all; the scheduler holds resources back until the whole request can be satisfied, then hands them over as a single unit. (SLURM does have a feature called gang scheduling, but that term refers specifically to time-slicing oversubscribed jobs, not to this all-or-nothing allocation behavior.) If a job can't tolerate a partial start, you need that all-or-nothing contract, and that's true for distributed training as well as for large homogeneous batch inference, where you want guaranteed ownership of a fixed set of resources for a fixed duration. Neither system is faster than the other in some abstract sense; the real question is which scheduling assumption matches the shape of the workload sitting in front of it, and the next two sections work through that match in both directions.
Agent Rollouts and Batch Job Scheduling
Agent rollouts break the assumptions that make gang scheduling efficient. Their completion times swing widely from task to task, their resource needs mix CPU and GPU work unevenly, and when something fails, it's usually the environment around the agent, not the compute node itself. Research out of Tongyi Lab, Alibaba Group (AgentJet) lays out the structural problem directly: agentic reinforcement learning trajectories run orders of magnitude longer than single-turn reasoning chains, touch heterogeneous external environments like code sandboxes, browsers, and APIs, and show high variance in how long they take to finish. General-purpose batch frameworks were never built around those three properties at once.
The variance gets worse as a trajectory runs longer, not better. Research on OSWorld tasks finds that later steps in an agent's run can take roughly three times as long as early steps, so if you run a 200-task suite sequentially, batch resources sit wildly underused for stretches and overloaded for others. The pattern that resolves this splits the work by resource type rather than by job: VM orchestration, which is CPU-bound and handles the environment feedback loop, runs on one class of worker, while model inference, which is GPU-bound, runs on another, with Ray's actor graph coordinating the two. A single SLURM job allocation has no natural way to express that split. Training agents on OSWorld environments with GRPO shows the pattern concretely: VMs get launched as Ray remote tasks, and the heterogeneous resource mix forces the CPU-bound feedback loop to be disaggregated from GPU-bound generation. The practitioner pattern that came out of Ray Summit 2026, decoupling orchestration from execution with native autoscaling and live traffic management, exists for this exact reason: agent workloads arrive in bursts and finish unpredictably, so locking in a fixed allocation at submission time wastes capacity when rollouts finish early and starves the pipeline when they run long. AgentJet's detachable swarm client design takes this further: client nodes can hot-swap mid-run, and each task gets an isolated, detachable runtime, so one environment failure doesn't take down the whole training job.
SLURM's Batch Contract for Large GPU Eval Suites
Flip the workload shape and SLURM's rigidity becomes what you want. Large, homogeneous GPU evaluation suites, overnight batch scoring runs, embedding generation jobs, fixed-suite benchmark sweeps, all benefit from the same gang scheduling and on-premises posture that looked restrictive against agent rollouts. When a job is intolerant of a partial start, allocating every node at once and holding resource ownership for the job's full duration isn't a constraint to design around, it's the feature that makes the run predictable.
Batch inference deserves more credit as a SLURM use case than it usually gets. Overnight embedding generation, batch LLM evaluation, and dataset scoring pipelines that run on a known, fixed schedule gain nothing from Ray's dynamic dispatch model, because there's no burstiness to absorb. What they need is predictable queuing and guaranteed node ownership, and that's what SLURM gives them. There's a compliance dimension too: data that lives on a SLURM cluster running on-premises HPC infrastructure never crosses the organization's network boundary, which makes GDPR data-residency compliance considerably simpler than it tends to be on managed cloud runtimes. And the boundary between the two systems isn't as fixed as the philosophical split might suggest. RASER (Resilient Agent Scheduling and Execution Runtime for HPC Clusters, submitted to arXiv in September 2026 as 2609.03598) adapts SLURM for agent workloads using job arrays combined with shared-filesystem work stealing, and reports close to 39% faster makespan on SWE-bench tasks as a result. That result matters because it shows SLURM can absorb some of the agent-shaped irregularity that this piece just spent a section arguing against, provided the job design accounts for it.
Failure recovery and statefulness: how each system handles interrupted evaluation
Agent evaluation sessions that run for an hour or more can't be treated as disposable, and both systems have an answer for interruption, just at different layers of the stack. Ray's Actor model keeps state alive across distributed nodes: a named Actor holds its state between method calls, and if it fails, it can restart with that state recovered from Ray's object store. That recovery happens at the level of the task graph itself, not at the level of the whole job, so a single failed actor doesn't necessarily take the rest of the run down with it.
SLURM's answer sits one layer up. When a node fails, SLURM requeues the job, and the agent picks back up from a checkpoint written to the filesystem rather than starting over from the beginning, a property that matters enormously for coding evaluation tasks that run for hours at a stretch. RASER implements this pattern directly, pairing SLURM job arrays with shared-filesystem work stealing so that a requeue restarts the job and the agent resumes from its last checkpoint. The underlying need here isn't that long sessions are inherently fragile. Agents require persistent state to hold onto execution history, tool context, and intermediate results across a session, and the real risk comes from that state being uncontrolled, not from the session's length by itself. OSWorld 2.0's workflows, with median human completion times around 1.6 hours, make the stakes obvious: a failure at step 90 of a 100-step workflow that forces a restart from scratch burns most of the compute budget that went into the run, so how finely a system checkpoints determines how expensive that kind of failure actually is.
Isolation requirements for executing untrusted agent code at eval time
Execution isolation is the other question: what happens when the agent's own generated code runs is not a matter of Ray or SLURM specifically. Shared-kernel containers built on Docker or runc widen the blast radius of any failure, because a compromised container shares its host kernel with every other workload running on that node. That's a fine tradeoff for code the team already trusts. Agent-produced code has to be treated as potentially hostile until proven otherwise, so it's not adequate for code an LLM wrote on the fly.
Three tiers of isolation have become the standard response to that risk. MicroVM runtimes, Firecracker and Kata Containers among them, give each workload full hardware virtualization with its own dedicated kernel (Firecracker itself is the microVM virtual machine monitor, while Kata Containers is an OCI runtime that orchestrates microVMs through backends like Firecracker, QEMU, or Cloud Hypervisor). That tier gives you the strongest isolation available, at the highest resource cost. User-space kernels like gVisor sit a step down, intercepting system calls before they ever reach the host kernel, trading some isolation strength for better performance. Hardened containers, Docker paired with seccomp and AppArmor policies, use namespace isolation and kernel-level restrictions, and you should reach for that level only when the code being run is already trusted. The threats these tiers defend against are concrete: prompt injection attacks that craft evaluation inputs to manipulate agent behavior, and code generation exploits where an agent produces code carrying malicious logic of its own. Both call for sandboxed execution with no network access and minimal system privileges by default. None of this isolation requirement changes based on which scheduler is running the job. The scheduling layer decides how work gets spread across the cluster; the execution layer decides whether that work can break out of the boundary it's supposed to stay inside. Fast-provisioning, stateful sandboxes built with this kind of isolation in mind are what let either scheduling architecture run agent code safely at scale, and retrofitting ordinary developer containers onto an eval pipeline leaves that gap wide open.
The hybrid architecture: running Ray and SLURM together for production eval pipelines
The most capable production eval pipelines tend not to pick one system and walk away from the other. They give Ray the layer it's built for, give SLURM the layer it's built for, and bridge the two with an abstraction that lets each do its job without forcing a rewrite every time a workload moves between them. That convergence layer is already showing up in practice: a 2025 arXiv preprint (2511.21413) demonstrated automated, dynamic AI inference scaling by wiring together Kubernetes, SLURM, and vLLM, suggesting the industry is already building toward composed systems. Apple's engineering team made a related case at QCon AI Boston 2026, presenting framework-agnostic agent infrastructure that uses Ray as a reference implementation for a deployment layer that stays agnostic to the specific agent framework running on top of it, bridging local development work and production-grade reliability in one design.
Ray Summit 2026 showed what this looks like at the upper end of scale: NVIDIA's Nemotron team coordinated rollout engines, environment workers, and policy training across thousands of GPUs at once on Ray Core, training Nemotron 3 Ultra, a mixture-of-experts model with hundreds of billions of total parameters and tens of billions active at any point. The practical split that falls out of all this gives Ray the dynamic agent rollout orchestration, the heterogeneous resource mixing, and the real-time autoscaling, while SLURM takes the batch scoring jobs, the large GPU array allocations, and the workloads bound by on-premises data-residency rules, with a bridging layer letting each system handle the part it was actually designed for. None of this makes the hybrid mandatory. A team running only homogeneous batch evaluation has no real need for Ray, and a team running only dynamic agent rollouts has no real need for SLURM. You earn the hybrid's complexity specifically when both workload shapes exist side by side in your organization, not when you reach for it as a default regardless of what's actually running.
Choosing the right architecture for your eval workload shape
The right scheduling architecture for agent evaluation comes down to three properties of the workload itself: how heterogeneous the tasks are, how much completion times vary, and what compliance constraints apply, rather than to which system an engineering team happens to know best. Ray tends to be the better fit when eval tasks mix environments, browser, desktop, code sandbox, API, and the time each task takes varies widely from one to the next, especially when the eval loop blends CPU-bound environment feedback with GPU-bound model inference in a way that needs to run as a single coordinated actor graph. Autoscaling carries real weight in that setting too, since fixed batch allocations waste capacity when workloads run quiet and fall short when they spike, and long-running agent sessions generally need the kind of stateful Actor checkpointing Ray provides.
SLURM tends to be the better fit when eval tasks are homogeneous, same model, same environment type, a fixed compute budget per task, and can be declared in full at submission time. It's also the stronger choice when data-residency rules or GDPR compliance require that data never leave the organization's network perimeter, when workloads run on a predictable overnight schedule where gang scheduling's guaranteed resource ownership works in the team's favor rather than against it, and when the team already runs on HPC infrastructure where the compliance posture of managed cloud would be a step backward. A hybrid approach makes sense specifically when both workload shapes coexist inside the same organization, dynamic agent rollout evaluation running alongside large batch scoring or embedding generation, composing Ray for the former and SLURM for the latter through a bridging layer, along the lines the RASER project has shown is achievable without rewriting code. Whichever architecture a team lands on, the execution environment sitting beneath the scheduler still has to satisfy isolation requirements that neither Ray nor SLURM supplies on its own: fast-provisioning, stateful sandboxes with strong isolation, at the microVM or user-space kernel level, are what make parallel agent evaluation safe to run at production scale, and that calls for infrastructure purpose-built for agent code execution rather than developer containers pressed into service after the fact.
