Nov 4, 2026 - 8 min read

RL training for an LLM agent needs two kinds of compute. GPUs generate the agent’s actions and update its weights, and a fleet of isolated CPU sandboxes runs those actions so they can be scored. On Nodus, the GPUs come from a TrainingJob or a Job, and the sandboxes are Nodus Sandboxes that start with no network access and stop at a dollar cap. In our worked example below, the sandboxes are about a tenth of the bill; the expensive part is GPUs waiting while tool calls finish.

That last point decides most design choices, so this post walks through the parts of an agent RL setup, what each one costs, and when the managed path is enough.

What are the moving parts of an agent RL run?

An agent RL run is a loop of three jobs: generate, act, and learn. The policy model writes a reply or a tool call. An environment executes it, returns the result, and the episode continues for several turns. At the end, a grader turns the episode into a reward, and the trainer updates the model.

GRPO is a common choice for agent RL. It samples a group of episodes per task and rewards the ones that beat their group’s average. It needs no critic model, which keeps GPU memory down. The DeepSeekMath paper introduced it, and our post on what GRPO is and how to run it covers the method itself.

The three jobs want different hardware:

Part	What it does	Hardware it needs
Policy and trainer	Samples actions, computes the loss, updates weights	GPUs sized to the model and method
Environment	Runs code, shell commands or tool calls for each episode	Isolated CPU containers, one per live episode
Grader	Scores the finished episode, holds the answers	A separate process the model cannot read

The environment is the part that is new compared with ordinary RL on text. A coding agent needs a real filesystem, a shell and a test runner for every episode in flight. That is why agent RL is as much a sandbox problem as a GPU problem.

How much GPU does the trainer need?

The trainer’s GPU follows from the model and method, the same as any fine-tune. With a LoRA adapter on a small model, one GPU is enough. The nodus/grpo-lora runtime’s presets put Qwen3-8B with QLoRA on one L40S, RTX A6000 or A100 80GB, and the L40S is listed at $0.793/hr on the Nodus pricing page.

Full-parameter training is another matter. In our PPO vs DPO vs GRPO comparison we worked out that full GRPO on an 8B model needs about 131 GB at 16 bytes per parameter. That means several 80 GB GPUs before you count activations.

The group size also multiplies generation work. The Nodus rl.train recipe samples 8 replies for each of 4 tasks per step, capped at 256 tokens each. That is up to 8,192 generated tokens per step for single-turn tasks. A multi-turn agent episode repeats that generation on every turn, so plan generation capacity per turn, not per task.

How many sandboxes does agent RL need, and what do they cost?

Plan on one sandbox per episode that is running at the same moment. With 16 tasks per step and a group size of 8, that is 128 episodes in flight, so 128 sandboxes. Fewer sandboxes means episodes queue, and the GPUs wait for them.

On Nodus, a 1 vCPU, 2 GiB Sandbox is listed at $0.0179/hr on the pricing page. Keeping 128 of them running costs 128 x $0.0179 = $2.29 per hour. Sandboxes run on CPU machines Nodus operates, not on the GPU machines used for your jobs, so they never take GPU memory from the trainer.

Each one is isolated in ways that matter for code a model wrote. The Sandbox isolation page lists the layers: a user-space kernel that filters system calls, a default-deny firewall with cloud metadata addresses always blocked, a 256-process limit and a non-root user. The Ephemeral continuity mode starts every run empty, which fits a fresh sandbox per task.

Why does GPU wait time matter more than sandbox price?

Because the GPUs bill for every minute a step takes, including the minutes they spend waiting on tool calls. The worked example below shows the size of the effect.

Take an illustrative run on one 8 x H100 node with 128 sandboxes at 1 vCPU and 2 GiB. It trains for 300 steps at 3 minutes of wall-clock time per step, most of it spent waiting on tests and shell commands. These inputs are our assumptions for illustration, not a benchmark. The rates are from the Nodus pricing page, where the H100 SXM is listed at $2.600/hr. Prices checked November 4, 2026.

The run lasts 300 x 3 minutes = 15 hours. The GPUs cost 8 x $2.60 x 15 = $312.00. The sandboxes cost 128 x $0.0179 x 15 = $34.37. The total is $346.37, and the sandboxes are 9.9% of it.

In this example the sandboxes are under a tenth of the bill; the GPUs are the rest.

Now cut the step time in half, to 1.5 minutes, by running tool calls in parallel or overlapping generation with execution. The run drops to 7.5 hours. GPUs cost $156.00 and sandboxes $17.18, for $173.18 in total.

Halving the step time saves $156.00 of GPU time, more than four times the entire $34.37 sandbox bill.

From this arithmetic we get a decision rule. Never starve rollouts of sandboxes to save money, because a sandbox costs cents per hour while every idle minute costs the whole GPU node. Spend your engineering effort on step time: parallel episodes, fast environment resets and generation that does not sit idle during tool calls.

When is the managed Nodus runtime enough?

The managed path fits when your environment’s tools are plain Python that runs offline. Nodus Environments are versioned task sets with a grader, and the Environments guide shows nodus/grpo-lora training on multi-turn games and tool-calling environments. The tools run inside the grader, and the trainer never sees the expected answers.

A tool-calling run is a short manifest, copied from that guide:

spec:
  runtime: nodus/grpo-lora
  environment: {name: [email protected], trainTasks: 21, heldOutTasks: 43, seed: 42}
  parameters:
    maxTurns: 3
    maxCompletionLength: 256

Each episode can call tools for up to maxTurns rounds, and the reward covers the whole episode. Program graders run in grading Sandboxes with no network access, billed to the TrainingJob and counted against its maxCostUSD. Before you launch, the recipe’s preview() returns a cost estimate from the runtime’s published per-step times.

Grading results are also built to keep training honest. A grader that crashes returns InfrastructureFailure, which carries no reward and does not count toward the pass rate. An outage never trains the model to believe a correct action was wrong.

When do you need your own trainer and sandboxes?

You need your own trainer when tools need the network, a browser, an MCP server or another model as judge. The managed grader has no network, so the Environments guide says those environments fail the build. For them, run an open-source trainer such as verl or SkyRL on GPUs, and give each episode a Nodus Sandbox through the Python SDK.

This sketch shows the sandbox side of one episode, using calls from the Python Sandboxes guide. The policy and score functions are placeholders for your own generation and reward code:

import nodus

def run_episode(task, policy, score, max_turns=6):
    sb = nodus.Sandbox.create(cpu=1, memory="2Gi", idle_timeout="5m", max_cost=1)
    try:
        with sb.open("/workspace/task.json", "w") as f:
            f.write(task.setup_json)
        history = [task.prompt]
        for _ in range(max_turns):
            command = policy(history)            # your model proposes a shell command
            if command is None:
                break
            p = sb.exec("bash", "-c", command)
            history.append(p.stdout.read())
            p.wait()
        return score(sb, task)                   # e.g. run the hidden tests
    finally:
        sb.terminate()

The egress default is deny, so the agent cannot reach the internet unless you open it with network=nodus.Egress.open(). Keep it closed unless a tool needs it, because code in a sandbox can read every secret you put there.

Where does this setup fall short?

Be clear about the limits before you plan around them. Nodus has no managed PPO runtime, so a learned reward model with a value head means running your own trainer. Allow-list egress, where only named hosts are reachable, is available in Sandbox manifests but not yet in the Python SDK. Our worked example uses assumed step times; your environment’s real reset and tool latency decide the numbers, so time a few episodes before you size a run.

Training recipes, TrainingJobs and GPU Sandbox state snapshots are Beta; Environments are generally available.

FAQ

Can agent RL run on a single GPU?

Yes, for a LoRA adapter on a model up to about 8B parameters. The smallest nodus/grpo-lora presets, Qwen3-0.6B and Qwen3-1.7B with LoRA, fit on one RTX 4090, listed at $0.290/hr on the pricing page. That is a cheap place to debug an environment before you scale up.

Should rollouts run on GPU sandboxes?

Only if the agent’s tools need a GPU, such as an agent that writes and benchmarks CUDA kernels. Shell and test-running tools work fine on CPU sandboxes, which cost cents per hour. Nodus GPU Sandboxes run on dedicated on-demand machines and are never preempted; our GPU sandboxes post compares the options.

How do you stop a reward hack from reading the answers?

Keep the expected answers outside the model’s reach. In Nodus grading, code from a completion runs as a separate user that cannot read the answers, and the trainer only sees prompts and metadata. With your own trainer, run hidden tests from a location the agent’s sandbox never mounts.

Can I train on environments built for verifiers, OpenEnv or verl?

Many of them, yes. The Environments guide covers verifiers packages, OpenEnv environments and verl datasets with their reward functions unchanged, as long as grading needs no network.

Tags:

• agents
• reinforcement-learning
• sandboxes
• grpo

How do you fine-tune Mistral 7B on your own data?

How do you measure latency for streaming LLM responses?