Oct 26, 2026 - 10 min read

GRPO (Group Relative Policy Optimization) is a reinforcement learning method that scores several answers to the same prompt and moves the model toward the ones that beat their group’s average, so it needs no critic model. Nodus runs it as a managed runtime, nodus/grpo-lora, which trains a LoRA adapter, by default on one GPU, grades every answer on the platform and caps the run at a dollar limit you set. With QLoRA, an 8B model needs only about 4.5 GB for weights and trainable state, and the Nodus preset runs it on a single GPU with 48 GB or more.

Where does GRPO come from?

GRPO was introduced in the DeepSeekMath paper as “a variant of Proximal Policy Optimization (PPO)” that also cuts PPO’s memory use. In that paper it lifted DeepSeekMath-Instruct 7B from 82.9% to 88.2% on GSM8K and from 46.8% to 51.7% on MATH.

It became widely known a year later. The DeepSeek-R1 paper names GRPO as the reinforcement learning algorithm used to train both DeepSeek-R1-Zero and DeepSeek-R1, with rewards based on whether final answers were correct.

How does GRPO training work?

GRPO replaces a learned value model with a simple comparison inside a group of answers. Each training step does four things.

1. Sample a group. For each prompt, the current model generates G completions. DeepSeekMath used 64 per question; Hugging Face TRL’s GRPOTrainer defaults to 8, and the Nodus runtime defaults to 4.
2. Score each completion with a reward function or reward model. For verifiable tasks this is ordinary code: does the final answer match, do the unit tests pass, is the format right.
3. Turn scores into advantages. Each reward has the group mean subtracted and is divided by the group standard deviation. Every token in that completion gets the same advantage.
4. Update the policy with PPO’s clipped objective, optionally adding a KL divergence penalty that keeps the model close to a frozen reference copy.

A small example makes step 3 concrete. Say a math prompt gets 8 completions and 3 are correct, so the rewards are 1, 0, 0, 1, 1, 0, 0, 0.

	Value
Group mean	3 / 8 = 0.375
Group standard deviation (population)	√(0.375 × 0.625) = 0.484
Advantage of each correct answer	(1 − 0.375) / 0.484 = +1.29
Advantage of each wrong answer	(0 − 0.375) / 0.484 = −0.77

Libraries that use the sample standard deviation get slightly smaller numbers (0.518, so +1.21 and −0.72). The direction is what matters: correct answers are reinforced, wrong ones discouraged, and the size depends on how unusual each answer was within its own group.

The edge case is the important one. If all 8 answers are right, or all 8 are wrong, every advantage is zero and the step teaches nothing. That is why GRPO needs a base model that already solves the task some of the time.

What is the advantage of GRPO over PPO?

GRPO’s main advantage is that it drops PPO’s value model, which saves roughly a model’s worth of memory and compute. The DeepSeekMath authors note that PPO’s value function “is typically another model of comparable size as the policy model,” and it is hard to train well when only the last token gets a reward. GRPO uses the group average as the baseline instead.

Two smaller differences follow. GRPO adds the KL penalty directly to the loss rather than folding it into the reward, which keeps the advantage calculation clean. The KL term is now often switched off. TRL defaults beta to 0.0, and at 0 “the reference model is not loaded, reducing memory usage and improving training speed.” DeepSeekMath used a KL coefficient of 0.04, and the TRL docs cite 0.001 for DeepSeek-R1.

How much GPU memory does GRPO need for an 8B model?

For Qwen3-8B, GRPO with LoRA needs about 16.8 GB for weights and trainable state, and QLoRA about 4.5 GB, before activations and the generation cache. We worked them out from the Qwen3-8B model card and config: 8.2B parameters, 36 layers, hidden size 4,096, 8 KV heads of dimension 128. For trained parameters we use the ZeRO paper’s 16 bytes each for mixed-precision Adam.

• Frozen 16-bit base: 8.2B × 2 bytes = 16.4 GB.
• Frozen 4-bit (NF4) base for QLoRA: 8.2B × 0.5 bytes = 4.1 GB, plus small quantization constants.
• Rank-8 LoRA on every linear layer: per layer, rank × (input + output) summed over q, k, v, o, gate, up and down projections is 606,208 parameters. Times 36 layers that is 21.8M parameters, and at 16 bytes each, 0.35 GB.
• Full fine-tuning of the policy: 8.2B × 16 bytes = 131.2 GB. A 16-bit reference model for the KL term adds 16.4 GB, for 147.6 GB.
• PPO-style full training, with a trained value model the same size as the policy plus frozen reference and reward models: 131.2 × 2 + 16.4 × 2 = 295.2 GB.

Generation needs room too. The KV cache for Qwen3-8B is 2 × 36 layers × 8 heads × 128 × 2 bytes = 147,456 bytes per token. A group of 8 completions at 1,024 tokens each holds 8,192 tokens, or about 1.2 GB of cache. Activations depend on batch size and sequence length; gradient checkpointing, on by default in the Nodus runtime, keeps them small.

Dropping the critic halves the full-training footprint; LoRA and QLoRA bring GRPO onto one card.

These estimates match the presets in the grpo-lora runtime reference: Qwen3-8B with QLoRA gets 1 × L40S, RTX A6000 or A100 80 GB, all 48 GB or larger. For a 70B model, the same arithmetic is in our VRAM guide for 70B models.

Where does the time go in a GRPO run?

Generation usually dominates, because every step samples G full completions before any gradient is computed. Scoring is cheap when the reward is a Python check, and the update itself is a normal backward pass over the policy or its adapter.

Most GRPO stacks speed up the sampling phase with a dedicated inference engine. TRL supports vLLM in two modes: colocate, the default, where vLLM shares GPU memory with the trainer, and server, where vLLM runs on separate GPUs. Colocate saves hardware but “may lead to memory contention on the training GPUs,” so leave headroom.

When should you use GRPO instead of DPO or SFT?

Use GRPO when you can score an answer automatically and the base model already gets it right some of the time. Here is the rule we use.

• Use SFT when you have good example outputs and want the model to imitate them.
• Use DPO when you have pairs of preferred and rejected answers but no automatic checker. DPO learns from those fixed pairs and, per the DPO paper, removes the need to sample from the model during fine-tuning, which makes it cheaper per example.
• Use GRPO when correctness can be checked in code: math answers, unit tests, structured output, games. Run a baseline first. If the model scores 0% or 100% on your tasks, every group reward is identical and GRPO has nothing to learn from; do SFT first, or pick harder tasks.
• Use full PPO only if you have a reason to want a learned value function. For verifiable rewards, GRPO gives you the same policy-gradient training with one model fewer.

How do you run GRPO training on Nodus?

On Nodus you describe the run as a TrainingJob that names the nodus/grpo-lora runtime, a model and an Environment, and Nodus runs baseline evaluation, training and final evaluation on the same held-out tasks. The trainer never grades itself. Per the training guide, completions go to a grader in isolated containers with no network access, and a grader crash counts as an infrastructure failure, not a zero score.

Install the CLI and sign in:

$ pip install nodus-compute
$ nodus login

This is the example TrainingJob from the runtime reference. It trains a LoRA adapter on Qwen3-0.6B against the graph-coloring Environment with a $5 cap:

apiVersion: nodus.dev/v1beta1
kind: TrainingJob
metadata: {name: my-grpo-lora}
spec:
  runtime: nodus/grpo-lora
  model: {uri: "hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca"}
  environment: {name: nodus/[email protected], trainTasks: 200, heldOutTasks: 64, seed: 42}
  maxCostUSD: "5.00"

Save it as my-grpo-lora.yaml. Price it with a dry run, then start it, watch it and download the results; these are the commands from the training guide, with the file name changed:

$ nodus apply -f my-grpo-lora.yaml --dry-run -o yaml     # status.estimate: cost to completion, first hold, start time
$ nodus apply -f my-grpo-lora.yaml
$ nodus get tj -w
$ nodus cp tj/<name>:outputs/results.json ./results.json

The runtime’s GRPO knobs live under spec.parameters: numGenerations (the group size, default 4), beta (default 0, which disables the reference model), maxCompletionLength (default 64 tokens), lora.r (default 8), method (LoRA, QLoRA or Full) and saveSteps (default 25). After a run, status.diagnostics.rewardSignal warns if every training reward was the same, which is the all-right or all-wrong case above. If you would rather write a reward function in Python than pick a catalog Environment, the nodus.recipes.rl SDK reference shows rl.train, which packages your reward as an Environment.

Recovery is built in. The run checkpoints every saveSteps, and after an interruption it reuses the baseline saved in its state so it does not repeat baseline grading. The general pattern is in how to make spot GPU training survive preemption.

What does a GRPO run cost on Nodus?

Training steps are cheap; the bill is mostly a question of how many steps and how long your completions are. The runtime reference publishes the seconds per step its dry-run estimate uses, re-measured nightly. They are 0.45 s on an H100, 0.7 s on an A100 80 GB, 1 s on an L40S, 1.2 s on an RTX 4090 and 1.3 s on an RTX A6000.

Multiplying by the GPU’s rate on the Nodus pricing page gives the cost of 1,000 training steps:

GPU	Listed rate	1,000 steps	Cost of 1,000 steps
H100 SXM 80 GB	$2.600/hr	450 s (0.125 h)	$0.33
A100 80 GB	$1.090/hr	700 s (0.194 h)	$0.21
L40S 48 GB	$0.793/hr	1,000 s (0.278 h)	$0.22
RTX 4090 24 GB	$0.290/hr	1,200 s (0.333 h)	$0.10
RTX A6000 48 GB	$0.333/hr	1,300 s (0.361 h)	$0.12

Prices checked October 26, 2026 (listed as hourly supply prices on the Nodus pricing page).

The fastest GPU is not the cheapest per step: the H100 is 2.7x faster than the RTX 4090 but costs 3.4x more per step.

Treat these as the training-step floor, not the whole bill. The step times come from the runtime’s own measured runs, not your model or completion length. Evaluation adds GPU time, and an 8B QLoRA run needs a 48 GB preset card, not a 24 GB RTX 4090.

The dry run prices your actual configuration before anything is charged, and maxCostUSD stops the run at your cap. For how startup, idle time and redone work change the per-job price on bigger GPUs, see what an H100 or B200 hour costs per finished job.

FAQ

Does GRPO need a reward model?

No. It needs a score for each completion, and for verifiable tasks a short function that checks the answer is enough. DeepSeekMath used a trained reward model, while DeepSeek-R1 rewarded whether final answers were correct.

What group size should I use for GRPO?

Start with 4 to 8 and raise it if rewards are noisy, since larger groups give a steadier baseline but cost more generation per step. On the Nodus runtime numGenerations accepts 2 to 64, and generationBatchSize must be a multiple of both batchSize and numGenerations.

Can GRPO train a model with full fine-tuning instead of LoRA?

Yes. The Nodus runtime accepts method: Full, and its distributed.strategy parameter can shard the model with FSDP across several GPUs. Managed multi-GPU runs reject the ZeRO strategies, so pick FSDP when the model does not fit on one card.

Is GRPO only for math and reasoning?

No. It works for any task where a program can score the output, such as code that must pass tests, JSON that must match a schema, or puzzles with a checkable answer. It is a poor fit for open-ended writing, where DPO on human preferences is usually the better tool.

Tags:

• training
• grpo
• reinforcement-learning
• fine-tuning

What does an H100 or B200 hour cost per finished job?

Modal vs RunPod: which is better for training and fine-tuning?