# Why is GPU utilization low during PyTorch training?

> Low GPU utilization in PyTorch usually means the GPU is waiting on the CPU. How to measure it, what each symptom means, and what idle time costs.

Source: https://www.nodus-compute.ai/blog/low-gpu-utilization-pytorch/
Build revision: 936cf216ddbbf8ea6f98a5b1e2cf687fee2025f2

Low GPU utilization in PyTorch training usually means the GPU is waiting on the CPU: for the next batch, for a sync like `.item()`, or for Python to launch the next small kernel. On Nodus, as on any platform that bills by the hour, an NVIDIA H100 SXM listed at $2.60/hr on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/) costs the same whether its kernels run 40% or 95% of the time. Measure first with `nvidia-smi dmon` and `torch.profiler`, then fix the data loader, the syncs and the batch size, in that order.

## What does GPU utilization in nvidia-smi actually measure?

It measures time, not work. NVIDIA’s [nvidia-smi documentation](https://docs.nvidia.com/deploy/nvidia-smi/index.html) defines GPU utilization as the “percent of time over the past sample period during which one or more kernels was executing on the GPU”, with a sample period between 1/6 of a second and 1 second.

That definition has two consequences. A low number is a clear signal: the GPU had nothing to run for much of each second. A high number is weaker evidence: one tiny kernel keeping the GPU busy counts the same as a full matrix multiply. For how much of the GPU’s arithmetic you actually use, the measure is model FLOPs utilization, which we cover in [how long it takes to train an AI model](https://www.nodus-compute.ai/blog/how-long-to-train-an-ai-model/).

Watch it while training runs, one line per second, with a timestamp:

Terminal window

```bash
nvidia-smi dmon -s ut -o T
```

`-s u` prints SM and memory utilization and `-s t` adds PCIe receive and transmit throughput, per the same NVIDIA page. A sawtooth that drops to near zero between steps is the classic starved-loader pattern. A steady 30% to 50% points to syncs or small kernels instead.

## How do you tell whether the data loader is the bottleneck?

Time how long the training loop waits for each batch. This is the first check we run on any slow job, because it splits the problem in two with about ten lines of code:

```python
import time, torch

data_wait, compute = 0.0, 0.0
t0 = time.perf_counter()
for i, (x, y) in enumerate(loader):
    t1 = time.perf_counter()
    data_wait += t1 - t0
    x, y = x.cuda(non_blocking=True), y.cuda(non_blocking=True)
    loss = loss_fn(model(x), y)
    loss.backward()
    opt.step()
    opt.zero_grad(set_to_none=True)
    torch.cuda.synchronize()  # diagnosis only: makes the step time honest
    t0 = time.perf_counter()
    compute += t0 - t1
    if i == 200:
        break
print(f"share of time waiting for data: {data_wait / (data_wait + compute):.0%}")
```

Our rule of thumb: if more than 10% of the loop is spent waiting for data, fix the loader before touching the model. If the share is near zero and utilization is still low, the GPU is idle inside the step, and the profiler is the next tool.

Remove the `torch.cuda.synchronize()` line once you are done. The [PyTorch performance tuning guide](https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html) lists syncs like it as a cause of low utilization in their own right.

## How do you fix a starved DataLoader?

Load batches in parallel worker processes and copy them asynchronously. The [DataLoader docs](https://docs.pytorch.org/docs/2.15/data.html) set `num_workers=0` by default, which loads data in the main process. The tuning guide explains the cost: the training process “has to wait for the data to be available to continue the execution.”

```python
from torch.utils.data import DataLoader

loader = DataLoader(
    dataset,
    batch_size=256,
    shuffle=True,
    num_workers=8,            # tune to your CPU cores and storage
    pin_memory=True,          # page-locked host memory for faster async copies
    persistent_workers=True,  # keep workers alive between epochs
    prefetch_factor=4,        # batches loaded ahead by each worker
)

for x, y in loader:
    x = x.cuda(non_blocking=True)
    y = y.cuda(non_blocking=True)
    ...
```

Each argument does one job. `pin_memory=True` enables “faster and asynchronous memory copy from the host to the GPU”, per the tuning guide, and `non_blocking=True` lets the copy overlap with compute. `prefetch_factor` defaults to 2 when workers are on, meaning 2 x `num_workers` batches are queued across all workers. `persistent_workers=True` stops the loader from shutting workers down after each epoch.

There is no universal right `num_workers`. The tuning guide says it “should be tuned depending on the workload, CPU, GPU, and location of training data.” Try 4, 8 and 16 and keep the smallest value that drives the data-wait share near zero.

If the workers are all running and the CPU cores sit at 100%, more workers will not help. Your augmentation or decoding is the real bottleneck. Preprocess the dataset once and save the result, or move heavy transforms onto the GPU. If the CPU is idle but batches are still late, you are waiting on storage: copy the dataset to local disk before training instead of reading every file from network storage.

## Which code patterns make the GPU wait inside a training step?

Anything that forces the CPU to wait for the GPU. PyTorch queues GPU work asynchronously, and the tuning guide says to “let the CPU run ahead of the accelerator as much as possible.” It lists the usual offenders: `print(cuda_tensor)`, `cuda_tensor.item()`, copies like `.cpu()` and `.to(device)`, `nonzero()`, and Python `if` statements that depend on a CUDA tensor’s value.

The most common one is logging the loss every step:

```python
# Stalls every step: .item() waits for the GPU to finish
running_loss += loss.item()

# Better: accumulate on the GPU and read it every 100 steps
running_loss += loss.detach()
if step % 100 == 0:
    print(f"loss {running_loss.item() / 100:.4f}")
    running_loss.zero_()
```

Small batches are the second pattern. Each step then launches many short kernels, and the GPU finishes each one before Python has queued the next.

Raise the batch size as far as memory and your optimizer settings allow. Where you cannot, `torch.compile` fuses pointwise operations into fewer kernels. The tuning guide adds that `torch.compile(model, mode="reduce-overhead")` uses CUDA graphs to cut launch and sync overhead, at some cost in device memory.

## Why can a busy GPU still train slowly?

Because full FP32 math leaves the Tensor Cores unused. The tuning guide says mixed precision “offers up to 3x overall speedup on Volta and newer GPU architectures”, and asks for matrix dimensions in multiples of 8 so Tensor Core kernels apply. That does not raise the utilization number. It does finish the same work in less GPU time.

```python
for x, y in loader:
    x, y = x.cuda(non_blocking=True), y.cuda(non_blocking=True)
    with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
        loss = loss_fn(model(x), y)
    loss.backward()
    opt.step()
    opt.zero_grad(set_to_none=True)
```

The [PyTorch AMP docs](https://docs.pytorch.org/docs/2.15/amp.html) say autocast should wrap only the forward pass and the loss, as above. If you choose float16 instead of bfloat16, read their `torch.amp.GradScaler` section first.

## How do you read torch.profiler output for a slow step?

Profile a handful of steps after warm-up and look for gaps on the GPU timeline. The [torch.profiler docs](https://docs.pytorch.org/docs/2.15/profiler.html) provide a `schedule` that skips `wait` steps, runs `warmup` steps, then records `active` steps:

```python
from torch.profiler import profile, schedule, ProfilerActivity, tensorboard_trace_handler

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=schedule(wait=5, warmup=2, active=5),
    on_trace_ready=tensorboard_trace_handler("./prof"),
) as prof:
    for step, (x, y) in enumerate(loader):
        train_step(x, y)
        prof.step()
        if step >= 12:
            break

print(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=15))
```

Open the trace in TensorBoard or a Chrome trace viewer. Blank stretches on the GPU row between steps mean data waiting. Blank stretches inside a step, lined up with `cudaStreamSynchronize` or `aten::item` on the CPU row, mean a sync. A dense row of tiny kernels means launch overhead.

## Which symptom points to which fix?

This is the decision table we use, built from the PyTorch and NVIDIA docs above and the measurements in the two sections before it.

|What you see|Likely cause|First fix|
|-|-|-|
|Sawtooth in `dmon`, data-wait share above 10%|Loader in the main process or too few workers|`num_workers`, `pin_memory`, `persistent_workers`, `prefetch_factor`|
|Workers running, CPU cores at 100%|CPU-bound decoding or augmentation|Preprocess once, or move transforms to the GPU|
|CPU idle, batches still late|Reading from network storage|Copy the dataset to local disk first|
|Gaps inside the step, `aten::item` in the trace|Host-device sync from logging or Python control flow|Accumulate on device, log every N steps|
|Steady low utilization, many tiny kernels|Small batch, kernel launch overhead|Larger batch, `torch.compile`|
|High utilization, slow steps|FP32 math, no Tensor Cores|`torch.autocast` with BF16, sizes in multiples of 8|

To compare fixes on real hardware, a [Nodus Sweep](https://www.nodus-compute.ai/docs/guides/sweeps/) runs the same Job once per parameter value and reports cost, time and throughput per cell. That turns “try 4, 8 and 16 workers” into one submission instead of three.

## What does low GPU utilization cost?

You pay for the hour, not for the kernels. Take an 8 x H100 SXM job that runs 24 hours at $2.60 per GPU-hour, the rate on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/):

8 GPUs x $2.60 = $20.80 per hour, and $20.80 x 24 h = $499.20 for the run.

At 40% utilization, 60% of that bill buys waiting: $499.20 x 0.60 = $299.52. At 60% it is $199.68, at 80% it is $99.84, and at 95% it is $24.96.

![Spend on idle time in a 24-hour 8 x H100 run at $20.80/hr: $299.52 at 40% utilization, $199.68 at 60%, $99.84 at 80%, $24.96 at 95%.](https://www.nodus-compute.ai/blog-charts/low-gpu-utilization-pytorch/idle-spend.svg) *At 40% utilization, more than half of a $499.20 run pays for a GPU that has nothing to do.*

The better question is what the same work costs once fixed. That 24-hour run at 40% has 9.6 hours of GPU busy time (24 x 0.40). If a fix keeps that busy time constant and only removes waiting, the wall time becomes 9.6 h divided by the new utilization:

|Utilization|Wall time|Cost on 8 x H100|
|-|-|-|
|40%|24 h|$499.20|
|60%|16 h|$332.80|
|80%|12 h|$249.60|
|95%|10.1 h|$210.19|

*Prices checked November 6, 2026, at the Nodus H100 SXM rate of $2.60/hr. Our arithmetic: 9.6 h / utilization x $20.80/hr.*

![Cost of the same training work on 8 x H100 at $20.80/hr: $499.20 at 40% utilization, $332.80 at 60%, $249.60 at 80%, $210.19 at 95%.](https://www.nodus-compute.ai/blog-charts/low-gpu-utilization-pytorch/same-work-cost.svg) *Going from 40% to 80% utilization halves the cost of the same work, from $499.20 to $249.60.*

This is a model, not a benchmark. Real fixes rarely keep busy time exactly constant, and mixed precision shrinks it too. The direction holds, though: on this run, doubling utilization saves as much as halving the hourly rate would. For sizing the run itself, see [how many GPUs you need to train a model](https://www.nodus-compute.ai/blog/gpus-needed-to-train-a-model/).

## Does Nodus charge for idle GPU time?

Yes, while your Job is running. Nodus bills a running Job by time, so it costs its hourly rate whether the GPU is busy or waiting. The [Nodus Jobs guide](https://www.nodus-compute.ai/docs/guides/jobs/) says a Job “charges only for the time it used”. Once it finishes, “nothing is running or billed”, and `--max-cost` stops a Job that would spend more than its cap.

Terminal window

```bash
nodus run --gpu L4 --image nodus/pytorch --max-cost 5 --timeout 2h -- python train.py
```

That ends the waste after the job. Raising utilization is still the only way to cut the waste during it.

## FAQ

### Does high GPU memory use mean the GPU is busy?

No. Memory that PyTorch has allocated says nothing about activity. The memory column in `nvidia-smi dmon` is different: NVIDIA defines it as the percent of time device memory was being read or written, so it does track activity.

### Does increasing num_workers always help?

No. Past the point where data waiting reaches zero, extra workers only use more host memory, and if the CPU is already saturated they can slow the loop. Each worker also holds `prefetch_factor` batches in memory.

### Should I set torch.backends.cudnn.benchmark = True?

For convolutional networks with fixed input sizes, yes: the tuning guide says the autotuner picks the fastest convolution algorithm for your hardware. With highly variable input sizes it can cost more than it saves, and the guide says other layer types are not supported.

### How do I check utilization on every GPU of a multi-GPU job?

`nvidia-smi dmon` reports up to 16 devices at once, one row per GPU. Look for one GPU that lags: in data-parallel training, the others wait for it at every gradient sync.