# What happens to your training job when GPU capacity runs out?

> Capacity can run out before a training job starts or halfway through it. What each platform does in both cases, and what a reclaim costs you.

Source: https://www.nodus-compute.ai/blog/gpu-capacity-failover/
Build revision: 7b37c4bab16ee8776cdf5b8839e021253d37edbe

<!-- vale Nodus.Providers = NO -->

When GPU capacity runs out before a training job starts, the job fails with a capacity error or waits in a queue; when it runs out mid-run, you lose everything since your last checkpoint. Nodus handles both cases: a job with no matching capacity waits in `Queued` for up to 24 hours by default, and a reclaimed job resumes from its latest checkpoint on new capacity. How much a reclaim costs you comes down to how often you save.

This post prices one reclaim on a 10-hour, 8 x H100 run, lists the warning each major cloud gives before it takes a machine back, and gives a rule for choosing a checkpoint interval.

## What happens if there is no GPU free when you launch?

On a raw cloud account, the launch call fails and retrying is your job. Amazon EC2 returns [`InsufficientInstanceCapacity`](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshooting-launch.html) when it lacks the On-Demand capacity you asked for, and its advice is to wait a few minutes or ask for fewer instances per request. Google Compute Engine returns [`ZONE_RESOURCE_POOL_EXHAUSTED`](https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-resource-availability) and suggests another zone or trying later.

A script that launches one instance type in one zone stops right there. Teams usually end up writing a retry loop over zones and instance types, which is the start of a scheduler.

On Nodus a job asks for hardware, not for a zone, and a run with no matching capacity stays `Queued` until something fits. It is placed as soon as capacity appears, up to `placement.queueTimeout` (24 hours by default), and only then fails with `CapacityUnavailable`. The [Jobs guide](https://www.nodus-compute.ai/docs/guides/jobs/) lists the levers that make it start sooner: accept more GPU types or regions, allow interruptible capacity, or raise the maximum hourly rate.

`nodus describe` shows every offering that was considered and why each one was rejected, so a queued job is never a mystery. If the run spans several clouds you own, we covered the setup in [how to run training jobs across multiple GPU clouds](https://www.nodus-compute.ai/blog/run-training-across-multiple-gpu-clouds/).

## What happens when your machine is reclaimed mid-run?

Interruptible capacity can be taken back at any time, and the warning you get is short. These are the published windows for the three largest clouds:

|Platform|Warning before the machine goes|What restarts your run|
|-|-|-|
|Amazon EC2 Spot|[2 minutes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html) (120 seconds, interruption notice)|You|
|Google Cloud Spot VMs|[0 seconds](https://cloud.google.com/compute/docs/instances/spot) of notice by default (configurable to 120 seconds), then a best-effort shutdown period of up to 30 seconds|You|
|Azure Spot VMs|[Up to 30 seconds](https://learn.microsoft.com/en-us/azure/virtual-machines/spot-vms), best effort, through Scheduled Events|You|
|Modal|An interrupt signal runs your exit handler within its [grace period](https://modal.com/docs/guide/preemption)|Modal restarts the Function on the same input|
|Nodus|Urgent checkpoint on the reclaim notice|Nodus, from the latest committed checkpoint|

Sources linked in the table; Nodus behavior from [Attempts and recovery](https://www.nodus-compute.ai/docs/concepts/attempts-and-recovery/). Checked November 1, 2026.

![Warning before a reclaimed spot VM shuts down, by default: Amazon EC2 Spot 120 seconds, Google Cloud Spot VMs up to 30 seconds, Azure Spot VMs up to 30 seconds.](https://www.nodus-compute.ai/blog-charts/gpu-capacity-failover/reclaim-warning.svg)

*By default, two of the three largest clouds give you 30 seconds or less to save a run.*

Modal states that all of its Functions can be preempted by default, and its `nonpreemptible` option is [not supported for GPU Functions](https://modal.com/docs/guide/preemption). Modal does restart the input, but your training code still has to save and reload its own state.

## How much does one reclaim cost you?

Take a 10-hour fine-tune on 8 x NVIDIA H100. The H100 SXM is listed at $2.60/hr on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/), so the node costs 8 x $2.60 = $20.80 per hour and a clean run costs 10 x $20.80 = $208.00.

Now the machine is reclaimed at hour 7.5. We ignore restart time here, so the numbers are a floor.

* With no checkpoints, all 7.5 hours are lost and the run starts again. You pay 17.5 hours, or $364.00, which is 75% more than the clean run.
* With a checkpoint every hour, the last save was at hour 7, so 0.5 hours are lost. You pay 10.5 hours, or $218.40.

![Total GPU bill for a 10-hour 8 x H100 run reclaimed at hour 7.5: $208.00 with no reclaim, $364.00 with no checkpoints, $218.40 with hourly checkpoints, at $20.80 per hour.](https://www.nodus-compute.ai/blog-charts/gpu-capacity-failover/reclaim-cost.svg)

*One reclaim without checkpoints costs $156.00 of repeated work; hourly checkpoints cut it to $10.40.*

This arithmetic is ours, from the listed rate. The point holds at any price: the cost of a reclaim is the time since your last save multiplied by what the whole node costs per hour.

## How often should a training job checkpoint?

Pick the interval from the loss you can accept per reclaim, not from habit. Our rule: worst-case loss per reclaim equals the checkpoint interval times the node’s hourly cost. At $20.80 per hour, a 30-minute interval caps it at $10.40, and a 15-minute interval at $5.20.

The reclaim warning only helps if your save fits inside it. If writing model and optimizer state takes three minutes, a 30-second or 2-minute notice cannot save the run, and the periodic checkpoint is your real protection.

The other side of the trade is time spent saving. On Nodus, `interval: auto` sets the cadence from how often the capacity your job runs on is interrupted and how long a save takes. Per the [checkpoints guide](https://www.nodus-compute.ai/docs/guides/checkpoints/), a job gets at least four checkpoints over its expected run time, and checkpointing takes no more than about a tenth of it. Nodus also saves when the capacity gives notice, whatever the interval.

```yaml
recovery:
  continuity: Checkpointed      # restore the latest checkpoint into each new attempt
  checkpoint:
    paths: [/nodus/state]       # up to 64 absolute paths, all saved in one checkpoint
    interval: auto              # or a fixed interval from 1m to 6h
```

That block is copied from the checkpoints guide. Your program still writes its own model, optimizer and step files under `/nodus/state` and loads them on start; Nodus saves and restores the directory, never process memory. We walk through the training-loop side in [automatic checkpointing and resume for training](https://www.nodus-compute.ai/blog/automatic-checkpointing-and-resume-for-training/).

## How does Nodus move a reclaimed run to new capacity?

Nodus recovers make-before-break. On a reclaim notice it asks your run for an urgent checkpoint and, at the same time, prepares a replacement machine. The replacement starts only once the old machine is provably gone, so two copies of your run never write over each other.

A machine that simply goes silent is handled the same way. After 20 seconds without heartbeats a replacement is prepared, and after 60 seconds an attempt on shared capacity is declared lost and replaced. Each new attempt gets a higher epoch number, and Nodus accepts checkpoints only from the current one.

From the command line, the request names hardware and a spend cap:

Terminal window

```sh
nodus run --gpu H100:8 --max-cost 40 -d -- torchrun train.py
```

`nodus get attempts -l nodus.dev/job=<name>` then lists every attempt of the run, and `nodus describe` shows the placement, the latest checkpoint, cost and events.

## When does a run still fail?

Recovery is bounded, so a broken job does not loop forever on your money. Per the [recovery docs](https://www.nodus-compute.ai/docs/concepts/attempts-and-recovery/), a run fails with a typed reason when two attempts in a row make no progress (`NoProgress`), when the same checkpoint fails to restore twice (`RestoreFailed`), or when it uses up `recovery.maxAttempts`. That limit is 8 recoveries by default and 3 for distributed jobs.

Failures caused by your own command, such as a non-zero exit or running out of memory, fail fast and are not retried. A preemption after real progress never counts against the `NoProgress` rule.

## When is it better not to rely on failover at all?

Recovery reduces lost work; it does not make a reclaim free. If your run cannot tolerate any restart, for example a short benchmark whose timing must be uninterrupted, run it on capacity that is never reclaimed. On Nodus that is `--interruptible never`; elsewhere it means on-demand or reserved instances.

Reserved capacity is also the better choice when you need the same large cluster for weeks. A queue that waits for capacity does not help when you need 64 GPUs on a fixed date. On Nodus, [Reserved Compute (Beta)](https://www.nodus-compute.ai/docs/guides/marketplace/) holds capacity for an agreed configuration and reservation window.

For everything else, the cheapest setup is interruptible capacity plus checkpoints sized to the loss you can accept. We collected published interruption numbers in [how often spot GPUs are interrupted](https://www.nodus-compute.ai/blog/how-often-are-spot-gpus-interrupted/).

## FAQ

### Will Nodus move my run to another region when capacity runs out?

Only to the regions you allow. Set `placement.regions` to region classes such as `us` or `eu` and the run waits rather than landing elsewhere; capacity with no reported location never satisfies a non-empty list.

### Can I make a run fail instead of recovering?

Yes. Set `recovery.onInterruption: Fail` and the first interruption ends the run with `Preempted` or `NodeLost`. That suits work you would rather rerun by hand.

### Do multi-node training jobs recover the same way?

They recover from a collective checkpoint. A stateful multi-node Nodus job uses `recovery.checkpoint.format: Dcp`, every rank saves at the same step, and a new gang restores the committed checkpoint. Distributed jobs get 3 recoveries by default instead of 8.

### What if the old machine comes back after a replacement was prepared?

The replacement is released and your run continues where it was. Per the recovery docs, you are not charged for the replacement in that case.