# How do you train a model on a cloud GPU, step by step?

> Make your script save and resume, size the GPU by memory, time a 200-step probe, then launch with a cost cap. A worked $1.67 run, with the code.

Source: https://www.nodus-compute.ai/blog/train-a-model-on-a-cloud-gpu/
Build revision: 364dd57152437ff26b6a4ebad4f685ff5f4fcb5f

To train a model on a cloud GPU, get your script working locally on a small slice of data, make it save and reload its own checkpoints, pick the cheapest GPU whose memory fits, time a short probe, then launch the full run with a spending cap. On Nodus the launch is one `nodus run` command, and if the machine under it is reclaimed, Nodus restores your checkpoint directory on a new one and your script carries on. In the worked example below, a 1.5B-parameter model on an NVIDIA RTX A6000 listed at $0.333/hr costs $1.67 for the whole run, and an interruption adds at most 15 minutes of repeated work.

This post is the path from a laptop script to a cloud job that survives interruption. The checkpoint code is a pattern we ran ourselves on CPU on November 21, 2026; the cost math uses listed prices from the [Nodus pricing page](https://www.nodus-compute.ai/pricing/) and an example step time you replace with your own.

## What should work on your laptop before you pay for a GPU?

Your script should run end to end on your laptop, on a tiny slice of data, before it touches a paid GPU. Every bug you find locally is one you do not pay GPU rates to find.

Check four things. The script picks its device with `torch.cuda.is_available()` instead of hardcoding `"cuda"`, and its paths come from environment variables, not your home directory. It finishes a few hundred steps on CPU or a small local GPU without errors. And it writes the file you care about, usually the trained weights, to one known place.

That last point matters in the cloud. On Nodus, results you want to download go to `/nodus/outputs`, and recovery state goes to `/nodus/state`. They are different on purpose: the state directory is restored into the next attempt, while outputs are what you copy back at the end.

## How do you make a PyTorch script save and resume?

Save everything needed to continue, write it atomically, and load it when the script starts. A cloud machine can disappear mid-run, and the restart begins your command from the top, so the script has to find its own progress.

The pattern below is plain PyTorch. It saves the model, optimizer, learning-rate scheduler, RNG state and step every 1,000 steps to `NODUS_STATE_DIR`, falling back to `./state` on your laptop. It is a pattern, not a finished trainer: the model and the random batches are stand-ins for yours.

```python
import os, time, torch
from pathlib import Path

STATE_DIR = Path(os.environ.get("NODUS_STATE_DIR", "./state"))
CKPT = STATE_DIR / "ckpt.pt"
TOTAL_STEPS = int(os.environ.get("STEPS", "20000"))
SAVE_EVERY = 1000

device = "cuda" if torch.cuda.is_available() else "cpu"
torch.manual_seed(0)
model = torch.nn.Sequential(torch.nn.Linear(64, 256), torch.nn.ReLU(), torch.nn.Linear(256, 1)).to(device)
opt = torch.optim.AdamW(model.parameters(), lr=1e-3)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=TOTAL_STEPS)

def save(step):
    STATE_DIR.mkdir(parents=True, exist_ok=True)
    tmp = CKPT.with_suffix(".tmp")
    torch.save({"step": step, "model": model.state_dict(), "opt": opt.state_dict(),
                "sched": sched.state_dict(), "rng": torch.get_rng_state()}, tmp)
    os.replace(tmp, CKPT)  # atomic: a crash mid-save never leaves a half-written ckpt.pt

start = 0
if CKPT.exists():
    s = torch.load(CKPT, map_location=device)
    model.load_state_dict(s["model"]); opt.load_state_dict(s["opt"])
    sched.load_state_dict(s["sched"]); torch.set_rng_state(s["rng"])
    start = s["step"]
    print(f"resumed from step {start}", flush=True)

t0 = time.time()
for step in range(start, TOTAL_STEPS):
    x = torch.randn(128, 64, device=device)  # stand-in for your next batch
    loss = (model(x) - x.sum(1, keepdim=True)).pow(2).mean()
    opt.zero_grad(); loss.backward(); opt.step(); sched.step()
    if (step + 1) % SAVE_EVERY == 0:
        save(step + 1)
        print(f"step {step + 1} loss {loss.item():.4f} saved", flush=True)
    if step + 1 == start + 200 and device == "cuda":  # the probe numbers you need for step 4
        print(f"{(time.time() - t0) / 200:.3f} s/step, peak {torch.cuda.max_memory_allocated() / 1e9:.1f} GB", flush=True)

out = Path("/nodus/outputs") if Path("/nodus").exists() else Path("./outputs")
out.mkdir(parents=True, exist_ok=True)
torch.save(model.state_dict(), out / "model.pt")  # the file you download; the state dir is only for recovery
```

We tested the recovery path on CPU with PyTorch 2.15: we killed the process with `kill -9` right after its step 1,000 save, started it again with the same state directory, and it printed `resumed from step 1000` and finished. That checks the save and load logic, nothing about a cloud machine. Two gaps to close in your own trainer: save your data loader’s position, and on GPU also save `torch.cuda.get_rng_state_all()`. Our post on [surviving spot preemption](https://www.nodus-compute.ai/blog/resume-training-after-spot-preemption/) goes deeper on what belongs in a checkpoint.

## How do you pick the right GPU for training?

Pick the cheapest GPU whose memory holds your training state with room to spare, then let a timed probe decide whether a faster card is worth its price. Memory is a hard floor; speed is a cost question.

For full training in mixed precision with the Adam optimizer, model state takes about 16 bytes per parameter, per the [ZeRO paper](https://arxiv.org/abs/1910.02054): weights, gradients, and the optimizer’s FP32 copies. Activations come on top and grow with batch size and sequence length. Our rule of thumb is to leave at least a quarter of the card free for activations and the CUDA runtime, then confirm with the probe’s peak-memory line.

Worked example: a 1.5B-parameter model needs 1.5 x 16 = 24 GB of model state. Add a quarter and you want about 30 GB, so every 24 GB card is out and 48 GB is the smallest class that fits. LoRA fine-tunes and inference need far less; our [VRAM calculation guide](https://www.nodus-compute.ai/blog/how-to-calculate-llm-vram/) covers those cases, and [how many GPUs you need to train a model](https://www.nodus-compute.ai/blog/gpus-needed-to-train-a-model/) covers runs that outgrow one card.

|GPU (memory)|Listed Nodus price|Price vs RTX A6000|
|-|-|-|
|RTX A6000 (48 GB)|$0.333/hr|1.00x|
|RTX 6000 Ada (48 GB)|$0.743/hr|2.23x|
|L40S (48 GB)|$0.793/hr|2.38x|
|A100 (80 GB)|$1.090/hr|3.27x|
|RTX PRO 6000 (96 GB)|$1.350/hr|4.05x|
|H100 SXM (80 GB)|$2.600/hr|7.81x|

Prices checked November 21, 2026 on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/) (hourly supply price). The ratio column is our arithmetic.

![Listed Nodus hourly prices for GPUs with at least 48 GB: RTX A6000 $0.333, RTX 6000 Ada $0.743, L40S $0.793, A100 80 GB $1.090, RTX PRO 6000 $1.350 and H100 SXM $2.600.](https://www.nodus-compute.ai/blog-charts/train-a-model-on-a-cloud-gpu/gpu-prices.svg) *Every card here fits the 30 GB example; the H100 costs 7.81 times as much per hour as the cheapest one.*

The ratio column is your decision rule. A faster card is cheaper for the whole run only if it finishes a step faster by more than its price ratio. An H100 SXM at 7.81x the A6000’s hourly price must run each step more than 7.81 times as fast to win on cost. If your deadline matters more than the bill, pay for speed knowingly.

## How do you estimate the cost before the full run?

Run a 200-step probe on the GPU you picked, read the seconds per step and the peak memory, and multiply out. It costs cents and replaces guessing with a measurement on the real hardware.

The `nodus run` flags below come from the [quickstart](https://www.nodus-compute.ai/docs/getting-started/quickstart/) and the CLI reference. The first command only returns an estimate; the second, without `--dry-run`, starts paid compute with a $1 cap:

Terminal window

```sh
nodus run --name probe-1p5b --gpu A6000 --image nodus/pytorch \
  --env STEPS=200 --max-cost 1 --timeout 20m --no-warm --dry-run -- python train.py
nodus run --name probe-1p5b --gpu A6000 --image nodus/pytorch \
  --env STEPS=200 --max-cost 1 --timeout 20m --no-warm -- python train.py
```

Now the worked cost. Suppose the probe prints `0.900 s/step, peak 31.5 GB`. Those are example numbers, not a measurement: use the ones your probe prints. The arithmetic:

* Probe: 200 steps x 0.9 s = 180 s, which is 0.05 h x $0.333 = about $0.02 of GPU time, plus any minimum charge the dry run shows.
* Full run: 20,000 steps x 0.9 s = 18,000 s = 5 h.
* Cost: 5 h x $0.333/hr = $1.665, so $1.67.

If you also probe an H100 SXM, the break-even step time is 0.9 s / 7.81 = 0.115 s. Slower than that, and the A6000 is the cheaper way to finish.

## How do you launch the run so it survives interruption?

Launch with a checkpoint path, a cost cap and a timeout, and detach. The `--checkpoint /nodus/state` flag tells Nodus which directory to save while the Job runs and to restore into the next attempt, per the [checkpoints guide](https://www.nodus-compute.ai/docs/guides/checkpoints/).

Terminal window

```sh
nodus run --name train-1p5b --gpu A6000 --image nodus/pytorch \
  --checkpoint /nodus/state --interruptible allow --max-cost 4 --timeout 8h -d \
  --output model=/nodus/outputs/model.pt -- python train.py
nodus logs -f job/train-1p5b
nodus describe job/train-1p5b
```

`--interruptible allow` lets Nodus use interruptible capacity only when it is cheaper to finish there. Nodus picks the checkpoint cadence from how often that capacity is interrupted, with at least four checkpoints over the expected run time, and it also checkpoints when capacity gives notice before it is reclaimed. Your own save every 1,000 steps is the floor under all of that.

Here is what checkpointing is worth in the worked example. Say the machine goes away at hour 4. With no checkpoint, you have paid for 4 hours and start the 5-hour run again: 9 h x $0.333 = $3.00. With a save every 1,000 steps (every 15 minutes at 0.9 s per step), you repeat at most 15 minutes: 5.25 h x $0.333 = $1.75 at most.

![Cost of the worked 5-hour run on an RTX A6000 at $0.333 per hour: $1.67 with no interruption, $3.00 if interrupted at hour 4 with no checkpoint, and at most $1.75 if interrupted with a save every 15 minutes.](https://www.nodus-compute.ai/blog-charts/train-a-model-on-a-cloud-gpu/interruption-cost.svg) *Without a checkpoint, one interruption at hour 4 nearly doubles the bill; with one, it adds at most 8 cents.*

The $4 cap sits above both the $1.75 worst case and the $3.00 no-checkpoint case, and still stops a runaway loop. To pause on purpose, `nodus suspend job/train-1p5b` checkpoints first and releases the machine, and `nodus resume job/train-1p5b` continues from that save.

## How do you get the data in and the model out?

Small datasets ride along with your code; large ones go in a Volume; the trained model comes back with `nodus cp`. `nodus run` uploads your current directory and skips files in `.gitignore` or `.nodusignore`. Uploads above 500 MiB compressed need an explicit flag, which is the sign your data belongs in a Volume instead.

Terminal window

```sh
nodus create volume data --size 100Gi
nodus volume put data ./data /data
```

Mount it with `--volume data:/data:ro` on the `nodus run` line, as the [Volumes guide](https://www.nodus-compute.ai/docs/guides/volumes/) describes. When the Job succeeds, copy the declared output back; the download is checked against its SHA-256 digest:

Terminal window

```sh
nodus cp job/train-1p5b:outputs/model model.pt
```

Finished Jobs, with their logs and outputs, are kept for 30 days and then deleted, so copy what you need.

## FAQ

### Can I train on a cloud GPU from a Jupyter notebook instead?

Yes, for exploring. A notebook is the right tool for the laptop stage of this guide, and our post on [Jupyter on a cloud GPU](https://www.nodus-compute.ai/blog/jupyter-notebook-on-cloud-gpu/) covers it. For the long run, convert the notebook to a script, because a Job restarts your command from the top after an interruption and a notebook’s in-memory state does not survive that.

### Does Nodus resume my training loop for me?

No. Nodus saves and restores the files in your state directory and restarts your command on a new machine; your script still has to load the checkpoint on start, as the pattern above does. With the Python SDK installed, `nodus.checkpoint.on_request()` can also call your save function when Nodus asks for a checkpoint.

### What happens if the run hits the cost cap?

The `--max-cost` flag is the Job’s spending limit, so a run that reaches it does not keep spending. The checkpoints guide says to review the cost before raising the cap; set it above your worst-case estimate, not your best case.

### Should I use a Hugging Face Trainer instead of a custom loop?

If your model fits its abstractions, yes: Trainer already saves and resumes. The Nodus checkpoints guide says to point `output_dir` at the state directory and call `trainer.train(resume_from_checkpoint=True)` when `NODUS_RESTORED` is `1`.
