# Which GPU platforms checkpoint and resume training for you? > Who saves, who restarts and who resumes a training run after a lost GPU, compared across Nodus, Modal, AWS SageMaker, RunPod and Vast.ai. Source: https://www.nodus-compute.ai/blog/automatic-checkpointing-and-resume-for-training/ Build revision: 07ca2047fc4921f13a96d7a4c6f072854484b5ad We checked five platforms against their own docs. Nodus, AWS SageMaker managed spot training and Modal all restart a preempted training run and hand it back the files it saved, but only Nodus also tells your program when to save, including an urgent request when capacity is about to be reclaimed. RunPod Pods and Vast.ai interruptible instances keep your disk and leave the restart to you. One thing is true everywhere: no platform resumes your training loop for you. Your script still has to look for its last checkpoint when it starts and load it. What differs is everything around that load, and that is the part that decides whether a run finishes unattended. This post compares the platforms. For the code itself, see [how to make spot GPU training survive preemption](https://www.nodus-compute.ai/blog/resume-training-after-spot-preemption/). ## Who saves, who restarts and who resumes? Recovery after a lost GPU has five steps. Something has to trigger a save, copy it off the machine, start a new machine, put the files back and continue the loop. Here is who does each one, according to each platform’s own documentation. |Step|Nodus|Modal|AWS SageMaker managed spot|RunPod Pods|Vast.ai interruptible| |-|-|-|-|-|-| |Trigger a save before a reclaim|Platform sends an urgent checkpoint request|Your exit handler, 30 s grace|Your code or framework callback|Your code|Your code| |Scheduled saves|Platform-timed (`interval: auto`) or fixed, from 1 min to 6 h|Your code|Your code or framework callback|Your code|Your code| |Copy the checkpoint off the machine|Platform, from `/nodus/state`|Platform, once you write to a Volume|Platform, syncs `/opt/ml/checkpoints` to S3|You, onto a network volume|You, to your own storage| |Start a replacement|Platform, on another machine|Platform, on the same input|Platform, on the next Spot instance|You, or a beta migration you trigger|Same instance resumes when it regains priority| |Put the files back|Platform restores the state directory|Volume mounted again|Platform copies S3 back to the local path|Attach the network volume|Disk stays on that machine| |Continue the training loop|Your code|Your code|Your code (some built-in algorithms do it)|Your code|Your code| Sources: [Nodus checkpoints](https://www.nodus-compute.ai/docs/guides/checkpoints/), [Nodus attempts and recovery](https://www.nodus-compute.ai/docs/concepts/attempts-and-recovery/), [Modal preemption](https://modal.com/docs/guide/preemption), [Modal exit handlers](https://modal.com/docs/guide/lifecycle-functions), [Modal Volumes](https://modal.com/docs/guide/volumes), [SageMaker checkpoints](https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints.html), [SageMaker managed spot training](https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html), [RunPod maintenance and data safety](https://docs.runpod.io/pods/maintenance-and-outages), [RunPod Pod migration](https://docs.runpod.io/pods/troubleshooting/pod-migration), [Vast.ai instance types](https://docs.vast.ai/guides/instances/choosing/instance-types), [Vast.ai rental types FAQ](https://docs.vast.ai/guides/reference/faq/rental-types). Checked October 27, 2026. Read across the rows and three groups appear. Nodus, SageMaker and Modal own the restart and the file handling once your code writes to the right place. Only Nodus also owns the save trigger and the interval. On RunPod and Vast.ai the platform keeps your disk, and the rest of recovery is yours to build. ## What does Nodus do automatically? Nodus treats every run as a series of attempts. When a provider reclaims interruptible capacity, Nodus sends your program an urgent checkpoint request and starts preparing a replacement machine at the same time. The replacement only starts once the old machine is provably gone, so two machines never write the same checkpoint. A checkpoint committed during the notice still counts. Between notices, `interval: auto` sets the cadence from how often the capacity is interrupted and how long your saves take. A run gets at least four checkpoints, and checkpointing takes no more than about a tenth of the run time. A machine that stops sending heartbeats gets a replacement prepared after 20 seconds. On shared capacity it is replaced after 60 seconds of silence. Recovery is bounded. A run fails with `NoProgress` after two attempts in a row make no progress, and it gets 8 recoveries by default, or 3 for distributed Jobs. Your code’s share is to write its state to `/nodus/state` and read it back on start. Hugging Face Trainer users can set `integration: HFTrainer` and Nodus answers the checkpoint requests from inside Trainer. ## Does AWS SageMaker managed spot training do the same? Mostly. With managed spot training and a checkpoint S3 URI, SageMaker syncs your local checkpoint directory to Amazon S3 during training. After a Spot interruption it resumes the job on the next Spot instance and copies the checkpoints back before your script starts ([SageMaker checkpoints](https://docs.aws.amazon.com/sagemaker/latest/dg/model-checkpoints.html)). AWS Deep Learning Containers and some built-in algorithms checkpoint without script changes. Plan around two limits. First, managed spot training never falls back to on-demand instances. If Spot capacity is unavailable, the job waits until `MaxWaitTimeInSeconds` and then stops ([managed spot training](https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html)). Second, the save itself is still triggered by your code or framework, since SageMaker copies what is in the directory rather than asking your program for a consistent state. Our [Nodus vs AWS SageMaker](https://www.nodus-compute.ai/blog/nodus-vs-aws-sagemaker/) post compares the two on price and setup. ## What does Modal handle, and what is left to you? Modal restarts. Every Modal Function is preemptible by default, `nonpreemptible` is not supported for GPU Functions, and after a preemption Modal restarts the Function on the same input ([Modal preemption](https://modal.com/docs/guide/preemption)). On a preemption your exit handler runs with a 30 second grace period ([lifecycle functions](https://modal.com/docs/guide/lifecycle-functions)). The saving is yours. Modal’s [long-training example](https://modal.com/docs/examples/long-training) has your code write checkpoints to a Volume, look for the latest one on start, and use `modal.Retries` to restart. Volumes commit in the background every few seconds, so once a file is written it leaves the machine without extra work ([Modal Volumes](https://modal.com/docs/guide/volumes)). A single call can run for at most 24 hours ([timeouts](https://modal.com/docs/guide/timeouts)), which is another reason long runs need checkpoints there. The full comparison is in [Nodus vs Modal](https://www.nodus-compute.ai/blog/nodus-vs-modal/). ## What about RunPod and Vast.ai? Both keep your data and leave recovery to you. RunPod’s docs say hardware failures can happen without warning, that container storage is lost when a Pod is interrupted, restarted, stopped or terminated, and that long jobs should checkpoint every hour to every few hours onto a network volume ([RunPod data safety](https://docs.runpod.io/pods/maintenance-and-outages)). If your Pod’s GPU is taken while it is stopped, a beta migration you trigger copies `/workspace` to a new Pod with a new ID ([Pod migration](https://docs.runpod.io/pods/troubleshooting/pod-migration)). Vast.ai interruptible instances are stopped when a higher bid wins, which kills running processes. A paused instance resumes when it regains priority, possibly after a long wait, and the docs tell you to save to disk often and checkpoint long jobs ([rental types FAQ](https://docs.vast.ai/guides/reference/faq/rental-types)). ## What does a lost checkpoint interval cost? On every platform, an interruption without a save on the notice costs the work since the last scheduled checkpoint. That is half an interval on average and a full interval at worst. We priced it on two common training shapes using rates listed on the [Nodus pricing page](https://www.nodus-compute.ai/pricing/): 8 x NVIDIA H100 SXM at $2.60/hr each, or $20.80/hr, and 8 x NVIDIA B200 at $5.50/hr each, or $44.00/hr. |Checkpoint every|Average redone work|8 x H100 ($20.80/hr)|8 x B200 ($44.00/hr)| |-|-|-|-| |15 min|7.5 min|$2.60|$5.50| |30 min|15 min|$5.20|$11.00| |60 min|30 min|$10.40|$22.00| |120 min|60 min|$20.80|$44.00| Prices checked October 27, 2026. Our calculation: half the interval times the hourly rate of the run. Save time and restart time are not included. ![Average redone work per interruption: on 8 x H100 it is $2.60 at a 15-minute interval, $5.20 at 30, $10.40 at 60 and $20.80 at 120; on 8 x B200 it is $5.50, $11.00, $22.00 and $44.00.](https://www.nodus-compute.ai/blog-charts/automatic-checkpointing-and-resume-for-training/per-interruption.svg) *Each extra hour between checkpoints adds about half an hour of redone work to every interruption.* Now take one run: 24 hours on 8 x H100, which is 192 GPU-hours or $499.20 at $2.60/hr, interrupted three times. Saving every 2 hours redoes $62.40 of work on average (3 x 1 h x $20.80), or $124.80 at worst (3 x 2 h x $20.80). That is 12.5% to 25% on top of the run. Saving every 30 minutes cuts that to $15.60 on average (3 x 0.25 h x $20.80) and $31.20 at worst, before counting the extra save time. ![Redone work from three interruptions on a 24-hour, 8 x H100 run costing $499.20: $62.40 on average and $124.80 at worst when saving every 2 hours, $15.60 and $31.20 when saving every 30 minutes.](https://www.nodus-compute.ai/blog-charts/automatic-checkpointing-and-resume-for-training/run-24h.svg) *A save that commits during the reclaim notice takes the redone work close to zero, which is why who triggers it matters.* A save on the reclaim notice changes the picture, but only if the save fits the warning. On Nodus the platform sends the request as soon as the notice arrives. On Modal you write that save in an exit handler with 30 seconds. On the others you watch for the notice yourself. When the save does not fit, the scheduled interval is what you pay, and [our interval model](https://www.nodus-compute.ai/blog/resume-training-after-spot-preemption/) shows how to pick it. ## Which one should you pick? Our decision rule, from the table above: * Your runs are short, or a lost hour is cheap, and you are happy to restart by hand: any platform works. RunPod on-demand Pods are dedicated to you and cannot be displaced by other users ([Pod pricing](https://docs.runpod.io/pods/pricing)). * You want Spot prices and live in AWS, with data in S3: SageMaker managed spot training, accepting that it waits for Spot capacity rather than falling back. * Your training is a Python function that fits Modal’s model and its 24 hour call limit with retries: Modal, with your own save and resume code on a Volume. * You want runs to finish unattended on the cheapest capacity, with the platform asking for saves and choosing the interval: Nodus. Whether interruptible capacity pays off at all for your run is a separate question, covered in [spot vs on-demand GPUs](https://www.nodus-compute.ai/blog/spot-vs-on-demand-gpus/). ## Where are the other platforms the better choice? SageMaker is the better fit when your training already runs on AWS Deep Learning Containers. Some frameworks there checkpoint without script changes, and your data in S3 and your AWS commitments stay where they are. Modal is the better fit for teams that already write Modal Functions, since background Volume commits make the copy step disappear. RunPod suits interactive work on a dedicated Pod, with a network volume that moves between Pods. Vast.ai’s bidding lets you set your own price, and a paused instance keeps its disk on the same machine. If your job cannot checkpoint at all, an on-demand machine anywhere beats any recovery feature. ## How do you turn it on with Nodus? Write your state to `/nodus/state`, load it on start, and run: Terminal window ```sh pip install nodus-compute nodus login nodus run --gpu H100:8 --interruptible=allow --checkpoint /nodus/state --max-cost 550 -d -- torchrun --nproc_per_node=8 train.py ``` `--interruptible` accepts `allow`, `prefer` or `never`. The `--max-cost` cap stops the run at $550, a little above the $499.20 the 24-hour example costs at list price. `nodus describe job/` shows each attempt, why it ended and the latest checkpoint. ## FAQ ### Does Nodus edit my command to add a resume flag? No. Nodus restores files, never process memory, and never adds resume flags. A restored attempt sets `NODUS_RESTORED=1`, which your script can check before loading. ### What happens when the run hits its `--max-cost` cap? A checkpointed Nodus Job checkpoints and then suspends with reason `MaxCostReached`. It continues from that checkpoint when you raise the cap, and a Job stopped by an empty balance resumes on its own once it is funded again. ### Does SageMaker copy checkpoints I upload to S3 mid-job? No. Checkpoints already in S3 are copied into the container when the job starts, and files added to the S3 folder later are not. Deleting a checkpoint in the container also deletes it in S3. ### How large can a Nodus checkpoint be? Up to 1 TiB by default (`maxSize: 1Ti`), across up to 64 paths. After a Job finishes, Nodus keeps only its final checkpoint once seven days have passed.