Spot capacity and direct SSH

Choose Instances → Launch instance → Capacity → Spot (interruptible) for interruptible compute. Existing managed offerings retain their terminal, VS Code and Jupyter access; whole-VM quotes show Direct SSH. Quotes do not guarantee launch capacity. Direct-SSH instances are organization-wide and appear in each project’s Instances list.

The direct-SSH API gives your scripts a Nodus SSH command for the dedicated node. Nodes can be preempted at any time. Your own scripts remain responsible for saving progress and restarting work during the pilot.

Pilot access

Spot is available to enabled organizations. Add a payment method and prepaid credit in Usage & billing. Dedicated nodes require confirmed funding or an account review. Starter credit alone does not unlock them. Approved pilot organizations can instead use invoice-funded credit with ACH payment due in seven days.

Sign in and save an SSH key

pip install --upgrade nodus-compute
nodus login
nodus create sshkey laptop --from-file ~/.ssh/id_ed25519.pub

You can also save public keys in the console’s Account → SSH keys. Keep the private key on your machine. Spot uses the same organization, API keys and billing account as your other Nodus work.

A quoted price is not a reservation. A location can be unavailable when no admitted price ceiling fits your bid, even if its displayed market price is lower. The request’s maximum falls back to your account default; without either, a finite platform limit still applies. If capacity checks fail, launch is refused safely.

Check prices and launch

nodus spot prices
nodus spot up h200x8 --max-price YOUR_NODE_PRICE_CAP --wait

For training, create a retained Workspace below and pass --workspace WORKSPACE_ID. A launch without it has only a temporary boot disk; releasing or preempting it does not retain training files.

Prices and ceilings are USD per complete node-hour. The quote table shows available GPU configurations and named locations; a quote does not reserve capacity. The command prints an idempotency key, waits for capacity and SSH readiness, then prints an SSH command. Run that command from the machine with your private key.

Use --ssh-key PATH.pub to override saved keys, --regions eu-1 to restrict placement to Finland, and -o json for machine-readable output. Use the stable location ID shown by nodus spot prices, or a region class such as eu. Locations keep distinct storage: Finland (eu-1), France (eu-2), Iceland (eu-3) and United States 1 (us-1) are separate choices when quoted. Without --wait, the command returns the accepted node’s current state immediately.

nodus spot ls
nodus spot rm NODE_ID

Release ends compute billing immediately and starts deletion of the VM and its boot disk. A preempted node is cleaned up automatically. Save needed files outside the boot disk before releasing it. A terminal node status reports the end of compute; resource cleanup can continue in the background.

Connect through Nodus

nodus spot ssh NODE_ID -- nvidia-smi
nodus spot ssh NODE_ID --identity ~/.ssh/id_ed25519
nodus spot ssh NODE_ID --config

The command carries SSH through the authenticated Nodus endpoint. Your private key stays on your computer. Use --identity for a nondefault key that matches one installed at launch. Review the SSH host key on first connection. --config prints an entry you can add to ~/.ssh/config for plain SSH, SCP, rsync or VS Code. Keep your Nodus CLI session or NODUS_API_KEY available when using that entry. The key needs both spot:read and spot:write; project-restricted keys cannot access these organization-wide nodes.

Upgrade the CLI and Python SDK before using this connection contract. The deprecated ip field is empty; read ssh_command after the node becomes running. Scripts that connect directly to node["ip"] must use nodus spot ssh NODE_ID instead. SSH port forwarding works through the same connection. Releasing the node closes existing relay sessions after the next authorization check.

Retain a Workspace (pilot)

A Workspace in Spot mode owns regional storage mounted at /workspace on linked nodes. Creating it starts storage allocation and billing. It does not start a machine. This differs from a managed development Workspace, which runs a session and saves its home directory.

In the console, open Workspaces → New retained storage. Choose the project, location, capacity and access, review the storage rate and reserved credit, then create it. Capacity choices include 512 GiB, 1 TiB, 2 TiB and 4 TiB, plus Custom for any whole number from 512 through 4096 GiB. For example, 1222 GiB is valid and the estimate uses exactly 1222 GiB. One instance uses a Block slot; Shared by instances uses SharedFilesystem. Each retained slot is billed for its allocated capacity. Open the Workspace to see retained regional slots and charges, and choose Launch linked instance. Only compatible regional Workspaces appear in a direct-SSH launch. Managed home storage cannot attach to these VMs. Capacity and access mode are fixed at creation; storage does not grow automatically. Compute release keeps linked storage and its charges; deleting the Workspace requires a separate confirmation. The launch form lets you reuse a slot and choose whether to wait for home-region capacity. Choose Temporary disk for disposable files, or Keep files in a workspace to attach retained storage.

apiVersion: nodus.dev/v1
kind: Workspace
metadata:
  name: spot-training
spec:
  mode: Spot
  regionalStorage:
    provider: nodus
    homeRegion: eu-1
    sizeGiB: 512
    mode: Block

Apply the manifest, read its metadata.uid with nodus get workspace spot-training -o json, then launch:

nodus spot up h200x8 --workspace WORKSPACE_ID --placement local_only --max-price YOUR_NODE_PRICE_CAP --wait --timeout 30m

Save files under /workspace. Releasing the node retains those files and their storage charge. A replacement in the same region and slot mounts the retained filesystem before becoming ready. A block slot permits one writer; 409 volume_in_use means its previous attachment has not cleared. For independent block disks use --workspace-slot worker-2; each slot allocates and bills its own capacity. SharedFilesystem supports simultaneous writers to the default slot in a region when enabled for the pilot.

The default local_first placement waits up to five minutes for home-region capacity before considering other regions. Change the bound with --local-wait 5m (zero to 15 minutes), choose local_only to keep files local, or choose cheapest to select by price immediately. Placement always respects your compute price ceiling. A different region receives separate empty storage; files are not copied automatically.

Python accepts the same options: client.spot.request("h200x1", workspace_id="wsp_...", workspace_slot="default", placement="local_only", local_wait_s=300, max_price=3).

Storage bills allocated GiB over retained time, separately from GPU usage, including while no node is running. The ordinary object storage free allowance does not apply. Storage quota and funded credit must cover each allocation; For 429 storage_quota, an Owner or Admin can raise the available storage limit in Settings › Quotas, or release storage that is no longer needed. Stopping compute does not delete retained files or free their storage allocation. Settings › Compute separately shows active Spot nodes, simultaneous starts and their editable account limits. Nodes awaiting cleanup remain counted; raising a limit does not guarantee available capacity.

To remove the data, release linked nodes and run nodus delete workspace spot-training --confirm. The API requires DELETE .../workspaces/spot-training?confirm=true; Python supports client.api.delete(..., confirm=True). Deletion waits for attachments and confirmed storage removal. This is permanent deletion of the Workspace’s regional files, including any model checkpoints saved there. In the console, deletion requires typing the Workspace name and confirming that its files will be lost.

Retained files do not restart a training process. During this pilot your command must write valid checkpoints, load them on restart, and request replacements. Managed checkpoint recovery is a separate next stage.

Retry initial capacity acquisition from the CLI

Retry is opt-in and bounded. This command permits at most three accepted startup attempts within one 30-minute deadline, including the first attempt. It uses saved SSH keys and temporary boot storage:

nodus spot up h100x8 --max-price 12 --wait --retry-startup --max-attempts 3 --timeout 30m --state ./h100-acquisition.json

Only a confirmed capacity_unavailable failure before SSH readiness can be retried, and only after cleaned_at confirms compute cleanup. Each attempt has a new node ID and request key; previous nodes remain in history with zero compute charge. Quotes, limits and funding are checked again. Supply is not guaranteed. The command honors the location, GPU and price constraints; it never falls back to on-demand or reserved compute. Explicit --ssh-key public keys remain unchanged. If omitted, each attempt uses the account’s currently saved keys.

Keep the state file and its .lock companion. They contain public request and node data, not API credentials, and are private to your user. After an interruption, use the printed resume command with the same API/account:

nodus spot up h100x8 --wait --retry-startup --state ./h100-acquisition.json --resume

Resume retains the original deadline and attempt limit. A lost POST response without a saved node ID stops for manual inspection with nodus spot ls; the CLI cannot safely resend that request. Timeout or cancellation does not release an accepted node. Inspect it and use nodus spot rm NODE_ID when appropriate.

With --workspace, retries stay in the first accepted attachment’s location and keep the same slot; files are never moved to another location. Use --placement local_only to keep initial placement at the Workspace’s home. There is no retry after a node has ever become ready, and no automatic restart of training or recovery from running-node preemption. The console retains its single-accepted-node behavior. Python waiting uses the separate timeout-bounded behavior below.

Poll safely from Python

Set SPOT_WORKSPACE_ID to the retained Workspace UID, SPOT_MAX_NODE_PRICE to your approved price ceiling, and SPOT_REQUEST_KEY to a unique identifier for one logical request. Keep the key for inspection and recovery. After an ambiguous initial POST, inspect node history before resending. This example polls every 30 seconds and releases compute after inspecting its GPUs and mount.

examples/spot/poll.py

"""Request one node with retained storage, inspect its GPUs, then release compute."""

import os
import subprocess

import nodus

client = nodus.Client()
node = client.spot.request(
    "h200x8",
    max_price=float(os.environ["SPOT_MAX_NODE_PRICE"]),
    workspace_id=os.environ["SPOT_WORKSPACE_ID"],
    placement="local_only",
    poll=30,
    timeout=1800,
    idempotency_key=os.environ["SPOT_REQUEST_KEY"],
)
try:
    subprocess.run(
        [
            "nodus",
            "spot",
            "ssh",
            node["id"],
            "--",
            "nvidia-smi && nvidia-smi topo -m && df -h /workspace",
        ],
        check=True,
    )
finally:
    client.spot.release(node["id"])

client.spot.request() waits by default, with a 15-minute timeout. Set wait=False for one immediate POST, or set timeout=1800 for a longer wait. Waiting polls the accepted node. If it fails before readiness with end_reason: "capacity_unavailable", waiting keeps polling until cleanup is confirmed, then retries after retry_after seconds with the same key and price ceiling. Retained storage stays in the first accepted location. Other ended nodes, including preemptions after readiness, raise nodus.errors.SpotNodeEnded; a timeout raises nodus.errors.SpotTimeout with node and idempotency_key for recovery. Timeout or cancellation does not release the node. Inspect or release it before starting a different request. Async callers use await client.spot.request.aio(...).

Other methods are client.spot.prices(), list(), get(node_id), release(node_id) and usage(from_="2026-10-24T00:00:00Z", to="2026-10-25T00:00:00Z"). Results are ordinary JSON dictionaries and lists. Optional schemas are available in nodus.api.models.spot.

Preemption webhooks

In Settings → Compute → Preemption webhooks, add your public HTTPS URL and save the signing secret shown once. The form selects spot.preempted; existing endpoints subscribed to that type, spot.* or all events also receive it. Owners and admins can edit, disable or rotate these endpoints using the same webhook controls.

The signed notification arrives after Nodus observes preemption. Its data.object contains the node’s id, gpu, status: "preempted" and ended_at billing cutoff. It carries no advance eviction warning and does not restart work. Verify the signature and deduplicate webhook-id; failed deliveries retry for up to three days, and each endpoint exposes delivery history and replay controls.

API and usage

Send your existing bearer token to the platform API. The Spot OpenAPI document describes the REST schemas. The SSH route additionally uses a WebSocket upgrade:

Method	Path	Result
GET	/v1/spot/prices	Current node-hour quotes
POST	/v1/spot/nodes	Accepted node or a capacity/payment refusal
GET	/v1/spot/nodes	Your nodes
GET	/v1/spot/nodes/{id}	Status, Nodus SSH command and accrued cost
GET	/v1/spot/nodes/{id}/ssh	Authenticated binary WebSocket carrying SSH (spot:write)
DELETE	/v1/spot/nodes/{id}	Release and cleanup status
GET	/v1/spot/usage?from=...&to=...	Per-node price intervals and charges

Include Idempotency-Key or idempotency_key in every POST. A retry with that key returns the same node. When a node is preempted, the key becomes available for a replacement. An unstarted capacity failure retains its key until physical cleanup is confirmed. A lost response requires inspecting node history before resending; a terminal state alone is not proof of cleanup. A POST without a key reuses an existing provisioning node of the same GPU type in your account.

503 no_capacity means no node was created; retry after retry_after seconds using the same key. 503 price_above_max requires a cheaper quote within your approved ceiling. 402 payment_required needs action in billing, while 429 node_limit requires releasing a node or changing your account limit.

An accepted node can still fail before SSH is ready. When capacity failure is confirmed, its JSON includes startup_failure_code: "capacity_unavailable" and a plain-language startup_failure_message; the console and CLI show that explanation. The node remains failed with no compute charge. A retryable failure uses end_reason: "capacity_unavailable"; cleanup exposes retry_after and cleaned_at. Missing diagnostic fields mean the specific cause is unknown. By default the CLI does not replace an accepted failed node. The opt-in initial-acquisition mode waits for confirmed cleanup before a capacity retry; an available quote does not guarantee that the next startup will succeed.

Billing begins when SSH is ready. Price changes appear as separate intervals in usage, and charges appear under Spot GPU in the existing billing history. The instance detail page shows accrued cost; posted billing usage updates on the metering cycle. Provisioning failures are unbilled. A billing warning includes a release time so you can save work and restore funding before the node is released.

Organizations approved for invoice terms see their credit limit, outstanding invoices, payment links and due dates in Usage & billing. Invoice credits become available when the approved invoice is issued; paying it settles that advance without adding the credits a second time. ACH processing can take several days. Overdue invoices or returned payments pause new launches and can trigger a release warning for running nodes.