# Spot capacity and direct SSH

> Request dedicated interruptible GPU nodes, connect over SSH, and track their charges.

Source: https://www.nodus-compute.ai/docs/guides/spot/
Build revision: d5a74a46071fc8a88f3e478e0764758e38ec0069

Choose **Instances → Launch instance → Capacity → Spot (interruptible)** for interruptible compute. Existing managed offerings retain their terminal, VS Code and Jupyter access; whole-VM quotes show **Direct SSH**. Quotes do not guarantee launch capacity. Direct-SSH instances are organization-wide and appear in each project’s Instances list.

The direct-SSH API gives your scripts a Nodus SSH command for the dedicated node. Nodes can be preempted at any time. Your own scripts remain responsible for saving progress and restarting work during the pilot.

Pilot access

Spot is available to enabled organizations. Add a payment method and prepaid credit in **Usage & billing**. Dedicated nodes require confirmed funding or an account review. Starter credit alone does not unlock them. Approved pilot organizations can instead use invoice-funded credit with ACH payment due in seven days.

## Sign in and save an SSH key

Terminal window

```console
pip install --upgrade nodus-compute
nodus login
nodus create sshkey laptop --from-file ~/.ssh/id_ed25519.pub
```

You can also save public keys in the console’s **Account → SSH keys**. Keep the private key on your machine. Spot uses the same organization, API keys and billing account as your other Nodus work.

A quoted price is not a reservation. A location can be unavailable when no admitted price ceiling fits your bid, even if its displayed market price is lower. The request’s maximum falls back to your account default; without either, a finite platform limit still applies. If capacity checks fail, launch is refused safely.

## Check prices and launch

Terminal window

```console
nodus spot prices
nodus spot up h200x8 --max-price YOUR_NODE_PRICE_CAP --wait
```

For training, create a retained Workspace below and pass `--workspace WORKSPACE_ID`. A launch without it has only a temporary boot disk; releasing or preempting it does not retain training files.

Prices and ceilings are USD per **complete node-hour**. The quote table shows available GPU configurations and named locations; a quote does not reserve capacity. The command prints an idempotency key, waits for capacity and SSH readiness, then prints an SSH command. Run that command from the machine with your private key.

Use `--ssh-key PATH.pub` to override saved keys, `--regions eu-1` to restrict placement to Finland, and `-o json` for machine-readable output. Use the stable location ID shown by `nodus spot prices`, or a region class such as `eu`. Locations keep distinct storage: Finland (`eu-1`), France (`eu-2`), Iceland (`eu-3`) and United States 1 (`us-1`) are separate choices when quoted. Without `--wait`, the command returns the accepted node’s current state immediately.

Terminal window

```console
nodus spot ls
nodus spot rm NODE_ID
```

Release ends compute billing immediately and starts deletion of the VM and its boot disk. A preempted node is cleaned up automatically. Save needed files outside the boot disk before releasing it. A terminal node status reports the end of compute; resource cleanup can continue in the background.

## Connect through Nodus

Terminal window

```console
nodus spot ssh NODE_ID -- nvidia-smi
nodus spot ssh NODE_ID --identity ~/.ssh/id_ed25519
nodus spot ssh NODE_ID --config
```

The command carries SSH through the authenticated Nodus endpoint. Your private key stays on your computer. Use `--identity` for a nondefault key that matches one installed at launch. Review the SSH host key on first connection. `--config` prints an entry you can add to `~/.ssh/config` for plain SSH, SCP, rsync or VS Code. Keep your Nodus CLI session or `NODUS_API_KEY` available when using that entry. The key needs both `spot:read` and `spot:write`; project-restricted keys cannot access these organization-wide nodes.

Upgrade the CLI and Python SDK before using this connection contract. The deprecated `ip` field is empty; read `ssh_command` after the node becomes `running`. Scripts that connect directly to `node["ip"]` must use `nodus spot ssh NODE_ID` instead. SSH port forwarding works through the same connection. Releasing the node closes existing relay sessions after the next authorization check.

## Retain a Workspace (pilot)

A Workspace in `Spot` mode owns regional storage mounted at `/workspace` on linked nodes. Creating it starts storage allocation and billing. It does not start a machine. This differs from a managed development Workspace, which runs a session and saves its home directory.

In the console, open **Workspaces → New retained storage**. Choose the project, location, capacity and access, review the storage rate and reserved credit, then create it. Capacity choices include 512 GiB, 1 TiB, 2 TiB and 4 TiB, plus **Custom** for any whole number from 512 through 4096 GiB. For example, 1222 GiB is valid and the estimate uses exactly 1222 GiB. **One instance** uses a Block slot; **Shared by instances** uses SharedFilesystem. Each retained slot is billed for its allocated capacity. Open the Workspace to see retained regional slots and charges, and choose **Launch linked instance**. Only compatible regional Workspaces appear in a direct-SSH launch. Managed home storage cannot attach to these VMs. Capacity and access mode are fixed at creation; storage does not grow automatically. Compute release keeps linked storage and its charges; deleting the Workspace requires a separate confirmation. The launch form lets you reuse a slot and choose whether to wait for home-region capacity. Choose **Temporary disk** for disposable files, or **Keep files in a workspace** to attach retained storage.

```yaml
apiVersion: nodus.dev/v1
kind: Workspace
metadata:
  name: spot-training
spec:
  mode: Spot
  regionalStorage:
    provider: nodus
    homeRegion: eu-1
    sizeGiB: 512
    mode: Block
```

Apply the manifest, read its `metadata.uid` with `nodus get workspace spot-training -o json`, then launch:

Terminal window

```console
nodus spot up h200x8 --workspace WORKSPACE_ID --placement local_only --max-price YOUR_NODE_PRICE_CAP --wait --timeout 30m
```

Save files under `/workspace`. Releasing the node retains those files and their storage charge. A replacement in the same region and slot mounts the retained filesystem before becoming ready. A block slot permits one writer; `409 volume_in_use` means its previous attachment has not cleared. For independent block disks use `--workspace-slot worker-2`; each slot allocates and bills its own capacity. `SharedFilesystem` supports simultaneous writers to the default slot in a region when enabled for the pilot.

The default `local_first` placement waits up to five minutes for home-region capacity before considering other regions. Change the bound with `--local-wait 5m` (zero to 15 minutes), choose `local_only` to keep files local, or choose `cheapest` to select by price immediately. Placement always respects your compute price ceiling. **A different region receives separate empty storage; files are not copied automatically.**

Python accepts the same options: `client.spot.request("h200x1", workspace_id="wsp_...", workspace_slot="default", placement="local_only", local_wait_s=300, max_price=3)`.

Storage bills allocated GiB over retained time, separately from GPU usage, including while no node is running. The ordinary object storage free allowance does not apply. Storage quota and funded credit must cover each allocation; For `429 storage_quota`, an Owner or Admin can raise the available storage limit in **Settings › Quotas**, or release storage that is no longer needed. Stopping compute does not delete retained files or free their storage allocation. **Settings › Compute** separately shows active Spot nodes, simultaneous starts and their editable account limits. Nodes awaiting cleanup remain counted; raising a limit does not guarantee available capacity.

To remove the data, release linked nodes and run `nodus delete workspace spot-training --confirm`. The API requires `DELETE .../workspaces/spot-training?confirm=true`; Python supports `client.api.delete(..., confirm=True)`. Deletion waits for attachments and confirmed storage removal. This is permanent deletion of the Workspace’s regional files, including any model checkpoints saved there. In the console, deletion requires typing the Workspace name and confirming that its files will be lost.

Retained files do not restart a training process. During this pilot your command must write valid checkpoints, load them on restart, and request replacements. Managed checkpoint recovery is a separate next stage.

## Retry initial capacity acquisition from the CLI

Retry is opt-in and bounded. This command permits at most three accepted startup attempts within one 30-minute deadline, including the first attempt. It uses saved SSH keys and temporary boot storage:

Terminal window

```console
nodus spot up h100x8 --max-price 12 --wait --retry-startup --max-attempts 3 --timeout 30m --state ./h100-acquisition.json
```

Only a confirmed `capacity_unavailable` failure **before SSH readiness** can be retried, and only after `cleaned_at` confirms compute cleanup. Each attempt has a new node ID and request key; previous nodes remain in history with zero compute charge. Quotes, limits and funding are checked again. Supply is not guaranteed. The command honors the location, GPU and price constraints; it never falls back to on-demand or reserved compute. Explicit `--ssh-key` public keys remain unchanged. If omitted, each attempt uses the account’s currently saved keys.

Keep the state file and its `.lock` companion. They contain public request and node data, not API credentials, and are private to your user. After an interruption, use the printed resume command with the same API/account:

Terminal window

```console
nodus spot up h100x8 --wait --retry-startup --state ./h100-acquisition.json --resume
```

Resume retains the original deadline and attempt limit. A lost POST response without a saved node ID stops for manual inspection with `nodus spot ls`; the CLI cannot safely resend that request. Timeout or cancellation does not release an accepted node. Inspect it and use `nodus spot rm NODE_ID` when appropriate.

With `--workspace`, retries stay in the first accepted attachment’s location and keep the same slot; files are never moved to another location. Use `--placement local_only` to keep initial placement at the Workspace’s home. There is no retry after a node has ever become ready, and no automatic restart of training or recovery from running-node preemption. The console retains its single-accepted-node behavior. Python waiting uses the separate timeout-bounded behavior below.

## Poll safely from Python

Set `SPOT_WORKSPACE_ID` to the retained Workspace UID, `SPOT_MAX_NODE_PRICE` to your approved price ceiling, and `SPOT_REQUEST_KEY` to a unique identifier for one logical request. Keep the key for inspection and recovery. After an ambiguous initial POST, inspect node history before resending. This example polls every 30 seconds and releases compute after inspecting its GPUs and mount.

examples/spot/poll.py

```python
"""Request one node with retained storage, inspect its GPUs, then release compute."""

import os
import subprocess

import nodus

client = nodus.Client()
node = client.spot.request(
    "h200x8",
    max_price=float(os.environ["SPOT_MAX_NODE_PRICE"]),
    workspace_id=os.environ["SPOT_WORKSPACE_ID"],
    placement="local_only",
    poll=30,
    timeout=1800,
    idempotency_key=os.environ["SPOT_REQUEST_KEY"],
)
try:
    subprocess.run(
        [
            "nodus",
            "spot",
            "ssh",
            node["id"],
            "--",
            "nvidia-smi && nvidia-smi topo -m && df -h /workspace",
        ],
        check=True,
    )
finally:
    client.spot.release(node["id"])
```

`client.spot.request()` waits by default, with a 15-minute timeout. Set `wait=False` for one immediate POST, or set `timeout=1800` for a longer wait. Waiting polls the accepted node. If it fails before readiness with `end_reason: "capacity_unavailable"`, waiting keeps polling until cleanup is confirmed, then retries after `retry_after` seconds with the same key and price ceiling. Retained storage stays in the first accepted location. Other ended nodes, including preemptions after readiness, raise `nodus.errors.SpotNodeEnded`; a timeout raises `nodus.errors.SpotTimeout` with `node` and `idempotency_key` for recovery. Timeout or cancellation does not release the node. Inspect or release it before starting a different request. Async callers use `await client.spot.request.aio(...)`.

Other methods are `client.spot.prices()`, `list()`, `get(node_id)`, `release(node_id)` and `usage(from_="2026-10-24T00:00:00Z", to="2026-10-25T00:00:00Z")`. Results are ordinary JSON dictionaries and lists. Optional schemas are available in `nodus.api.models.spot`.

## Preemption webhooks

In **Settings → Compute → Preemption webhooks**, add your public HTTPS URL and save the signing secret shown once. The form selects `spot.preempted`; existing endpoints subscribed to that type, `spot.*` or all events also receive it. Owners and admins can edit, disable or rotate these endpoints using the same webhook controls.

The signed notification arrives after Nodus observes preemption. Its `data.object` contains the node’s `id`, `gpu`, `status: "preempted"` and `ended_at` billing cutoff. It carries no advance eviction warning and does not restart work. [Verify the signature and deduplicate `webhook-id`](https://www.nodus-compute.ai/docs/guides/webhooks/); failed deliveries retry for up to three days, and each endpoint exposes delivery history and replay controls.

## API and usage

Send your existing bearer token to the platform API. The [Spot OpenAPI document](https://www.nodus-compute.ai/docs/openapi-spot.v1.json) describes the REST schemas. The SSH route additionally uses a WebSocket upgrade:

|Method|Path|Result|
|-|-|-|
|GET|`/v1/spot/prices`|Current node-hour quotes|
|POST|`/v1/spot/nodes`|Accepted node or a capacity/payment refusal|
|GET|`/v1/spot/nodes`|Your nodes|
|GET|`/v1/spot/nodes/{id}`|Status, Nodus SSH command and accrued cost|
|GET|`/v1/spot/nodes/{id}/ssh`|Authenticated binary WebSocket carrying SSH (`spot:write`)|
|DELETE|`/v1/spot/nodes/{id}`|Release and cleanup status|
|GET|`/v1/spot/usage?from=...&to=...`|Per-node price intervals and charges|

Include `Idempotency-Key` or `idempotency_key` in every POST. A retry with that key returns the same node. When a node is preempted, the key becomes available for a replacement. An unstarted capacity failure retains its key until physical cleanup is confirmed. A lost response requires inspecting node history before resending; a terminal state alone is not proof of cleanup. A POST without a key reuses an existing provisioning node of the same GPU type in your account.

`503 no_capacity` means no node was created; retry after `retry_after` seconds using the same key. `503 price_above_max` requires a cheaper quote within your approved ceiling. `402 payment_required` needs action in billing, while `429 node_limit` requires releasing a node or changing your account limit.

An accepted node can still fail before SSH is ready. When capacity failure is confirmed, its JSON includes `startup_failure_code: "capacity_unavailable"` and a plain-language `startup_failure_message`; the console and CLI show that explanation. The node remains `failed` with no compute charge. A retryable failure uses `end_reason: "capacity_unavailable"`; cleanup exposes `retry_after` and `cleaned_at`. Missing diagnostic fields mean the specific cause is unknown. By default the CLI does not replace an accepted failed node. The opt-in initial-acquisition mode waits for confirmed cleanup before a capacity retry; an available quote does not guarantee that the next startup will succeed.

Billing begins when SSH is ready. Price changes appear as separate intervals in usage, and charges appear under **Spot GPU** in the existing billing history. The instance detail page shows accrued cost; posted billing usage updates on the metering cycle. Provisioning failures are unbilled. A billing warning includes a release time so you can save work and restore funding before the node is released.

Organizations approved for invoice terms see their credit limit, outstanding invoices, payment links and due dates in **Usage & billing**. Invoice credits become available when the approved invoice is issued; paying it settles that advance without adding the credits a second time. ACH processing can take several days. Overdue invoices or returned payments pause new launches and can trigger a release warning for running nodes.