Guides

# Retries & idempotency

A lost response must never become a second run: retries repeat only failures that say nothing about the request, and an Idempotency-Key binds each submission to one workload.

Client and AsyncClient retry up to max_retries times (default 2, so three attempts) with exponential backoff. “No” means a second attempt would return the same answer.

The key is scoped to tenant and payload — two tenants can use the same string without colliding. Within one tenant:

• Same key, same payload → the original workload, with Idempotent-Replayed: true. Nothing new is scheduled.
• Same key, different payload → 409, IdempotencyConflictError.
• New key → a new workload, even for a byte-identical brief.

>>> import nodus
>>>
>>> def submit(client, key):
...     return client.run(
...         model="7B fine-tune",
...         command=["python", "train.py", "--epochs", "3"],
...         peak_memory_gb=40,
...         expected_runtime_hours=9,
...         budget=600,
...         continuity="checkpointed",
...         idempotency_key=key,
...     )
>>>
# stable in your domain — not random, not a timestamp
>>> key = "train:job_8412:attempt_1"
>>> with nodus.Client() as client:
...     try:
...         wl = submit(client, key)
...     except nodus.APITimeoutError:
...         # the submit may already have landed; the key resolves it
...         wl = submit(client, key)
...     except nodus.IdempotencyConflictError:
...         # bound to a different brief — do not retry
...         raise
...     print(wl.id, wl.status)

Reads (get, list, events, artifacts, ledger) carry no key and repeating them changes nothing; cancel is safe to repeat — a terminal workload stays in the state it reached.

# Multi-stage workloads

Pass stages to run() and the list compiles to a DAG: depends_on names upstream stages, inputs reference their named outputs, and stages with no unmet dependency run concurrently. Unknown names and cycles are rejected with ValidationError at submission.

Each stage declares its own requirements and continuity and is routed on its own terms. budget and finish_by apply to the whole graph, not any one stage.

>>> import nodus
>>>
>>> with nodus.Client() as client:
...     wl = client.run(
...         model="7B fine-tune",
...         image="ghcr.io/acme/trainer:2026.07",
...         budget=900,
...         data_regions=["us"],
...         stages=[
...             nodus.StageSpec(
...                 name="prepare",
...                 command=["python", "-m", "pipeline.prepare", "--shards", "64"],
...                 peak_memory_gb=16,
...                 expected_runtime_hours=1.5,
...                 continuity="restartable",
...                 outputs=[dict(name="dataset", path="/out/shards")],
...             ),
...             nodus.StageSpec(
...                 name="train",
...                 command=["python", "train.py", "--data", "/in/dataset"],
...                 peak_memory_gb=40,
...                 expected_runtime_hours=9,
...                 continuity="checkpointed",
...                 interrupt_tolerance="high",
...                 depends_on=["prepare"],
...                 inputs=[dict(source="prepare.dataset", path="/in/dataset")],
...                 outputs=[dict(name="weights", path="/out/ckpt")],
...             ),
...             nodus.StageSpec(
...                 name="eval",
...                 command=["python", "eval.py", "--weights", "/in/weights"],
...                 peak_memory_gb=24,
...                 expected_runtime_hours=2,
...                 continuity="restartable",
...                 total_units=512,
...                 depends_on=["train"],
...                 inputs=[dict(source="train.weights", path="/in/weights")],
...             ),
...         ],
...     )
...     done = wl.wait()
...     for stage in done.stages:
...         print(stage.id, stage.status, stage.completed_units, "/", stage.total_units)
#
# stg_prepare  completed  None / None
# stg_train    completed  None / None
# stg_eval     completed  512 / 512

Progress reads from workload.stages, a list of StageRun; which stage and attempt produced which manifest, and where the digests live, is documented at workload.artifacts() in the reference.

# Runtime contract

Nodus runs your argv as a real process on the capacity it routed to, watches its exit status, and treats any non-zero exit as a failed workload. Everything the process needs to know about its lease arrives in environment variables — argv belongs to you.

HOME, TMPDIR and PWD point inside your lease; the rest of the environment is deny-by-default — only PATH, LANG, LC_ALL and TZ pass through. Anything else must come from the image or the brief.

$ set -u

# a declared input must be a readable, non-empty file
[ -s "$NODUS_INPUT_dataset" ] || { echo "no dataset" >&2; exit 3; }

# only your process can prove a restore carried prior local state
if [ "${NODUS_RESTORED:-0}" = "1" ] && [ ! -s cursor ]; then
    echo "restored from $NODUS_RESTORED_FROM with no prior state" >&2
    exit 4
fi

# resume at the cursor, not at zero
u=${NODUS_CURSOR_COMPLETED:-0}
total=${NODUS_TOTAL_UNITS:-0}
while [ "$u" -lt "$total" ]; do
    process_shard "$u"
    u=$((u + 1))

    # absolute count first — Nodus reads the number, not the lines;
    # a rerun would over-report and the next generation would skip work
    printf '%s units of %s done\n' "$u" "$total" >> "$NODUS_PROGRESS"

    # write-then-rename: a snapshot never sees a partial file
    printf '%s\n' "$u" > cursor.tmp && mv cursor.tmp cursor
done

# outputs last — collected only after a clean exit
cp weights.bin "$NODUS_OUTPUT_model"
exit 0

Exit status is the contract. Exit 0 triggers the final snapshot that completes the stage and publishes your declared outputs; any other status fails the workload — status recorded, no final manifest. stdout and stderr are captured per generation.

# Versioning

SDK and control plane follow SemVer and version independently; the wire contract is the boundary. Pin the SDK to a major at minimum (nodus-sdk>=1.0,<2); the control-plane pin is the path segment /v1, and a breaking wire change arrives as a new segment beside the old, never as a change to an existing one.

Within a major, wire changes are additive — new fields appear on responses, older SDKs ignore them — and a webhook handler that rejects unknown keys breaks on a change designed to be safe. Enumerations are open sets, new WorkloadStatus values and event types included: keep a default branch, and prefer workload.is_terminal and workload.succeeded over enumerating terminal statuses.

# Billing & budgets

Billing is for execution: the capacity a workload consumed, recovery included — not seats, reservations, or capacity evaluated but unused. The unit is the hour a route was held, at that nodus:… route's rate. Pricing is per account during the pilot — no public price list; rates are agreed before your first billable run.

The budget is a cap, not an estimate. budget becomes budget_usd and bounds the whole graph. Planning checks it against expected cost to completion, so a brief that cannot finish inside the cap is rejected up front with BudgetExceededError and a 402 — a decision the SDK does not retry. The cap holds mid-run: routing reserves recovery headroom before reserving a machine, and a resumable workload nearing the cap gets to commit a checkpoint, then settles FAILED with an error saying so. You pay for what was consumed; anything committed first stays in workload.artifacts().

>>> with nodus.Client() as client:
...     wl = client.get("wl_9f3c1b2a")
...     print(f"{wl.spend_usd:.2f} of {wl.budget_usd:.2f} budgeted")
...     for entry in wl.ledger():
...         print(entry.type, entry.debit_usd, entry.credit_usd)
164.20 of 180.00 budgeted

Cancelling is safe at any time but does not undo work already done: cancel lets a run commit its checkpoint before settling, the hours up to that moment are billed, and everything after is saved.