Run jobs on your own machines

A pool is a group of your own machines that Nodus schedules work onto before it rents anything. Review the terms shown in the console before enabling paid features.

Pools have four parts:

• Measure: enroll hosts with a one-line installer and see their inventory, utilization and health.
• Route: send jobs to the pool first, wait for it or burst to the market under rules you set.
• Predict: forecasts of demand and free capacity, and recommendations for sizing and wait policies.
• Act: routine actions such as reclaiming idle hosts or draining in a maintenance window, with approvals.

Create a pool

nodus create pool lab --routing prefer

The CLI and the console ask you to agree to the routing fee before the pool is created. In the console, open BYOC → Pools & machines and choose New pool.

Declare a private fabric for distributed work (Beta)

For a new Pool whose hosts share one directly reachable private network, set this declaration when creating it:

spec:
  network:
    mode: Private
  routing:
    mode: Only
    burstToMarket: Deny

You are declaring that every host you enroll belongs to that connected network and accepting unencrypted peer traffic between workers. Nodus does not treat this as provider-verified locality. The declaration cannot be added, changed or removed later; drain and create a new Pool to change networks. Ordinary routing settings remain editable. A Pool without this declaration cannot use Private or Colocated distributed placement.

This declaration alone does not make a distributed job runnable. The deployment needs a measured Private protocol qualification; each group still needs distinct Ready hosts with registered native endpoints, funding and successful peer probes. Isolated grading also needs the sandbox runtime. There is no geographic-region claim, cross-organization sharing or change to Route charges. Hardware qualification is still pending; the field is not a claim that a particular provider or network has passed.

Enroll a host

A host needs Linux on x86-64 or arm64 with a running systemd service manager and cgroup v2 with CPU and memory controllers. It also needs running containerd at /run/containerd/containerd.sock, a healthy overlayfs snapshotter, and ctr, runc, containerd-shim-runc-v2 and timeout on its system path. Follow the containerd setup instructions if these are missing. GPU hosts also need working NVIDIA drivers and the NVIDIA Container Toolkit with CDI.

The installer checks these prerequisites before downloading the agent or consuming the enrollment token. It does not install system packages, reconfigure Docker or restart containerd. Download the installer from the URL in the enrollment command to install.sh, then check the host without enrolling it:

sudo sh ./install.sh --check

A successful check confirms runtime prerequisites, not a completed GPU Job. Create an enrollment token for the pool; the response shows the installer once:

nodus create enrollmenttoken --pool lab

--ttl 2h shortens the token’s life and --label rack=a copies a label onto the node when it enrolls. The token needs no name; the server gives it one.

Run the command it prints as a user who can sudo on the host. It writes the token to a file only you can read, downloads the signed nodusd agent, checks its sha256 checksum, installs it as a systemd service and joins the pool. It also verifies the signature when cosign is installed; --require-signature makes that check mandatory. For a host with limited download access, copy the same release’s nodusd-amd64 or nodusd-arm64 into a local directory and pass --artifact-dir /absolute/path to the saved installer. Include the matching sandbox runtime archive for sandbox execution and the corresponding signature bundles when verifying signatures. The installer checks the same release digests and signatures, and refuses missing files without downloading replacements. The host still needs connectivity to Nodus to enroll and run work. The token works once and expires after 24 hours; the console shows the countdown. Wait for the node to report Ready before submitting work. The installer is for enrollment; upgrading an existing host requires draining its work, replacing the verified agent binary and restarting nodusd.

An enrolled host using the current agent keeps its connection across restarts and can renew its access after being offline for more than 24 hours. Deleting the node revokes that access. Upgrade older agents while their existing access is still valid; an older agent whose access already expired needs a new enrollment.

nodus get nodes
nodus get node/host-1 -o yaml

Each job runs in a network namespace of its own, and nodusd keeps its network rules in the nftables table inet nodus. Outside that table it adds two rules to the host’s firewall, -i ndv+ -j ACCEPT and -o ndv+ -j ACCEPT. They match only its job interfaces, because Docker and ufw drop all forwarded traffic and jobs with open egress could not reach the internet otherwise. With Docker installed they go at the head of the DOCKER-USER chain; without Docker they go at the head of FORWARD, and only when its policy is DROP. Your rules for every other interface are untouched, and inet nodus still blocks private, link-local and metadata addresses for every job. nodusd puts the two rules back if they go missing and removes them when it stops.

A node reports its GPUs, CPU, memory and disk, its utilization and a heartbeat. A node with no heartbeat for 10 minutes shows Offline and the pool’s members get one email about it.

To stop new work on a host without stopping what runs there, drain it; undrain it to take work again. Deleting a node drains it, then revokes its credential:

nodus node drain host-1
nodus node undrain host-1
nodus delete node/host-1

Route jobs to the pool

Name the pool in a job’s placement. With mode: Prefer the pool is used first and the market takes what the pool cannot; with mode: Only work never leaves the pool.

In the console, open your pool and choose Run a job. The job form selects your private pool under Where to run. Review the image, command and resources before launching. Your pool remains private to your organization.

Pool routing supports ordinary Jobs, including indexed jobs, and distributed Jobs through the shared gang scheduler. Managed TrainingJobs use that same route through their compiled Job. Each attempt reserves one whole host, so another attempt waits until that host is released; a GPU attempt sees only the requested devices. Distributed placement needs distinct Ready hosts, a qualified protocol and successful peer probes. Use the Private declaration above for a qualified Private/Colocated path, or Global placement for a qualified Public path. Neither a Pool name nor a dry run establishes live qualification for every topology or training strategy.

GPU jobs wait when the requested devices have existing GPU work, resident GPU memory, or no fresh utilization reading. After Nodus assigns a device, keep other host processes off it until the job ends; the host owner still controls processes started outside Nodus.

apiVersion: nodus.dev/v1
kind: Job
metadata:
  name: train
spec:
  image: nodus/pytorch:2.8-cuda12.8
  command: [python, train.py]
  resources: {gpu: "H100:1"}
  placement: {pool: lab}

The pool’s routing decides what happens when it is full:

Field	Values	Meaning
waitPolicy	Never, Cheaper, Timeout	Go to the market at once, wait while waiting costs less than the market, or wait up to waitTimeout.
burstToMarket	Allow, Approve, Deny	Burst at market rates, wait for an approval, or never burst.

With Approve, a job that would burst shows BurstApprovalRequired until someone approves it:

nodus request approve-burst job/train

In the console, BYOC → Pools & machines → Approvals and the pool’s Approvals section list each waiting job with its estimated hourly cost on Nodus capacity. Allow burst runs it there; Keep waiting leaves it queued for the pool.

Forecast demand

Enable Pool Predict on the pool’s Forecast tab or with spec.predict.enabled: true, after agreeing to its monthly price. Nodus checks for measured GPU utilization in every hour of the latest seven complete days before starting the subscription. CPU samples, offline placeholders, stale history and another pool’s data do not qualify. An account that is still collecting history is refused before charging.

Forecasts refresh hourly for 24 hours, 7 days, 30 days and the pool’s configured spec.predict.horizon; recommendations refresh daily. The configured horizon is rounded down to whole hours. Cloud-spend projections remain separate from these GPU capacity forecasts.

nodus pool utilization lab
nodus pool forecast lab --horizon 7d
nodus pool recommendations lab

The forecast gives demand and free capacity in GPUs per hour with p50 and p90 bands. Recommendations cover right-sizing, idle hosts, maintenance windows, fragmentation and wait-policy tuning. Dismissing one closes it and records your reason. A later run can raise a new recommendation if the conditions persist. Turning Predict off stops the charge at once.

Host utilization history requires an updated nodusd reporting GPU readings. Missing samples and long gaps do not establish idle capacity, and upgrading does not reconstruct past readings. Open a connected cloud account’s Cloud billing and forecast disclosure for cloud-spend projections. Those use your cloud billing history and are separate from Pool Predict capacity forecasts.

Automate actions

spec.act sets the mode and the policies. Every action is a PoolAction you can list, and every executed action is audited.

Mode	What happens
Off	Nothing is proposed.
Shadow	Records what would run and runs nothing.
Propose	Each action waits for approval and expires after 24 hours.
Auto	Runs actions within the policies at once.

nodus get poolactions
nodus pool approve lab-idle-1
nodus pool reject lab-idle-1
nodus pool revert lab-idle-1

Actions only drain and undrain hosts in the pool, adjust its routing within the policy’s bounds, or approve a burst. They never stop or delete your jobs. To stop all actions at once, pause them; the pool shows ActPaused until you resume:

nodus pool pause lab
nodus pool resume lab

See your cloud accounts

To list GPU instances and spend in your AWS or GCP accounts and enroll them into pools, connect them read-only: see Connect your cloud accounts.

Understand pool activity and controls

The pool overview shows ready hosts and enrolled GPUs separately from measured utilization. In Hourly utilization, each square identifies a host and an hour in UTC, including the date. Color indicates the amount of GPU activity. Missing measurements are not idle time; request errors show a retry instead of an empty chart. A device-hour is one GPU for one hour. The Nodus-scheduled device-hours carry the routing fee; your cloud or hardware costs are separate.

Under Routing, choose Use my pool first or Only use my pool. Overflow settings say whether Nodus may use its capacity within the Job budget, must ask you first, or must keep work queued for your own hosts. The summary shows the resulting route; changes apply after Save routing.

Automation separates previewing actions, asking you to approve each action, and automatically running the policies you enabled. Forecasts and recommendations are separate from those permissions. Predict still requires qualified measured history and explicit price consent. Its p50 line is the middle estimate, with shaded ranges showing uncertainty toward p90; forecast capacity is not reserved capacity.