Run and compare your agent evaluations

Bring your evaluation harness, task set, model calls and scoring logic. Execute it in an available agent sandbox for interactive tool use, or as a GPU workload for batch evaluation. Nodus runs your code and returns the evidence it produces. You supply the evaluation tasks and scorer.

Make evaluations repeatable

Pin your source revision and runtime, supply the dataset, and set spending and resource limits. Stream logs and inspect command outcomes. Write scores, per-task traces and reports to declared result files. Without explicit result paths, nonempty outputs/ and results/ folders are collected as archives.

Use async SDK submissions for independent experiments, each with its own request key and budget. Limit concurrency in your client and account. An experiment's total spending can include every submitted run. Explicit multi-stage workloads can hand named files between serial preparation, execution and scoring stages.

Compare execution measurements

The benchmark API, SDK and CLI accept your command, an explicit total budget and a matrix of GPU families, batch sizes, allowed regions and repetitions, up to 64 cells. The console displays each cell and links to its workload. Reported cost, wall time, throughput, latency and cost per unit depend on the recorded charges and unit observations. Missing measurements remain unavailable.

Inspect task evidence

For an evaluation associated with an RL environment or a custom RL task, emit task events to show passed, failed and error attempts, reported rewards and optional traces. Export the recorded evidence and download your output files. Scores come from the verifier or scorer you provide. A completed command or a research-catalog listing does not independently validate that scorer.

Run your evaluation · Concurrent experiments · Logs and outputs