letter-counting-legacy-eval-v1-0-1

Dependency patch revision of legacy letter-counting evaluation protocol: first 64 generated tasks, raw completion scoring.

Field	Value
Reference	nodus/[email protected]
Image	nodus/env-letter-counting-legacy-eval:1.0.1
Publisher	Open-Thought
Category	Reasoning
Readiness	Research
Modes	Evaluate
Reward	Scalar
Held-out measures	TrainedTask
Splits	0 train, 64 test (disjoint by canonical identity)
Licenses	code Apache-2.0, data Apache-2.0
Source	https://github.com/open-thought/reasoning-gym

Grading

Completions are graded by the platform, never by the trainer: the trainer submits {taskId, completion} batches and the verdicts come back as task events.

Grader	Kind	What it checks
score-answer	Program	Raw completion scored by pinned Reasoning Gym; only full credit is Correct

Sample task

One line of nodus-env tasks --split test --seed 0; tasks never carry answers.

{
  "metadata": {
    "family": "letter_counting",
    "systemPrompt": "Reply with the final answer only. Do not restate the question, do not show working, and do not add any words, labels or punctuation around it."
  },
  "prompt": "How many times does the letter \"b\" appear in the text: \"phrase Project Gutenberg associated with or appearing on the work you\"?",
  "taskId": "letter-counting-legacy-eval-v1-0-1:test:0:0"
}

Examples

Each example is a TrainingJob template. A baseline is shown only where the example was measured by running it.

Example	Mode	Runtime	Model	Tasks	Baseline	Trained	Measured on
letter-counting	Evaluate	nodus/evaluate	Qwen/Qwen3-1.7B @ 70d244c	64	not measured	n/a (evaluation)	not measured

From the repository root, preview the generated manifest using your configured API context:

$ nodus create -f examples/training/catalog/letter-counting-legacy-eval-v1-0-1/letter-counting.yaml --dry-run=server -o estimate

Or preview the catalog example with the Python SDK:

import nodus

client = nodus.Client()
nodus.api.set_default_client(client)
job = nodus.TrainingJob.from_example("nodus/letter-counting-legacy-eval-v1-0-1:letter-counting")
print(job.preview().estimate)
client.close()