Nov 13, 2026 - 9 min read

Knowledge distillation trains a small student model to copy a large teacher, either by matching the teacher’s full probability distribution over next tokens or by fine-tuning on text the teacher wrote. On Nodus, the managed nodus/distill runtime (Beta) trains a student on a teacher’s distributions from one manifest, with a dollar cap and a price estimate before the run starts. Which method you pick decides how often you pay for the teacher, and in our worked plan that swings the bill by almost 2x.

There are four families of LLM distillation in common use. Each one answers a different question: what signal does the student learn from, and whose text is it learning on?

What does the student actually learn from the teacher?

In the original method, the student learns the teacher’s soft labels: its whole probability distribution, not just its top answer. Geoffrey Hinton, Oriol Vinyals and Jeff Dean described it in Distilling the Knowledge in a Neural Network (2015). A teacher that puts 70% on “Paris” and 20% on “Lyon” tells the student which wrong answers are nearly right. A one-hot label throws that away.

The trick is temperature. You divide both models’ logits by a temperature T above 1 before the softmax, which flattens the distributions so the small probabilities carry signal. The loss is the KL divergence between the softened teacher and student distributions, usually mixed with ordinary cross-entropy on the true tokens. The paper notes that soft-target gradients shrink as 1/T², so you multiply that term by T² to keep the two losses balanced.

For an LLM, this happens at every token position. The teacher runs a forward pass over the same sequence, and the student matches its distribution over the full vocabulary at each step. That has one hard requirement: both models must share a tokenizer, or the distributions are over different vocabularies.

How is sequence-level distillation different?

Sequence-level distillation skips the logits. The teacher generates complete answers to your prompts, and the student is fine-tuned on those answers as if they were ordinary labelled data. Yoon Kim and Alexander Rush introduced it for translation in Sequence-Level Knowledge Distillation (2016). Their best student ran 10 times faster than its teacher with little loss in quality, and beat a baseline without distillation by 4.2 BLEU with greedy decoding.

This is what most people mean when they say they “distilled” a model from a bigger one. It works with any teacher you can sample from, including a closed API model or one with a different tokenizer. The cost is a weaker signal per token, because the student sees one sampled answer instead of a full distribution.

Why does on-policy distillation exist?

Both methods above train the student on text it did not write. At inference time it generates its own tokens, makes its own early mistakes, and ends up in states it never saw during training. On-policy distillation fixes that mismatch: the student generates, and the teacher scores every token of the student’s output.

Two papers set this up for LLMs. MiniLLM swaps the usual forward KL for reverse KL, which stops the student from overestimating regions the teacher considers unlikely. Generalized Knowledge Distillation (GKD) trains on student-generated sequences with a choice of divergence. Hugging Face TRL ships it as the GKDTrainer. Its lmbda sets the share of on-policy student outputs, and beta moves the loss between forward KL (0.0) and reverse KL (1.0). The TRL docs report that the authors found high lmbda worked better, while the best beta varied by task.

When is hidden-state distillation worth it?

Feature distillation matches internal representations as well as outputs. DistilBERT added a cosine loss on hidden states to the distillation and language-modelling losses, and cut BERT’s size by 40% while keeping 97% of its language understanding. TinyBERT used its own Transformer distillation method and reached more than 96.8% of its teacher on GLUE at 7.5x smaller.

The catch is that you need a mapping between teacher and student layers and hidden sizes. That is natural when you design the student yourself, as both papers did. When you start from an existing small open model, output-level distillation is usually the practical choice.

What does a distillation run cost?

The teacher dominates the bill, and the method decides how many times you pay for it. We priced one plan to show this: Qwen3-32B as teacher, Qwen3-1.7B as student, 100 million tokens of prompts and answers, two epochs of full student training. Both models are Apache 2.0 and share a 151,936-token vocabulary per their Hugging Face pages (32B, 1.7B), which list 32.76B and 2.03B parameters.

We used the standard approximations from Kaplan et al.: a forward pass costs 2 × parameters FLOPs per token, and training costs 6 × parameters. An NVIDIA H100 SXM peaks at about 989 teraFLOPS of dense BF16, and we assume 30% utilization, the same planning figure as our LLM training cost guide. That gives 1.068 × 10¹⁸ useful FLOPs per H100-hour. The H100 SXM is listed at $2.60/hr on the Nodus pricing page.

• Teacher pass over 100M tokens: 2 × 32.76B × 100M = 6.55 × 10¹⁸ FLOPs, or 6.13 H100-hours, $15.95.
• Student training, one epoch: 6 × 2.03B × 100M = 1.22 × 10¹⁸ FLOPs, or 1.14 H100-hours, $2.96.
• Student generation, one epoch (on-policy only): 2 × 2.03B × 100M = 4.06 × 10¹⁷ FLOPs, or 0.38 H100-hours, $0.99.

Totals are computed before rounding. Sequence-level distillation pays the teacher once, to generate the data: $15.95 plus two student epochs at $5.93, so $21.88. Logit distillation on a dataset you already have pays the teacher every epoch: $31.90 plus $5.93, so $37.83. On-policy distillation pays the teacher every epoch and adds student generation: $31.90 plus $7.91, so $39.80.

Teacher compute is most of every bill; only sequence-level distillation pays for it once.

Two caveats keep this honest. Generating tokens one at a time usually runs at lower utilization than training. At 10%, the sequence-level teacher step becomes $47.85 and that method’s total is $53.78. Measure your real tokens per second with a short Sweep before you trust any plan, including this one.

The second caveat is epochs. The gap widens with every pass over the data.

Epochs	Sequence-level	Logit (existing data)	On-policy (GKD)
1	$18.91	$18.91	$19.90
2	$21.88	$37.83	$39.80
3	$24.84	$56.74	$59.71
4	$27.81	$75.65	$79.61

Prices checked November 13, 2026. Our arithmetic at 30% utilization and $2.60 per H100-hour.

Each extra epoch of logit or on-policy distillation buys another full teacher pass.

Caching teacher logits looks like the way out, but full distributions are huge. One token’s distribution over 151,936 entries in 16-bit floats is 303,872 bytes, so 100M tokens would need about 30.4 TB. Teams that cache usually keep only the top few logits per token.

Which distillation method should you pick?

Pick by what you can access from the teacher first, then by budget. This is the rule we use.

1. If the teacher is a closed API or uses a different tokenizer, use sequence-level distillation. It is the only option, so read the provider’s terms first.
2. If you plan several epochs or a large dataset, start with sequence-level distillation. It pays for the teacher once.
3. If the teacher is open, shares the student’s tokenizer, and quality on your task matters more than the last few dollars, use logit distillation. Start at temperature 2.0, the value in the Nodus docs example.
4. If the student drifts on long outputs once it writes its own text, add an on-policy phase with GKD after a first supervised pass.
5. Use hidden-state distillation only when you are designing the student’s architecture yourself.

Whichever you pick, judge the result on a held-out test set from your real workload. A lower distillation loss does not prove the student answers better.

How do you run distillation on Nodus?

Logit distillation is one manifest on Nodus. The training docs show nodus/distill training a student on a teacher’s distributions, with both models pinned to exact Hugging Face commits:

spec:
  runtime: nodus/distill
  model: {uri: hf://Qwen/Qwen3-0.6B@c1899de289a04d12100db370d81485cdf75e47ca}
  teacher: {uri: hf://Qwen/Qwen3-4B@0e9e39f249a16976918f6564b8830bc894c89659}
  data: {volume: chats}
  parameters: {temperature: 2.0}
  resources: {gpu: {type: [h100-sxm-80g], count: 2}}

Add maxCostUSD and run nodus apply -f distill.yaml --dry-run -o yaml to see the estimated cost before anything runs. The first run on a pinned revision imports the weights once into a read-only Volume, so billed GPU time is not spent downloading. If the cap is reached, the run is suspended rather than failed, and raising the cap resumes it from its checkpoint. Set data.validationFraction and parameters.validation to export the checkpoint with the lowest validation loss.

For sequence-level distillation, generate the teacher’s answers in a Job with your own inference script, then fine-tune the student with nodus/sft. The Job pattern from the Jobs guide is nodus run --gpu L4 --image nodus/pytorch -- python hello.py. Swap in your GPU and script, and add --max-cost so generation stops at your budget. If the student then needs adapters rather than full training, our LoRA vs full fine-tuning guide covers that choice.

FAQ

Can I distill from a closed model’s API outputs?

Check each provider’s terms before you do. OpenAI’s Terms of Use forbid using output “to develop models that compete with OpenAI”. Open-weight licenses differ too: the Llama 3.1 license requires a distributed model trained on Llama outputs to start its name with “Llama”.

Is distillation the same as fine-tuning on synthetic data?

Sequence-level distillation is fine-tuning on synthetic data written by one teacher. Logit and on-policy distillation are different: the student learns from the teacher’s probabilities, not only its sampled text.

What temperature should I use?

Any value above 1 softens the teacher’s distribution, and higher values pass more information about unlikely tokens. The Nodus docs example uses 2.0. Treat it as a hyperparameter and pick it on validation data.

How big a gap between teacher and student works?

Published results span wide ratios. Kim and Rush combined distillation with pruning to reach a student with 13 times fewer parameters than its teacher, at a cost of 0.4 BLEU. MiniLLM tested models from 120M to 13B parameters. The GKD authors note that alternative divergences help when the student lacks the capacity to mimic the teacher.

Tags:

• inference
• training
• fine-tuning
• llm

How do you let Cursor's agent run code on a cloud GPU?

What are the best SaladCloud alternatives for GPU workloads?