Post-training
More ways to post-train a model
Use preference data, reward labels or a frozen teacher to adapt a language model. New run exposes the managed objectives supported by the deployment, alongside Custom code for your own post-training implementation.
Preference and reward methods
- DPO and ORPO use a prompt with preferred and rejected answers.
- KTO uses a prompt, completion and a desirable/undesirable label. Managed validation requires both feedback classes and a batch size of at least two.
- Reward modeling trains a scalar scoring model from preferred/rejected pairs and exports the trained score head with the model or adapter.
- Offline knowledge distillation trains a student using supplied examples, target answers and a frozen teacher. Pin the teacher revision and adjust its temperature and loss weight. Teacher and student must have compatible tokenizers, vocabulary and chat templates, with memory for both models.
Review, tune and compare
Choose compatible LoRA or full-weight settings, precision, quantization and hyperparameters. Preference methods expose beta. Dataset imports, column mapping, sample formats and pre-launch validation use the same training composer as fine-tuning. Prompt grouping keeps the same prompt out of both training and held-out evaluation.
Set the run's spending limit, inspect its workload specification, then launch. Results include the exported model or adapter, loading instructions, exact configuration, split identities and baseline/final held-out measurements. Recovery uses the managed trainer's saved state for these supported recipes.
The distillation recipe uses a frozen local teacher on the supplied dataset. For on-policy generation, remote teachers, custom rewards, PPO or other RLHF programs, supply your own implementation. Available managed GRPO workflows are covered in RL below.