Distillation trains a small “student” model to copy a larger “teacher” model. It is how many compact models get much of a frontier model's skill at a fraction of the running cost. This course explains how it works, who allows it, what it is good for and where it fails. It stays factual: each claim links to its source.
Start here · 2 min 30 s
Watch: distillation, explained simply
A short animated overview of the course: teacher and student, soft targets, the six steps, the rules and the limits. The video has spoken narration and on-screen captions.
Explainer video: distillation-explainer.mp4 (2 min 30 s), next to this page.
MODULE 01 OF 10 · CONCEPTS
What distillation is
Distillation is a training method. A large, capable model (the teacher) produces outputs. A smaller model (the student) trains on those outputs. The goal is a student that behaves like the teacher but is cheaper and faster to run.
Anthropic describes it as “training a less capable model on the outputs of a stronger one” and calls it “a widely used and legitimate training method.” (Anthropic, Feb 2026)
The idea is older than large language models. In 2006, Bucilă, Caruana and Niculescu-Mizil compressed large model ensembles into small, fast models with little accuracy loss (Model compression, KDD 2006). In 2015, Hinton, Vinyals and Dean named and formalized the method (Distilling the Knowledge in a Neural Network).
Teacher
The larger model whose behavior the student copies.
Student
The smaller model that trains on the teacher's outputs.
Logits / token probabilities
The scores a model gives every possible next token before it picks one.
Synthetic data
Training examples that a model generated, not a person.
MODULE CHECKPOINT
Mark this module after you can define teacher and student.
MODULE 02 OF 10 · CONCEPTS
Soft targets: why probabilities teach more than answers
A normal training label says only which answer is right. A teacher's probabilities say more. They show which wrong answers are close and which are absurd. Hinton et al. call these probability outputs “soft targets” (paper).
A setting called temperature (T) controls how soft the targets are. The model divides its scores by T before it turns them into probabilities. A higher T spreads the probability across more answers. In the original method, the student trains at the same high T, then runs at T = 1 (paper, section 2).
Try the temperature
The capital of Spain is ___
Madrid
0.0%
Barcelona
0.0%
Seville
0.0%
Paris
0.0%
banana
0.0%
Illustrative numbers, not from a real model.
MODULE CHECKPOINT
Mark this module after you can explain what temperature changes.
MODULE 03 OF 10 · CONCEPTS
Three ways to distill a language model
The methods differ in what the student sees from the teacher, and how much access you need.
Method
What the student learns from
Teacher access needed
Published examples
Logit (white-box) distillation
The teacher's probability for every possible next token
The teacher's weights or full logits
Gemma 2 2B and 9B used distillation instead of plain next-token prediction. Llama 3.2 1B and 3B used logits from Llama 3.1 8B and 70B as token-level targets.
Sequence-level (black-box) distillation
Complete answers the teacher wrote, used for supervised fine-tuning
Only the teacher's text output, for example through an API
On-policy methods address a mismatch. In sequence-level distillation, the student only ever sees the teacher's text in training, but it must continue its own text at run time. GKD trains on the student's “self-generated output sequences” with teacher feedback to close that gap (Agarwal et al.).
MODULE CHECKPOINT
Mark this module after you can distinguish the three methods.
MODULE 04 OF 10 · CONCEPTS
How distillation differs from related techniques
These terms often appear together. They solve different problems, and a project can combine several.
Technique
What it does
Relation to distillation
Fine-tuning
Continues training a model on a dataset with next-token loss (TRL SFTTrainer)
Sequence-level distillation is fine-tuning in which a teacher wrote the targets.
LoRA / QLoRA
Trains small adapter matrices instead of all weights; QLoRA does this on a quantized base model (MLX LoRA guide)
A low-cost way to run a distillation fine-tune. CyberQwen used QLoRA.
Quantization
Stores the same model's weights at lower numeric precision
No teacher and no new model. It can be applied to a distilled student afterwards.
DeepSeek releases six distilled students (Qwen2.5 and Llama bases) fine-tuned on 800,000 samples curated with DeepSeek-R1. The same month, OpenAI says it is “reviewing indications” that DeepSeek “may have inappropriately distilled” its models. DeepSeek-R1 · Axios
Oct 2025
Thinking Machines Lab publishes on-policy distillation results. Blog
Feb 2026
Anthropic reports “distillation attacks”: over 16 million exchanges through about 24,000 fraudulent accounts, attributed to three AI labs. Anthropic
Oct 2026
Anthropic describes its defenses against distillation attacks. Anthropic
MODULE CHECKPOINT
Mark this module after you have traced the method from 2006 to 2026.
MODULE 06 OF 10 · CONTEXT
What distillation is good for
Lower running cost and latency
A smaller model needs less compute for each answer. DistilBERT runs 60% faster than BERT. arXiv
Lower training cost
Minitron used up to 40× fewer training tokens per model than training from scratch. arXiv Thinking Machines reports on-policy distillation reached a higher AIME'24 score than reinforcement learning with about one tenth of the GPU hours, citing the Qwen3 report. Blog
On-device and local use
Meta positions Llama 3.2 1B and 3B for edge and mobile devices. A model that runs locally can keep data on the device. Meta
Specialization
A small student trained for one task can beat a much larger general model on that task. Distilling step-by-step
Strong open small models
The DeepSeek-R1 distilled Qwen 32B scored 72.6 on AIME 2024 and 94.3 on MATH-500, as reported by DeepSeek. Model card
Teaching and research
A distillation run is small enough for a laptop or one GPU, so it is a practical way to learn the full train–evaluate loop. The CyberQwen record ran on an 8 GB laptop GPU. Field guide
MODULE CHECKPOINT
Mark this module after you can name a suitable use case.
MODULE 07 OF 10 · CONTEXT
Who allows distillation, and who does not
The method is legal and common. The question is whose outputs you use. Closed frontier providers restrict the use of their outputs to build competing models. Many open-weight models use licenses that permit it, sometimes with conditions.
Closed frontier providers (restrict it)
Provider
What the terms say
Source
OpenAI
Prohibits using output “to develop models that compete with OpenAI.”
Outputs may improve other models, but you must display “Built with Llama” and start the model name with “Llama.” Companies above 700 million monthly active users need a separate license.
Enforcement so far is mostly technical and contractual. Providers detect and ban accounts (Anthropic, Oct 2026). One law-firm analysis notes that copyright protection for machine output is weak and that trade-secret claims are unsettled (Winston & Strawn).
MODULE CHECKPOINT
Mark this module after you know which terms to check before a run.
MODULE 08 OF 10 · TRADE-OFFS
Limits and risks
Strengths
Limitations
Smaller, faster, cheaper model to run
The student usually trails the teacher, and a very large size gap between them hurts results (Mirzadeh et al.)
Can run locally or on-device
Copying style is easier than copying capability (Gudibande et al.)
Can specialize on one domain
Errors in teacher outputs become training data
Black-box method needs only API text
Black-box use of closed models conflicts with their terms (module 07)
Low-cost experiments with LoRA/QLoRA
Safety behavior can weaken during fine-tuning (Qi et al.)
The imitation gap. Gudibande et al. found that models trained to imitate ChatGPT copied its style, but on targeted tests they closed “little to none of the gap.” Human raters could miss this because the style looked confident. (arXiv)
Inherited errors. A student learns the teacher's mistakes as if they were correct. The Vicuna authors list weak reasoning and math, factual errors and no safety tuning as limits. (LMSYS) Review teacher answers before training.
Model collapse. Repeated training on model-generated data causes “irreversible defects,” and rare cases at the tails of the data disappear. (Shumailov et al., Nature 2024)
Safety loss. Qi et al. removed GPT-3.5 Turbo's safety behavior with 10 adversarial fine-tuning examples costing under US$0.20. Fine-tuning on harmless data also reduced safety. (arXiv) Anthropic notes that illicitly distilled models are unlikely to keep the teacher's safeguards. (Anthropic)
Evaluation traps. Test questions that leak into training data inflate scores (arXiv). A judge model that was also the teacher may favor answers in its own style. Use a sealed test set, a judge from another model family and a human review sample.
Legal and terms risk. See module 07.
MODULE CHECKPOINT
Mark this module after you can name the main evaluation and safety risks.
MODULE 09 OF 10 · PRACTICE
Run a distillation project, step by step
This is the sequence-level workflow, the most accessible one. Each step links to the matching chapter of the CyberQwen field guide.