In AI, distillation usually refers to "knowledge distillation" or "model distillation." Its core idea is to use a larger, more capable teacher model to guide a smaller, lighter student model, so the student model learns the teacher model's behavior as much as possible.
The term comes from the everyday metaphor of distillation. Distillation doesn't move the original substance over whole—it refines out the important part. Model distillation works the same way: not copying a large model, but having a small model learn the patterns in the large model's outputs.
Grab It in One Sentence First
Distillation is "refining" a large model's capabilities into a small model, making the small model faster, cheaper, and easier to deploy.
An everyday analogy is a teacher guiding a student. The teacher is very good at solving problems, and the student doesn't have as much experience as the teacher, but by imitating the teacher's problem-solving process, judgment tendencies, and answer distributions, the student can learn more than by only looking at the standard answers.
How Distillation Actually Works
IBM's explanation of knowledge distillation emphasizes that it usually splits training into two roles, the teacher model and the student model. The teacher model is usually larger, stronger, and slower; the student model is smaller, faster, and easier to deploy. During training, the student learns not just the final correct answer but also the teacher model's preferences among different answers.
flowchart LR
Teacher["Teacher model<br/>large and strong"] --> Soft["Output distribution / reasoning examples"]
Data["Training data"] --> Teacher
Soft --> Student["Student model<br/>small and fast"]
Data --> Student
Student --> Deploy["Deploy to real applications"]For example, a question has four options; the standard answer only tells the student which one is correct, while the teacher model's output may also tell the student how close each of the other options is. This "soft information" helps the student learn finer judgment boundaries.
Why Distillation Is Needed
Large models usually perform better, but they have high running costs, large latency, and demanding deployment requirements. Many scenarios don't need to call the strongest model every time—for example, mobile applications, edge devices, fixed internal enterprise tasks, real-time customer service, and high-concurrency systems. Distillation can transfer part of a large model's capabilities to a small model, making real-world use faster and cheaper.
This also explains why distillation often appears together with local models, small models, and model compression. It isn't meant to replace all large models, but to let some capabilities be deployed at lower cost.
Where It's Easy to Misunderstand
Distillation isn't lossless copying. The student model usually gets smaller and faster, but it may also lose some capabilities, especially complex reasoning, long context, and rare tasks. It's more like organizing the teacher's frequently used abilities into a portable version, rather than moving the whole teacher into the small model.
Another misconception is thinking distillation is necessarily compliant. Which teacher model is used, what data is used, and whether it violates the terms of service or intellectual property rules all need to be looked at carefully. Being technically able to do it doesn't mean it can be done in any scenario.
How to Decide Whether to Use It
If you're after the highest quality, using a strong model directly may be more stable. If you need low latency, low cost, offline deployment, mobile operation, or high-frequency calls for a fixed task, distillation is worth considering.
Ordinary users can understand it this way: distillation is a trade-off among quality, speed, and cost. It makes a small model more capable, but it doesn't guarantee a small model becomes a large one.