Context Distillation
A technique where the behavior of a model prompted with detailed instructions is distilled into a model that exhibits the same behavior without the instructions.
Why It Matters
Context distillation makes models cheaper to run in production by 'baking in' the system prompt behavior so you do not need to send it with every request.
Example
Training a student model on outputs from a teacher model that was given detailed system instructions, so the student behaves correctly without needing those instructions.
Think of it like...
Like training a new employee so thoroughly that they eventually do not need to reference the manual — the guidelines become second nature.
Related Terms
Knowledge Distillation
A model compression technique where a smaller 'student' model is trained to mimic the behavior of a larger 'teacher' model. The student learns not just correct answers but the teacher's nuanced probability distributions.
System Prompt
Hidden instructions provided to an LLM that define its behavior, personality, constraints, and capabilities for a conversation. System prompts set the rules of engagement before the user interacts.
Fine-Tuning
The process of taking a pre-trained model and further training it on a smaller, domain-specific dataset to specialize its behavior for a particular task or domain. Fine-tuning adjusts the model's weights to improve performance on the target task.
Inference
The process of using a trained model to make predictions on new, previously unseen data. Inference is what happens when an AI model is deployed and actively serving results to users.