Large Language Models
Knowledge Distillation
Large Language Models· Advanced
Definition
A model compression technique where a smaller student model is trained to mimic the outputs of a larger teacher model. Distillation transfers knowledge while dramatically reducing model size and inference cost — enabling deployment on resource-constrained hardware. Key technique behind efficient small LLMs like DistilBERT.
Tags
#compression#efficiency#student-teacher#inference#deployment
MS
Maxx Stacks Editorial
Reviewed by enterprise AI practitioners
Maxx University
Keep learning. Keep building.
250+ terms. 5 learning paths. AI maturity assessment. Jargon translator. All free, always.