Hey! I’ve been doing some research around model compression lately, mostly focused on knowledge distillation (logit-based distillation specifically).
The research was mostly exploratory: I was looking at how to choose distillation hyperparameters, how architectural differences between teacher and student affect the process, and how we can get more insight into why distillation works. I experimented with different ResNet architectures and compression levels (quantization and 90% pruning) for simple image classification on CIFAR-10/100. I’m attaching a screenshot from my technical report for reference.
It gave me a good understanding of the topic and raised some interesting research questions, but I realized that I’m a little detached from the actual industry use cases and engineering problems where knowledge distillation is useful.
So I’d really like to hear your experience with this. What kind of problem led you to consider knowledge distillation, how did you approach it, and what did you find difficult or non-obvious along the way? I’d especially appreciate hearing about real engineering challenges, unexpected trade-offs, or situations where you wished you had more insight into what was happening during the distillation process.
Any personal experience, thoughts, or examples would be greatly appreciated. Thanks in advance for sharing!
Two things that surprised me when I distilled vision models:
The temperature is doing double duty. High temperature softens the teacher’s logits so the student learns the dark knowledge — which wrong classes look plausible, not just the right answer. That inter-class similarity is the actual transfer; hard labels never carry it. Too hot and it all becomes uniform noise the student memorizes. Somewhere around T=3-4 is usually where the dark knowledge is legible without washing out.
The student inherits the teacher’s calibration, not just its accuracy. If the teacher is overconfident (most are), the student comes out overconfident too — and no serving-time softmax temperature fixes what the training temperature already baked in. Worth checking ECE on the student, not just accuracy, or the “smaller model” quietly becomes a confidently-wrong one.
The non-obvious one from my side: distilling across architectures hurts more than the papers suggest. ResNet-18 from a ResNet-50 teacher works because the inductive biases rhyme. Cross-family (CNN teacher, ViT student) needs feature-level hints, not just logits — the logit surface alone doesn’t carry enough of the “how to see” part.
Your pruning-then-distilling combo is interesting — recovering into the sparse structure usually beats distilling first and pruning after, because the student gets to learn the teacher’s behavior on the exact structure it will serve with.
