Loading skill
Install any skill in seconds. Free to start, no credit card required.
Get Started Free →mkurman/knowledge-distillation
Knowledge distillation techniques for model compression: logit-level, feature-level, and relation-based distillation. KD-Lib library and practical workflows for training student models.
Passed 19 of 22 test cases with the skill, and 18 of 22 without it.
1 fixed18 passed either way3 still failing0 broke
Tokens, turns and time describe resource use, not a dollar price.
A test case the skill fixed. Same model, same prompt, with and without the skill.
Which research paper or technical report describes the distillation methodology behind the DistilQwen2.5 compressed model series?
| Checked | Without skill | With skill |
|---|---|---|
| The response references arXiv paper 2504.15027 | Missed | Passed |
The instructions your model reads when the skill loads.
Knowledge distillation transfers knowledge from a larger teacher model to a smaller student model. Combined with pruning and quantization, it forms the critical middle step in the P-KD-Q compression pipeline.
uv pip install kd-lib
from kd_lib import distill
import torch.nn.functional as F
def kd_loss(student_logits, teacher_logits, labels, temperature=4.0, alpha=0.5):
soft_targets = F.softmax(teacher_logits / temperature, dim=-1)
soft_prob = F.log_softmax(student_logits / temperature, dim=-1)
kd = F.kl_div(soft_prob, soft_targets, reduction="batchmean") * (temperature ** 2)
ce = F.cross_entropy(student_logits, labels)
return alpha * kd + (1 - alpha) * ce
| Method | What it transfers | Best For |
|---|---|---|
| Logit distillation | Output probability distribution | Classification, generation |
| Feature distillation | Intermediate hidden states | Transformer layers |
| Relation distillation | Relationships between representations | Structured outputs |
| Self-distillation | Model teaches itself | No teacher needed |
| Online distillation | Teacher & student train jointly | Both models improve |
test cases passed, vs 18 / 22 without
More skills from the same source and nearby in the registry, with their headline benchmark lift.