knowledge distillation

usually NN produces a class probabilities via softmax layer that converts logits $z_i$ computed for each class into probability $q_i$ by comparing $z_i$ with other logits:

where temperature $T$ is often set to 1.

We “distill” the knowledge by training the base specialist systems with a soft target distribution of the transfer set.

Each case in a transfer sets contributes to cross-entropy gradient $dC/d z_i$ with respects to each logit $z_i$ of the distilled model.

The gradient given by the training done at temperature $T$ that gives soft target probabilities $p_i$ ¹: