Kullback-Leibler divergence

denoted as $D_{\text{KL}}(P \parallel Q)$

definition

The statistical distance between a model probability distribution $Q$ difference from a true probability distribution $P$:

$$D_{\text{KL}}(P \parallel Q) = \sum_{x \in \mathcal{X}} P(x) \log (\frac{P(x)}{Q(x)})$$

Alternative form ¹:

For relative entropy if $\forall x > 0, Q(x) = 0 \implies P(x) = 0$ absolute continuity

For distribution $P$ and $Q$ of a continuous random variable, then KL divergence is:

where $p$ and $q$ denote probability densities of $P$ and $Q$