Groups
Category
Level
Cross-entropy measures how well a proposed distribution Q predicts outcomes actually generated by a true distribution P.
KL divergence measures how much information is lost when using model Q to approximate the true distribution P.
Mutual Information (MI) measures how much knowing one random variable reduces uncertainty about another.
Shannon entropy quantifies the average uncertainty or information content of a random variable in bits when using base-2 logarithms.