Concepts3
📚TheoryIntermediate
Mutual Information
Mutual Information (MI) measures how much knowing one random variable reduces uncertainty about another.
#mutual information#entropy#kl divergence+12
📚TheoryIntermediate
KL Divergence (Kullback-Leibler Divergence)
Kullback–Leibler (KL) divergence measures how one probability distribution P devotes probability mass differently from a reference distribution Q.
#kl divergence#kullback-leibler#cross-entropy+12
📚TheoryIntermediate
Information Theory
Information theory quantifies uncertainty and information using measures like entropy, cross-entropy, KL divergence, and mutual information.
#entropy#cross-entropy#kl divergence+12