Concepts4
📚TheoryIntermediate
Contrastive Learning Theory
Contrastive learning learns representations by pulling together positive pairs and pushing apart negatives using a softmax-based objective.
#contrastive learning#infonce#nt-xent+12
📚TheoryIntermediate
Mutual Information
Mutual Information (MI) measures how much knowing one random variable reduces uncertainty about another.
#mutual information#entropy#kl divergence+12
📚TheoryIntermediate
Shannon Entropy
Shannon entropy quantifies the average uncertainty or information content of a random variable in bits when using base-2 logarithms.
#shannon entropy#information gain#mutual information+12
📚TheoryIntermediate
Information Theory
Information theory quantifies uncertainty and information using measures like entropy, cross-entropy, KL divergence, and mutual information.
#entropy#cross-entropy#kl divergence+12