Concepts4
📚TheoryAdvanced
Transformer Theory
Transformers map sequences to sequences using layers of self-attention and feed-forward networks wrapped with residual connections and LayerNorm.
#transformer#self-attention#positional encoding+12
📚TheoryIntermediate
KL Divergence (Kullback-Leibler Divergence)
Kullback–Leibler (KL) divergence measures how one probability distribution P devotes probability mass differently from a reference distribution Q.
#kl divergence#kullback-leibler#cross-entropy+12
📚TheoryIntermediate
Shannon Entropy
Shannon entropy quantifies the average uncertainty or information content of a random variable in bits when using base-2 logarithms.
#shannon entropy#information gain#mutual information+12
📚TheoryIntermediate
Information Theory
Information theory quantifies uncertainty and information using measures like entropy, cross-entropy, KL divergence, and mutual information.
#entropy#cross-entropy#kl divergence+12