Concepts10
PAC-Bayes Theory
PAC-Bayes provides high-probability generalization bounds for randomized predictors by comparing a data-dependent posterior Q to a fixed, data-independent prior P through KL(Q||P).
Concentration Inequalities
Concentration inequalities give high-probability bounds that random outcomes stay close to their expectations, even without knowing the full distribution.
Information-Theoretic Lower Bounds
Information-theoretic lower bounds tell you the best possible performance any learning algorithm can achieve, regardless of cleverness or compute.
Variational Inference Theory
Variational Inference (VI) replaces an intractable posterior with a simpler distribution and optimizes it by minimizing KL divergence, which is equivalent to maximizing the ELBO.
ELBO (Evidence Lower Bound)
The Evidence Lower Bound (ELBO) is a tractable lower bound on the log evidence log p(x) that enables learning and inference in latent variable models like VAEs.
Information Bottleneck Theory
Information Bottleneck (IB) studies how to compress an input X into a representation Z that still preserves what is needed to predict Y.
Mutual Information
Mutual Information (MI) measures how much knowing one random variable reduces uncertainty about another.
KL Divergence (Kullback-Leibler Divergence)
Kullback–Leibler (KL) divergence measures how one probability distribution P devotes probability mass differently from a reference distribution Q.
Shannon Entropy
Shannon entropy quantifies the average uncertainty or information content of a random variable in bits when using base-2 logarithms.
Information Theory
Information theory quantifies uncertainty and information using measures like entropy, cross-entropy, KL divergence, and mutual information.