Concepts4
📚TheoryAdvanced
PAC-Bayes Theory
PAC-Bayes provides high-probability generalization bounds for randomized predictors by comparing a data-dependent posterior Q to a fixed, data-independent prior P through KL(Q||P).
#pac-bayes#generalization bound#kl divergence+12
📚TheoryAdvanced
Information-Theoretic Lower Bounds
Information-theoretic lower bounds tell you the best possible performance any learning algorithm can achieve, regardless of cleverness or compute.
#information-theoretic lower bounds#fano inequality#le cam method+12
📚TheoryAdvanced
Variational Inference Theory
Variational Inference (VI) replaces an intractable posterior with a simpler distribution and optimizes it by minimizing KL divergence, which is equivalent to maximizing the ELBO.
#variational inference#elbo#kl divergence+12
📚TheoryAdvanced
Information Bottleneck Theory
Information Bottleneck (IB) studies how to compress an input X into a representation Z that still preserves what is needed to predict Y.
#information bottleneck#mutual information#variational information bottleneck+12