Concepts174
Category
Complexity Theory
Complexity theory classifies problems by the resources required to solve or verify them, such as time and memory.
Contrastive Learning Theory
Contrastive learning learns representations by pulling together positive pairs and pushing apart negatives using a softmax-based objective.
Bellman Equations
Bellman equations express how the value of a state or action equals immediate reward plus discounted value of what follows.
Attention Mechanism Theory
Attention computes a weighted sum of values V where the weights come from how similar queries Q are to keys K.
Scaling Laws
Scaling laws say that model loss typically follows a power law that improves predictably as you increase parameters, data, or compute.
Universal Approximation Theorem
The Universal Approximation Theorem (UAT) says a feedforward neural network with one hidden layer and a non-polynomial activation (like sigmoid or ReLU) can approximate any continuous function on a compact set as closely as we want.
Minimax Theorem
The Minimax Theorem states that in zero-sum two-player games with suitable convexity and compactness, the best guaranteed payoff for the maximizer equals the worst-case loss for the minimizer.
PAC Learning
PAC learning formalizes when a learner can probably (with probability at least 1βΞ΄) and approximately (error at most Ξ΅) succeed using a polynomial number of samples.
Bias-Variance Tradeoff
The biasβvariance tradeoff explains how prediction error splits into bias squared, variance, and irreducible noise for squared loss.
Game Theory
Game theory studies strategic decision-making among multiple players where each playerβs payoff depends on everyoneβs actions.
Lagrangian Duality
Lagrangian duality turns a constrained minimization problem into a related maximization problem that provides lower bounds on the original objective.
Matrix Calculus
Matrix calculus extends ordinary calculus to functions whose inputs and outputs are vectors and matrices, letting you compute gradients, Jacobians, and Hessians systematically.