Concepts3
📚TheoryAdvanced
Neural Network Expressivity
Neural network expressivity studies what kinds of functions different network architectures can represent and how efficiently they can do so.
#neural network expressivity#depth separation#relu linear regions+12
📚TheoryIntermediate
Universal Approximation Theorem
The Universal Approximation Theorem (UAT) says a feedforward neural network with one hidden layer and a non-polynomial activation (like sigmoid or ReLU) can approximate any continuous function on a compact set as closely as we want.
#universal approximation theorem#cybenko#hornik+12
⚙️AlgorithmAdvanced
Slope Trick
Slope Trick is a technique to maintain a convex piecewise-linear function implicitly using two heaps and a running constant.
#slope trick#convex dp#piecewise linear+11