Groups
Category
Mixed precision training stores and computes tensors in low precision (FP16/BF16) for speed and memory savings while keeping a master copy of weights in FP32 for accurate updates.
Gradient clipping limits how large gradient values or their overall magnitude can become during optimization to prevent exploding updates.