Papers2

#Muon

On Surprising Effectiveness of Masking Updates in Adaptive Optimizers

Taejong Joo, Wenhan Xia et al.Feb 17arXiv

The paper finds a simple trick—randomly skipping some parameter updates—can train large language models better than fancy optimizers.

#Magma#random masking#adaptive optimizers

Nested Learning: The Illusion of Deep Learning Architectures

Intermediate

Ali Behrouz, Meisam Razaviyayn et al.Dec 31arXiv

The paper introduces Nested Learning, a new way to build AI that learns in layers (like Russian dolls), so each part can update at its own speed and remember different things.

#Nested Learning#Associative Memory#In-Context Learning