Papers2

All Beginner Intermediate Advanced

All Sources arXiv

#Perplexity reduction

Progressive Residual Warmup for Language Model Pretraining

Intermediate

Tianhao Chen, Xin Xu et al.Mar 5arXiv

Training big Transformers can wobble at the start because every layer tries to learn all at once.

#Progressive Residual Warmup#ProRes#Transformer training stability

Not triaged yet

Fast-weight Product Key Memory

Intermediate

Tianyu Zhao, Llion JonesJan 2arXiv

The paper introduces Fast-weight Product Key Memory (FwPKM), a memory layer that can quickly learn from the current text it reads, not just from past training.

#Fast-weight memory#Product Key Memory#Sparse retrieval

Not triaged yet