Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
IntermediateZeyuan Allen-ZhuDec 19arXiv
The paper introduces Canon layers, tiny add-ons that let nearby words share information directly, like passing notes along a row of desks.
#Canon layers#horizontal information flow#transformer architecture