Papers2

#Doubly Stochastic Matrix

KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices

Wuyang Zhou, Yuxuan Gu et al.Jan 29arXiv

Hyper-Connections (HC) make the usual single shortcut in neural networks wider by creating several parallel streams and letting the model mix them, but this can become unstable when stacked deep.

#Hyper-Connections#Manifold-Constrained Hyper-Connections#Doubly Stochastic Matrix

mHC: Manifold-Constrained Hyper-Connections

Intermediate

Zhenda Xie, Yixuan Wei et al.Dec 31arXiv

The paper fixes a stability problem in Hyper-Connections (HC) by gently steering the network’s mixing matrix onto a safe shape (a manifold) where signals don’t blow up or vanish.

#Residual Connections#Hyper-Connections#Manifold Projection