Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs
IntermediateXiaoran Liu, Yuerong Song et al.Dec 8arXiv
Big language models use RoPE to remember word order, but it throws away the imaginary half of a complex number during attention.
#RoPE++#Rotary Position Embeddings#Imaginary Attention