How I Study AI - Learn AI Papers & Lectures the Easy Way

Ministral 3

Beginner

Alexander H. Liu, Kartik Khandelwal et al.Jan 13arXiv

Ministral 3 is a new family of small-but-mighty AI language models (3B, 8B, 14B) that learn from a larger model using a step-by-step tutoring method called Cascade Distillation.

#Cascade Distillation#Model pruning#Logit distillation

RePo: Language Models with Context Re-Positioning

Intermediate

Huayang Li, Tianyu Zhao et al.Dec 16arXiv

Large language models usually line words up in fixed order slots, which can waste mental energy and make it harder to find the important parts of a long or noisy text.

#context re-positioning#positional encoding#self-attention

Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs

Intermediate

Xiaoran Liu, Yuerong Song et al.Dec 8arXiv

Big language models use RoPE to remember word order, but it throws away the imaginary half of a complex number during attention.

#RoPE++#Rotary Position Embeddings#Imaginary Attention

Papers3

Ministral 3

RePo: Language Models with Context Re-Positioning

Beyond Real: Imaginary Extension of Rotary Position Embeddings for Long-Context LLMs