Papers18

#knowledge distillation

LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval

Jiajie Jin, Yanzhao Zhang et al.Mar 2arXiv

LaSER teaches a fast search model to “think” quietly inside its hidden space, so it gets the benefits of step-by-step reasoning without writing those steps out as text.

#dense retrieval#chain-of-thought#latent reasoning

Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization

Intermediate

Zeyuan Liu, Jeonghye Kim et al.Feb 26arXiv

This paper teaches a language-model agent to explore smarter by combining two ways of learning (on-policy and off-policy) with a simple, self-written memory.

#EMPO#memory-augmented agents#on-policy learning

jina-embeddings-v5-text: Task-Targeted Embedding Distillation

Intermediate

Mohammad Kalim Akram, Saba Sturua et al.Feb 17arXiv

The paper teaches small AI models to make high‑quality text embeddings by first copying a big expert model (distillation) and then practicing four jobs with special mini‑modules (LoRA adapters): retrieval, similarity, clustering, and classification.

#text embeddings#knowledge distillation#contrastive learning

Weak-Driven Learning: How Weak Agents make Strong Agents Stronger

Intermediate

Zehao Chen, Gongxun Li et al.Feb 9arXiv

Big language models can get stuck after fine-tuning because they become too sure of themselves, so normal training stops helping.

#weak-driven learning#logit mixing#weak agents

Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better

Intermediate

Ji Zhao, Yufei Gu et al.Feb 5arXiv

Big idea: use a small, already-trained model to help a bigger model learn good habits early, so the big one trains faster and ends up smarter.

#Late-to-Early Training#LLM pretraining acceleration#representation alignment

Reinforced Attention Learning

Intermediate

Bangzheng Li, Jianmo Ni et al.Feb 4arXiv

This paper teaches AI to pay attention better by training its focus, not just its words.

#Reinforced Attention Learning#attention policy#multimodal LLM

Rethinking Selective Knowledge Distillation

Intermediate

Almog Tavor, Itay Ebenspanger et al.Feb 1arXiv

The paper studies how to teach a smaller language model using a bigger one by only focusing on the most useful bits instead of everything.

#knowledge distillation#selective distillation#student entropy

Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts

Intermediate

Yingfa Chen, Zhen Leng Thai et al.Jan 29arXiv

This paper shows how to turn a big Transformer model into a faster hybrid model that mixes attention and RNN layers using far less training data (about 2.3B tokens).

#hybrid attention#RNN attention hybrid#linear attention

Grounding and Enhancing Informativeness and Utility in Dataset Distillation

Intermediate

Shaobo Wang, Yantai Yang et al.Jan 29arXiv

This paper tackles dataset distillation by giving a clear, math-backed way to keep only the most useful bits of data, so models can learn well from far fewer images.

#dataset distillation#data condensation#Shapley value

Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment

Intermediate

Yuming Yang, Mingyoung Lai et al.Jan 20arXiv

The paper asks a simple question: Which step-by-step explanations from a teacher model actually help a student model learn to reason better?

#Rank-Surprisal Ratio#data-student suitability#chain-of-thought distillation

Toward Ultra-Long-Horizon Agentic Science: Cognitive Accumulation for Machine Learning Engineering

Intermediate

Xinyu Zhu, Yuzhu Cai et al.Jan 15arXiv

This paper builds an AI agent, ML-Master 2.0, that can work on machine learning projects for a very long time without forgetting what matters.

#Hierarchical Cognitive Caching#cognitive accumulation#ultra-long-horizon autonomy

LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoning

Intermediate

Linquan Wu, Tianxiang Jiang et al.Jan 15arXiv

LaViT is a new way to teach smaller vision-language models to look at the right parts of an image before they speak.

#multimodal reasoning#visual attention#knowledge distillation

1 2