This paper finds a precise way to describe and fix the Modality Gap, which is when image and text features that mean the same thing still sit in different places in the AI’s memory space.

#Modality Gap#Multimodal Large Language Models#Contrastive Learning

OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation

Intermediate

Letian Zhang, Sucheng Ren et al.Jan 21arXiv

OpenVision 3 is a single vision encoder that learns one set of image tokens that work well for both understanding images (like answering questions) and generating images (like making new pictures).

#Unified Visual Encoder#VAE#Vision Transformer

Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understanding

Intermediate

Yuqing Li, Jiangnan Li et al.Dec 19arXiv

Humans keep a big-picture memory (a “mindscape”) when reading long things; this paper teaches AI to do the same.

#Retrieval-Augmented Generation#Mindscape#Hierarchical Summarization