Papers4

All Beginner Intermediate Advanced

All Sources arXiv

#audio-visual reasoning

OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models

Intermediate

Yue Ding, Yiyan Ji et al.Feb 4arXiv

OmniSIFT is a new way to shrink (compress) audio and video tokens so omni-modal language models can think faster without forgetting important details.

#Omni-LLM#token compression#modality-asymmetric

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

Intermediate

Sangyun Chung, Se Yeon Kim et al.Jan 29arXiv

Multimodal AI models can mix up what they see and what they hear, making things up across senses; this is called cross-modal hallucination.

#multimodal large language models#cross-modal hallucination#contrastive decoding

FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

Intermediate

Qian Chen, Jinlan Fu et al.Jan 20arXiv

FutureOmni is the first benchmark that tests if multimodal AI models can predict what happens next from both sound and video, not just explain what already happened.

#multimodal LLM#audio-visual reasoning#future forecasting

JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation

Intermediate

Kai Liu, Jungang Li et al.Dec 28arXiv

JavisGPT is a single AI that can both understand sounding videos (audio + video together) and also create new ones that stay in sync.

#multimodal large language model#audio-video synchronization#SyncFusion