Instruction Anchors: Dissecting the Causal Dynamics of Modality Arbitration
IntermediateYu Zhang, Mufan Xu et al.Feb 3arXiv
The paper asks a simple question: when an AI sees a picture and some text but the instructions say 'only trust the picture,' how does it decide which one to follow?
#multimodal instruction following#modality arbitration#instruction tokens