ConceptMoE: Adaptive Token-to-Concept Compression for Implicit Compute Allocation
BeginnerZihao Huang, Jundong Zhou et al.Jan 29arXiv
ConceptMoE teaches a language model to group easy, similar tokens into bigger ideas called concepts, so it spends more brainpower on the hard parts.
#ConceptMoE#Mixture of Experts#Adaptive Compression