Arbitrage: Efficient Reasoning via Advantage-Aware Speculation
IntermediateMonishwaran Maheswaran, Rishabh Tiwari et al.Dec 4arXiv
ARBITRAGE makes AI solve step-by-step problems faster by only using the big, slow model when it is predicted to truly help.
#speculative decoding#step-level speculative decoding#advantage-aware routing