Papers2

#scheduling

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

Yongtong Wu, Shaoyuan Chen et al.Feb 25arXiv

Agent-style LLMs chat with tools over many short turns, so most tokens are repeats and the system spends more time fetching old memories (KV-Cache) than computing new answers.

#KV-Cache#prefill-decode disaggregation#dual-path loading

VoxServe: Streaming-Centric Serving System for Speech Language Models

Intermediate

Keisuke Kamahori, Wei-Tzu Lee et al.Jan 30arXiv

VoxServe is a new serving system that makes voice AIs respond fast and smoothly when streaming audio to users.

#Speech Language Models#streaming#Time-To-First-Audio