Robust KV Cache Management for LLM Serving under Output Token Length Uncertainty
A robust KV cache management framework using Wasserstein distributionally robust optimization (DRO) to jointly optimize GPU parallelism, cache reservation, request routing, and prefix caching under output token length uncertainty.