Topology-Aware Data Movement for Disaggregated GPU Inference
TopKV — 拓扑感知的KV缓存传输编排器，利用GPU互连层次结构将KV缓存传输延迟降低3-18倍
All tags
disaggregated-inference
5 posts tagged with "disaggregated-inference"
TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure
首个将张量生命周期管理从计算栈中解耦的分布式服务层，通过 Artifact/Operation/Plan/Signal 四抽象实现 TaaS 范式，集成 vLLM/SGLang 后权重物化加速 60×、KV 缓存共享 TTFT 降低 87.5%
HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory
Cost-Efficient Remote KV-Cache Tier Using CXL-Hybrid Memory for Scalable LLM Serving
FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling
FlowKV 提出低成本低延迟的 KV Cache 传输方案和负载感知调度器，实现分离式 LLM 推理框架的 96% 传输延迟降低
TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
提出基于 CXL 共享内存的机架级 KV Cache 系统 TraCT，消除 RDMA 网络跳数，实现 GPU-CXL 直接 DMA 传输，TTFT 降低 9.8x，吞吐量提升 1.6x