Blog Tags GitHub

distributed-inference

5 posts tagged with "distributed-inference"

TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale

提出基于 CXL 共享内存的机架级 KV Cache 系统 TraCT，消除 RDMA 网络跳数，实现 GPU-CXL 直接 DMA 传输，TTFT 降低 9.8x，吞吐量提升 1.6x

llm-serving disaggregated-inference kv-cache cxl shared-memory distributed-inference system-design

Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference

通过架构感知大小调整、六层内存层次和贝叶斯重用预测，实现 KV Cache 的预测性多层内存管理

kv-cache memory-management gpu-inference multi-tier-memory bayesian-prediction attention-architecture mla cxl distributed-inference

Frontier: Towards Comprehensive and Accurate LLM Inference Simulation

面向现代 LLM 推理服务的离散事件模拟器，支持解耦架构、运行时优化和有状态工作负载，吞吐量误差低于 4%

llm-serving simulation distributed-inference performance-modeling disaggregated-serving system-design

Federation of Experts: Communication Efficient Distributed Inference for Large Language Models

通过将 MoE 架构重构为多个独立的专家集群，消除 all-to-all 通信瓶颈，实现高效的分布式推理

llm mixture-of-experts distributed-inference communication-efficiency expert-parallelism

February 22, 2025

AIBrix: Towards Scalable, Cost-Effective Large Language Model Inference Infrastructure

云原生LLM推理基础设施框架，通过协同设计实现高密度LoRA管理、分布式KV缓存和异构GPU调度

inference-serving distributed-inference kv-cache lora autoscaling kubernetes system-optimization