Blog Tags GitHub

memory-management

5 posts tagged with "memory-management"

Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning

Prism 通过 GPU 内存气球技术实现多 LLM 共享服务，统一时间共享和空间共享，SLO 达标率下支持 2.3-3.5x 更多请求，已在 10K+ GPU 生产环境部署

llm-serving inference-serving gpu-sharing memory-management multi-model-serving

Efficient Memory Management for Large Language Model Serving with PagedAttention

通过 PagedAttention 算法实现高效 KV cache 内存管理，构建 vLLM 高吞吐量 LLM 服务系统

llm-serving memory-management paged-attention kv-cache system-design production-system vllm

Scalable Processing-Near-Memory for 1M-Token LLM Inference

CXL-Enabled KV-Cache Management Beyond GPU Limits

llm-inference kv-cache processing-near-memory cxl memory-management hardware-acceleration long-context

HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing

提出 CPU-GPU 混合注意力框架 HybridGen，通过注意力 logit 并行化、反馈调度器和语义感知 KV Cache 映射，实现长上下文 LLM 推理的高效协同计算

llm-inference kv-cache cpu-gpu-hybrid memory-management cxl attention-parallelism long-context

Predictive Multi-Tier Memory Management for KV Cache in Large-Scale GPU Inference

通过架构感知大小调整、六层内存层次和贝叶斯重用预测，实现 KV Cache 的预测性多层内存管理

kv-cache memory-management gpu-inference multi-tier-memory bayesian-prediction attention-architecture mla cxl distributed-inference