A Photonic-CXL Memory Appliance for Scalable KV Cache Management in LLM Inference
Marvell Photonic Fabric Memory Appliance — 用无源光纤全互连拓扑替代电交换机，实现32TB共享DDR5内存的CXL内存池化
All tags
memory-efficiency
34 posts tagged with "memory-efficiency"
HyMCache: A KV Cache Framework for Multi-Turn LLM Serving with CXL-Hybrid Memory
CXL-Hybrid Memory-based remote KV-cache tier for scalable multi-turn LLM serving
[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows
AAFLOW+系统将KV缓存作为一等分发系统对象，实现零拷贝的多智能体工作流执行状态共享
FastTPS: An Optimized Method for LLM Token Phase for AI Accelerators
面向AI加速器的LLM Token阶段优化框架FastTPS，包括全局KV缓存管理、序列维度扁平化、Fusion MLP等技术
FlashAccel: Leveraging High-Bandwidth Flash for High-Throughput LLM Inference
面向 LLM 推理的 HBF（高带宽闪存）与 HBM GPU 软硬件协同设计：通过架构级 SRAM 缓存/预取、专用数据布局与异构存储管理层，将 HBF 的容量优势转化为吞吐与能效。
TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation
TriRoute — 单一轻量控制器联合决策注意力分辨率、FFN 专家选择与 KV-Cache 位宽，端到端可训练，单一预算旋钮扫出 Pareto 前沿，缓解跨轴路由坍缩级联
LIMINAL: Exploring The Frontiers of LLM Decode Performance
An analytical performance model (LIMINAL) that abstracts application requirements and hardware capabilities to systematically explore LLM inference performance limits across current, near-future, and hypothetical hardware
ACTA: Automatic Configuration of the Tensor Memory Accelerator for High-End GPUs
A software library that heuristically determines optimal tile sizes and queue configurations for TMA-based GPU kernels using the GPU Specification Table (GST) and Little's Law
Orchestrated Scheduling and Prefetching for GPGPUs
A prefetch-aware (PA) warp scheduling policy that coordinates thread scheduling and data prefetching in GPGPUs to better tolerate long memory latencies
TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention
Position persistent sparse attention leveraging spatial coherence of high-attention tokens across Transformer layers for 2.1× decoding speedup
FP8 P-Cast Precision in Attention: Sink-Induced Collapse and the Optimality of S=2^8
A quantitative analysis of FP8 E4M3 attention P-casting revealing P-collapse under attention sink and characterizing S=256 as the optimal static scaling factor
DreamX-World 1.0: A General-Purpose Interactive World Model
A general-purpose interactive text/image-to-video world model supporting camera navigation, scene revisits, and promptable events with 16 FPS streaming on 8x RTX 5090
强化学习基础设施(RL Infra)优化必读论文与技术全景
覆盖分布式RL、RLHF系统、环境加速、采样优化、通信优化、内存优化等方向
Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
面向资源受限环境的MoE模型高效推理系统，通过智能CPU-GPU协作实现最优执行策略
Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
多模型LLM调度的实证研究，分析CPU-GPU卸载和抢占的性能影响，为下一代调度系统提供设计指导
ES-MoE: Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model
通过专家卸载和动态放置实现MoE模型训练的高效扩展，支持67倍专家扩展和17.5倍吞吐量提升
Minuet: Accelerating 3D Sparse Convolutions on GPUs
面向GPU的3D稀疏卷积加速引擎
MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs
基于CPU-GPU-I/O流水线调度的高吞吐量MoE推理系统
Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
通过细粒度专家卸载优化MoE大模型服务的延迟-内存权衡
FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference
通过灵活高效的卸载打破内存约束实现设备端LLM推理
MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching
基于模块级批处理的单GPU高吞吐量MoE推理系统
R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
面向推理模型的冗余感知KV缓存压缩
SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding
通过自辅助推测解码实现高效MoE推理，最高4.30×吞吐量提升
BlitzScale: Fast and Live Large Model Autoscaling with O(1) Host Caching
通过O(1)主机缓存实现快速、实时的大模型自动扩展
CasMoE: A Cascaded Framework for Efficient MoE Inference on Resource-constrained
通过级联规划器（EAM+EAP）实现高效专家预取，在资源受限设备上加速MoE推理，吞吐量提升65.13%
CommitMoE: Efficient Fallback-Free MoE Inference with Offloading under GPU Memory Constraints
通过量化填充、专家内存池和层感知调度实现无回退的高效MoE推理，相比MoE-Infinity加速2.86倍
DesireKV: Decoupling Sensitivity and Importance for Reasoning-Aware KV Cache
解耦敏感度与重要性：面向推理的KV缓存压缩框架
FlexInfer: Flexible LLM Inference with CPU Computations
利用CPU计算实现灵活的LLM推理
MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU
MegaTrain：在单GPU上全精度训练100B+参数大语言模型
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
通过更好的并行性和工作分配实现更快的注意力
TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
通过集体KV缓存共享扩展多智能体LLM服务
Comparative Characterization of KV Cache Management Strategies for LLM Inference
LLM推理中KV缓存管理策略的比较表征
GMLake: 高效透明的 GPU 内存碎片整理
GMLake 提出高效透明的 GPU 内存碎片整理方法，提升大模型训练效率。
PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Training
PipeFill利用流水线并行训练中的气泡时间执行额外计算任务，提升GPU利用率。