moe

9 posts tagged with "moe"

August 10, 2026

Topology-Aware Data Movement for Disaggregated GPU Inference

TopKV — 拓扑感知的KV缓存传输编排器，利用GPU互连层次结构将KV缓存传输延迟降低3-18倍

disaggregated-inference kv-cache gpu-interconnect nvlink cxl moe system-optimization

August 3, 2026

NVIDIA Nemotron 3: Efficient and Open Intelligence

NVIDIA Nemotron 3 系列（Nano/Super/Ultra）—— 混合 Mamba-Transformer MoE 架构，NVFP4 训练，LatentMoE，MTP，多环境 RL，1M token 上下文

llm-training moe mamba quantization nvfp4 latent-moe multi-token-prediction reinforcement-learning long-context inference-optimization

July 23, 2026

PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

A runtime memory-management system for MoE LLM serving that dynamically quantizes expert weights at runtime to balance GPU memory between model weights and KV cache, using a quality-aware planner with offline sensitivity, online routing statistics, and prompt residuals.

moe llm-serving weight-quantization gpu-memory-management runtime-quantization pagedattention any-precision

July 4, 2026

WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware

WiSP 将低资源 MoE serving 建模为专家权重与 KV 缓存两条 working-set 争夺同一 VRAM 的问题。核心是 routing-aware expert pager（基于 expert_map 的 LRU 分页，字节等价输出）加 MV-WSA 边际价值分配器，联合分配 expert scratch 与 KV pool。iso-VRAM 下相对 vLLM static offload 解码吞吐最高提升 1.95×，Jamba-52B/MiniMax-M2-229B 等 baseline 无法启动的模型上也能可服务，且揭示单流 decode 中 routing 预测只能省显存不能省延迟。