quantization

August 3, 2026

NVIDIA Nemotron 3: Efficient and Open Intelligence

NVIDIA Nemotron 3 系列（Nano/Super/Ultra）—— 混合 Mamba-Transformer MoE 架构，NVFP4 训练，LatentMoE，MTP，多环境 RL，1M token 上下文

llm-training moe mamba quantization nvfp4 latent-moe multi-token-prediction reinforcement-learning long-context inference-optimization

July 23, 2026

Sol Video Inference Engine

Agent-Native Full-Stack Acceleration Framework for Efficient Video Generation

video-diffusion inference-acceleration agent-framework quantization sparse-attention kernel-fusion token-pruning cache

July 16, 2026

SageAttention2++: A More Efficient Implementation of SageAttention2

基于 FP8 Matmul + FP16 累加器指令进一步加速 SageAttention2 的注意力量化内核实现，在保持精度的同时取得相对 FlashAttention 最高 3.9× 加速。

llm-inference kernel-optimization video-generation image-generation efficient-attention quantization

July 11, 2026

TriRoute: Unified Learned Routing for Joint Adaptive Attention, Experts, and KV-Cache Allocation

TriRoute — 单一轻量控制器联合决策注意力分辨率、FFN 专家选择与 KV-Cache 位宽，端到端可训练，单一预算旋钮扫出 Pareto 前沿，缓解跨轴路由坍缩级联

llm-inference memory-efficiency quantization mixture-of-experts kv-cache sparse-attention

July 8, 2026

Dissecting the NVIDIA Blackwell Architecture with Microbenchmarks

用微基准测试解剖 NVIDIA Blackwell 消费级 GPU（GeForce RTX 5080, GB203）架构，并与上一代 Hopper（H100 PCIe, GH100）逐子系统对比——涵盖统一 INT32/FP32 核、精简 FP64、第五代张量核心（FP4/FP6）、内存层次（L1/L2/global）与真实负载（D-GEMM、Transformer 推理）功耗。

benchmark system-optimization quantization

July 8, 2026

Networking-Aware Energy Efficiency in Agentic AI Inference: A Survey（面向网络感知的 Agentic AI 推理能效综述）

CGO 2026 预印本 arXiv 2604.07857。Cornell + NVIDIA 团队（与 Tawa 同作者群）。这是一篇综述论文，系统梳理了 Agentic AI（感知-推理-行动闭环智能体）的能耗挑战与优化方法。核心贡献：① 提出**能量核算框架**，将 Agentic AI 能耗拆解为**计算能量**（FLOPs + 内存访问，DRAM 访问 ~640 pJ vs 算术 <1 pJ）和**通信能量**（token 传输、中间结果交换、多智能体同步）两大瓶颈；② 建立**统一分类学**，四大支柱：模型简化（量化/剪枝/蒸馏/稀疏 MoE/动作简化）、计算控制（token 长度控制/早退/层跳过/解码简化/工作负载调度）、输入与注意力优化（token 剪枝/稀疏注意力/KV 缓存复用）、硬件感知推理（精度调度/DVFS/内存 I/O 优化）；③ 探讨**跨层协同设计**：跨层优化变量（传输-推理耦合/移动感知调度/模型-信道自适应）、用户-边缘-云协作（分割推理/自适应卸载/协作缓存）、通信-推理联合设计（语义通信/RAG 通信/能量感知调度）；④ 识别五大开放挑战：不确定性量化自适应推理、跨模态/跨智能体协作能效、绿色 AI 与性能权衡、联邦绿色学习、碳感知智能体、6G 原生 Agentic AI、能量收集自维持系统。

llm-serving edge-computing energy-efficiency kv-cache quantization wireless-networking

July 8, 2026

Blackwell GPU 架构综述

NVIDIA Blackwell GPU 架构的综合技术综述——涵盖第五代张量核心（tcgen05、FP4/FP6）、张量内存 TMEM、硬件解压引擎 DE、重构内存层次（L1/L2/HBM3e/GDDR7）、双芯 B200 互连，以及与 Hopper/H200 的微基准对比与调优策略。综合自 Jarmusch et al. 两篇微基准论文。

benchmark system-optimization quantization

July 8, 2026

GPU 架构：内存层次、CUDA 与张量核心

Michael Brenndoerfer《GPU Architecture:Memory Hierarchy, CUDA and Tensor Cores》全文中文翻译。从第一性原理讲解 GPU 内存层次与带宽阶梯、CUDA 核心与 SM/warp/SIMT 执行模型、张量核心 WMMA 与混合精度、Roofline 模型、GPU 规格解读、消费级 vs 数据中心 GPU，以及内存/带宽/精度/功耗四大实际约束，附 PyTorch 代码示例。

system-optimization quantization

July 6, 2026

Fine-Tuning and Serving Gemma 4 31B on Google Cloud TPU: A Technical Comparison with GPU Baselines

End-to-end TPU migration for Gemma 4 31B: 1.61× faster training at 2.12× lower cost, with 66% higher long-context inference throughput

llm-training inference-serving system-optimization quantization parallelism-tp

July 5, 2026

FP8 P-Cast Precision in Attention: Sink-Induced Collapse and the Optimality of S=2^8

A quantitative analysis of FP8 E4M3 attention P-casting revealing P-collapse under attention sink and characterizing S=256 as the optimal static scaling factor

kernel-optimization quantization system-optimization memory-efficiency efficient-attention

July 4, 2026

RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory

RateQuant用率失真理论把KV cache按注意力头分配bit宽度：假设量化MSE满足$D(b)=\alpha\beta^{-b}$，用reverse waterfilling闭式求最优分配，并识别distortion model mismatch这一失效模式——不同量化器$\beta$从3.6到5.3变化，套错模型比uniform更差。校准per-quantizer $\beta$并对K/V单独分配预算后，Qwen3-8B在KIVI 2.5bit的PPL从49.3降到14.9（−70%），QuaRot改善6.6 PPL；全流程校准仅1.6秒，推理零开销。

kv-cache quantization llm-inference long-context

July 4, 2026

SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference

SharQ提出FP4量化与N:M稀疏协同的免训练推理方案：在线抽取outlier主导的稀疏骨干做sparse FP4 GEMM，再用dense FP4 GEMM对稀疏路径的量化+掩码误差做残差补偿，两路径共享单份FP4权重与path-specific scale视图。Llama-3.1-8B/Qwen2.5-7B/Qwen3-30B-A3B/Qwen3-VL-8B上恢复43–63%的NVFP4-FP16精度差，RTX 5090延迟对FP16降2.2–2.4×、对FP8吞吐提升1.2–1.4×，Wan2.2-T2V加SageAttention最高1.58×。

quantization llm-inference system-optimization vision-language

July 4, 2026

CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

CAT-Q首次以纯PTQ路线量化LLM到1.58bit三元权重，无需QAT。两个核心模块：Learnable Modulation用可学习因子$(\delta_\mu,\delta_\alpha,\delta_\Delta)$调制预训练权重分布与三元阈值，Softened Ternarization用tanh-based smooth transition函数从identity渐变到hard ternarization。仅512校准样本、8卡A100 8–60小时即可把1.7B–235B模型三元化，性能超越用100B tokens训练的BitNet 1.58-bit v1/v2，训练token量减少约10^5倍。

quantization llm-inference

July 4, 2026

W4A4 Quantization for Inference on Wan2.2-I2V-A14B

面向ICME 2026 W4A4量化挑战赛的Wan2.2-I2V-A14B视频扩散模型4bit权重/4bit激活推理方案。将LLM量化领域的SmoothQuant逐通道平滑与MixQ稀疏离群列拆分迁移到视频扩散FFN，并叠加块级HiF4权重打包与双分支GEMM。在VBench I2V上各项指标相对FP16降幅控制在2–3.5%，运动平滑度反而提升+0.4%，全面优于原生HiFloat4基线（后者跌约5%）。

quantization diffusion-model video-generation system-optimization

July 3, 2026

KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving

首个面向解耦式 LLM 服务的服务感知自适应 KV 缓存压缩框架，通过统一压缩策略空间 + 贝叶斯剖析引擎 + 服务感知在线控制器，PD 分离场景 JCT 加速最高 9.13x，KV 解耦场景 TTFT 降低最高 32.8x

kv-cache llm-serving disaggregation quantization communication system-optimization

June 27, 2026

EfficientLLM: Efficiency in Large Language Models

首个全面评估 LLM 效率技术的大规模基准测试，涵盖架构预训练、微调和量化推理

benchmark efficiency llm quantization peft moe attention-mechanism

June 24, 2026

Full Stack Optimization of Transformer Inference: a Survey

Transformer 推理全栈优化综述，涵盖硬件设计、软件优化、调度策略和神经架构搜索

survey transformer inference-optimization hardware-accelerator neural-architecture-search quantization pruning kernel-optimization

June 17, 2026

Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving

从生产实践角度重新审视KV缓存压缩技术，揭示吞吐量、响应长度和负样本三个被忽视的关键维度

kv-cache inference-serving quantization sparsity system-optimization benchmark

June 15, 2026

ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization

改进极低比特 LLM 量化的缩放定律

llm-training quantization llm-inference

June 11, 2026

FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision

利用异步性和低精度实现快速准确的注意力机制

quantization kernel-optimization scheduling

June 11, 2026

DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers

基于 PCA 旋转的 DiT 4-bit 后训练量化框架

quantization diffusion-model vision-language vlm

June 7, 2026

An Inquiry into Datacenter TCO for LLM Inference with FP8

FP8 量化对数据中心 LLM 推理 TCO 的影响分析

system-optimization llm-inference quantization

June 7, 2026

H2SGEMM: Emulating FP32 GEMM on Ascend NPUs using FP16 Units with Precision Recovery

在华为 Ascend NPU 上使用 FP16 单元模拟 FP32 GEMM 的精度恢复方法

kernel-optimization quantization llm-training

June 5, 2026

LongLive: Real-time Interactive Long Video Generation

实时交互式长视频生成框架

video-generation attention-mechanism quantization llm-training model-compression

June 5, 2026

Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling

NVFP4 量化精度提升的自适应块缩放方法

quantization llm-training architecture survey

June 4, 2026

NanoLLM - 边缘设备优化的本地LLM推理框架

NVIDIA Jetson平台优化的本地大语言模型推理框架，支持量化、多模态、语音、向量数据库和RAG

quantization multimodal audio architecture llm-inference

June 2, 2026

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language

通过平滑激活离群值实现LLM的W8A8量化，支持高达530B参数模型的无损量化

quantization llm-training mixture-of-experts

June 2, 2026

DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using

基于查找表的超低精度CNN推理加速方法，2-bit实现比QNNPACK INT8快1.74倍

kernel-optimization llm-inference quantization

June 2, 2026

QQQ: Quality Quattuor-Bit Quantization for Large Language Models

W4A8量化方法，通过自适应平滑和Hessian补偿实现高质量4-bit权重量化，同时加速预填充和解码

quantization kernel-optimization llm-inference

June 2, 2026

MOSS: Efficient and Accurate FP8 LLM Training with Microscaling and Automatic Scaling

基于微缩放和自动缩放的高效FP8 LLM训练方法，实现34%吞吐量提升

efficient-attention llm-training inference-serving quantization

June 2, 2026

Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow

统一FP8精度流的强化学习训练框架，实现16%端到端加速

quantization reinforcement-learning kernel-optimization

June 2, 2026

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

一种低比特量化方法，通过混合精度、细粒度分组量化和动态量化实现高效LLM推理

llm-inference system-optimization quantization

May 30, 2026

OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

全方向校准的LLM量化技术，通过LWC和LET实现PTQ效率与QAT性能的统一

quantization model-compression llm-inference vla

May 30, 2026

TurboAttention: Efficient Attention Approximation For High Throughputs LLMs

首个将FlashAttention与量化算法结合的注意力加速方案，实现1.8倍延迟加速和2.37倍吞吐量提升

efficient-attention quantization kv-cache

May 30, 2026

Understanding Stragglers in Large Model Training Using What-if Analysis

使用What-if分析理解大模型训练中的Straggler问题

quantization survey llm-training

May 30, 2026

FloE: On-the-Fly MoE Inference on Memory-constrained GPU

面向内存受限GPU的即时MoE推理系统，通过混合压缩和稀疏预测实现48.7×加速

pruning mixture-of-experts quantization scheduling

May 29, 2026

TensorIR: An Abstraction for Automatic Tensorized Program Optimization

面向自动张量化程序优化的编译器抽象，通过Block抽象实现硬件张量计算原语的自动利用

quantization system-optimization benchmark

May 29, 2026

KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planning

KV-Cache-centric memory management system for efficient embodied planning with static-dynamic memory construction, multi-hop recomputation, and layer-balanced loading

llm-inference kv-cache quantization vla prefix-caching

May 27, 2026

FP8-LM: Training FP8 Large Language Models

FP8 大语言模型训练

quantization distributed-training communication

May 27, 2026

Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms

高效生成式大语言模型服务综述：从算法到系统

survey quantization parallelism-dp

May 27, 2026

KIVI: A Tuning-Free Asymmetric Quantization for KV Cache

面向KV缓存的免调优非对称量化

quantization llm-inference kv-cache vla

May 27, 2026

GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM

高效的KV缓存压缩框架，实现近无损生成推理

kv-cache llm-inference quantization

May 27, 2026

MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language

面向大语言模型的混合精度自回归并行推理

quantization efficient-attention llm-inference

May 27, 2026

COAT: Compressing Optimizer States and Activation for Memory-Efficient FP8 Training

通过压缩优化器状态和激活实现高效FP8训练

quantization llm-training distributed-training vla vlm

May 27, 2026

DeepSeek-V3 Technical Report

DeepSeek-V3: 671B 参数 MoE 语言模型技术报告

mixture-of-experts quantization llm-training

May 27, 2026

KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization

KVTuner：敏感度感知的层级混合精度 KV 缓存量化

quantization efficient-attention pruning

May 27, 2026

Charon: 面向大规模 LLM 训练和推理的统一细粒度模拟器

面向大规模 LLM 训练和推理的统一细粒度模拟器

efficient-attention llm-training llm-inference quantization kv-cache mixture-of-experts reinforcement-learning

May 27, 2026

OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

面向 2-bit KV 缓存量化的离线谱协方差感知旋转方法

quantization llm-inference kv-cache

April 1, 2026

大语言模型长上下文推理优化

大语言模型长上下文推理优化技术综述，涵盖 KV Cache 压缩、注意力优化等。

sparse-attention efficient-attention long-context llm-inference quantization

March 30, 2026

OdysseyLLM: A Speed Odyssey for Deployable Quantization of LLMs

OdysseyLLM 提出可部署的大语言模型量化加速方案。

quantization kernel-optimization inference-serving llm-inference multimodal

March 28, 2026

TriForce: 层次化推测解码实现长序列无损加速

TriForce 提出层次化推测解码方法，实现长序列生成的无损加速。

long-context quantization llm-inference

March 25, 2026

A Survey of Low-bit Large Language Models

低比特大语言模型综述：基础、系统和算法。

quantization survey llm-inference kernel-optimization kv-cache-optimization

March 18, 2026

OmniQuant: Omnidirectionally Calibrated Quantization for LLMs

OmniQuant 提出全方位校准的大语言模型量化方法。

quantization attention-mechanism llm-training multimodal inference-serving

March 8, 2026

Scalable Training of Mixture-of-Experts Models with Megatron Core

NVIDIA提出基于Megatron Core的可扩展MoE模型训练方法，支持大规模混合专家模型的高效分布式训练。

mixture-of-experts distributed-training quantization parallelism attention-mechanism

October 30, 2025

ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference

自适应专家调度与内存协调的MoE推理运行时系统，通过动态步长预测和缓存感知路由消除99.9%等待延迟。

mixture-of-experts system-optimization quantization communication scheduling

June 4, 2025

LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments

针对资源受限环境的低延迟视觉语言模型推理流水线

quantization speculative-decoding inference-serving reasoning vlm mllm

January 30, 2025

Streaming DiLoCo with Overlapping Communication: Towards a Distributed Free Lunch

大规模语言模型（LLM）训练通常分布 across 大量加速器以减少训练时间。由于内部状态和参数梯度需要在每个梯度步骤中交换，所有设备需要使用低延迟高带宽通信链路 co-locate。

communication attention-mechanism quantization distributed-training parallelism-dp parallelism-tp

December 2, 2024

Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking

通过动态输入剪枝和缓存感知掩码实现高效LLM推理，降低计算和内存开销。

token-pruning model-compression quantization sparse-attention llm-training

May 7, 2024

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

QServe提出W4A8KV4量化与系统协同设计，通过QoQ算法和QServe推理库实现LLM服务吞吐量提升2.36倍，成本降低3倍。

quantization llm-inference kernel-optimization parallelism