Flash Sparse Attention: An Alternative Efficient Implementation of Native Sparse Attention Kernel
FSA 通过交换 NSA selected attention 内核的两层循环顺序（从 query 分组改为 KV 块分组），消除小 GQA group 下的 padding 浪费，实现最高 3.5× 内核加速、1.25× 端到端训练加速。
All tags
kernel-optimization
41 posts tagged with "kernel-optimization"
FastTPS: An Optimized Method for LLM Token Phase for AI Accelerators
面向AI加速器的LLM Token阶段优化框架FastTPS，包括全局KV缓存管理、序列维度扁平化、Fusion MLP等技术
SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD
在华为昇腾 NPU SuperPOD 上对万亿参数 MoE 模型（DeepSeek-V4）进行全参数后训练的系统优化与面向运筹学的 CPT-SFT 工作流
SageAttention2++: A More Efficient Implementation of SageAttention2
基于 FP8 Matmul + FP16 累加器指令进一步加速 SageAttention2 的注意力量化内核实现，在保持精度的同时取得相对 FlashAttention 最高 3.9× 加速。
Redesigning GROMACS Halo Exchange: Improving Strong Scaling with GPU-initiated NVSHMEM
GPU-initiated NVSHMEM redesign of GROMACS domain decomposition halo exchange, fusing data packing and communication to eliminate CPU-GPU sync bottlenecks and improve strong scaling by up to 2x
A Performance Model for Warp Specialization Kernels
A performance model using differential equations to accurately predict execution time of warp specialization kernels, enabling optimal tile configuration search via Z3 SMT solver
LIMINAL: Exploring The Frontiers of LLM Decode Performance
An analytical performance model (LIMINAL) that abstracts application requirements and hardware capabilities to systematically explore LLM inference performance limits across current, near-future, and hypothetical hardware
ACTA: Automatic Configuration of the Tensor Memory Accelerator for High-End GPUs
A software library that heuristically determines optimal tile sizes and queue configurations for TMA-based GPU kernels using the GPU Specification Table (GST) and Little's Law
通过张量核心与 CUDA 核心并行执行提升 GPU 吞吐
UNT 与 William & Mary 提出的纯硬件 GEMM 卸载方案：在张量核心执行 GEMM 时，将部分 WMMA 指令翻译为 MAC 指令下放到空闲的 CUDA 核心并行执行，配合额外 load-store 单元缓解访存瓶颈，在 GPGPU-Sim 上最高提升 29% 性能，CUDA 核心利用率从近零提升至平均 ~73%。
Orchestrated Scheduling and Prefetching for GPGPUs
A prefetch-aware (PA) warp scheduling policy that coordinates thread scheduling and data prefetching in GPGPUs to better tolerate long memory latencies
FP8 P-Cast Precision in Attention: Sink-Induced Collapse and the Optimality of S=2^8
A quantitative analysis of FP8 E4M3 attention P-casting revealing P-collapse under attention sink and characterizing S=256 as the optimal static scaling factor
KernelBench: Can LLMs Write Efficient GPU Kernels?
A benchmark framework for evaluating LLMs' ability to generate fast and correct GPU kernels across 250 PyTorch workloads
Full Stack Optimization of Transformer Inference: a Survey
Transformer 推理全栈优化综述，涵盖硬件设计、软件优化、调度策略和神经架构搜索
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
利用异步性和低精度实现快速准确的注意力机制
RT-Lynx: Putting the GEMM Sparsity In a Right Way for Diffusion Models
从权重稀疏到激活稀疏：DiT 模型的 N:M 半结构化稀疏加速
TMA-Adaptive FP8 Grouped GEMM: Eliminating Padding
消除 Hopper GPU 上低精度训练和推理的填充需求
Understanding GEMM Performance and Energy on NVIDIA Ada Lovelace: A Machine Learning-Based Analytical Approach
基于机器学习的 NVIDIA Ada Lovelace GEMM 性能和能耗分析
H2SGEMM: Emulating FP32 GEMM on Ascend NPUs using FP16 Units with Precision Recovery
在华为 Ascend NPU 上使用 FP16 单元模拟 FP32 GEMM 的精度恢复方法
Tawa: Automatic Warp Specialization for Modern GPUs with Asynchronous References
基于异步引用的现代 GPU 自动 Warp 特化编译器
SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations
面向细粒度和稀疏 MoE 的 IO 和 Tile 感知优化加速器
Execution-Centric Characterization of FP8 Matrix Cores, Asynchronous Execution, and Structured Sparsity on AMD MI300A
AMD MI300A 上 FP8 矩阵核心、异步执行和结构化稀疏的执行特征分析
Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity
基于非结构化稀疏性的高效大语言模型推理加速框架，通过 Load-as-Sparse and Compute-as-Dense 方法在 Tensor Core 上实现高效 SpMM
DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using
基于查找表的超低精度CNN推理加速方法，2-bit实现比QNNPACK INT8快1.74倍
QQQ: Quality Quattuor-Bit Quantization for Large Language Models
W4A8量化方法，通过自适应平滑和Hessian补偿实现高质量4-bit权重量化，同时加速预填充和解码
DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
通过区分检索头和流式头，同时优化长上下文LLM推理的内存和计算效率
LiquidGEMM: Hardware-Efficient W4A8 GEMM Kernel for High-Performance LLM Serving
硬件高效的W4A8 GEMM内核，实现2.90倍加速和4.94倍系统级加速
Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flow
统一FP8精度流的强化学习训练框架，实现16%端到端加速
Minuet: Accelerating 3D Sparse Convolutions on GPUs
面向GPU的3D稀疏卷积加速引擎
EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
针对MoE模型的专家流水线调度器，通过计算通信重叠实现推理吞吐量提升
KPerfIR: Towards an Open and Compiler-centric Ecosystem for GPU Kernel Performance
面向现代AI工作负载的编译器中心GPU内核性能分析基础设施
DVM: A Bytecode Virtual Machine Approach for Dynamic Tensor Computation
DVM：基于字节码虚拟机的动态张量计算方法
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
通过更好的并行性和工作分配实现更快的注意力
FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion
通过内核融合实现 GPU 上快速软件通信重叠
AsyncSparse: Accelerating Sparse Matrix-Matrix Multiplication on Asynchronous GPU Architectures
面向异步GPU架构的稀疏矩阵乘法加速
DySHARP: Accelerating MoE with Dynamic In-Switch Computing on Multi-GPUs
Mixture-of-Experts (MoE) 已被 DeepSeek、GPT、Llama、Qwen 等主流大模型采用，以降低计算开销。然而，MoE 中的 Expert Parallelism (EP) 需要频繁的 GPU 间通信（Dispatch 和 Combine），这成为性能瓶颈——通...
Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel
现代 GPU 工作负载，特别是大语言模型（LLM）推理，受到内核启动开销和粗粒度同步的限制，阻碍了内核间并行性。
OdysseyLLM: A Speed Odyssey for Deployable Quantization of LLMs
OdysseyLLM 提出可部署的大语言模型量化加速方案。
A Survey of Low-bit Large Language Models
低比特大语言模型综述：基础、系统和算法。
WaferLLM: Large Language Model Inference at Wafer Scale
首个面向晶圆级加速器（Cerebras WSE-2，85 万核）的 LLM 推理系统。提出 PLMR 设备模型刻画晶圆级硬件特征，配合细粒度二维并行、Shift 式 KV 缓存管理，以及 PLMR 合规的 MeshGEMM（两跳传输）与 MeshGEMV（K-tree allreduce），相比多卡 A100 SGLang 端到端提速 10–20×、能效 2–2.5×。
LeanAttention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers
LeanAttention提出硬件感知的可扩展注意力机制，优化Transformer解码阶段的注意力计算效率。
QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
QServe提出W4A8KV4量化与系统协同设计，通过QoQ算法和QServe推理库实现LLM服务吞吐量提升2.36倍，成本降低3倍。