Blog Tags GitHub
All tags

training-free

6 posts tagged with "training-free"

The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs

最大规模的训练无关稀疏注意力经验分析，覆盖Qwen 2.5/Llama 3.1/Gemma 3三大模型家族(4B-72B)、16K-128K序列长度、稀疏度0-0.95，发现稀疏注意力Pareto前沿、prefill/decode两阶段行为差异、序列长度与稀疏容忍度关系

sparse attention training-free isoCost analysis Pareto frontier long-context transformer LLM taxonomy Qwen 2.5 Llama 3.1 Gemma 3 vertical-slash SnapKV Quest Block-Sparse FlexPrefill Ada-SnapKV

ScalingAttention: Discovering Intrinsic Sparse Attention Topology for Video Diffusion Transformers

ScalingAttention：通过权重编码的内在稀疏注意力拓扑发现机制（WEST）与保真度感知的稀疏度自适应调节（FAST）以及位级块稀疏注意力内核（crm kernel），实现视频扩散Transformer的高效训练无关推理加速

video diffusion transformers sparse attention intrinsic sparse topology WEST FAST Hellinger distance CRM kernel FlashAttention-3 Wan2.1 HunyuanVideo training-free block-sparse attention

Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification

训练无关动态稀疏注意力：通过查询依赖高斯校准阈值、在线 Softmax 内联稀疏化和零阶泰勒代理分数复用，在不修改权重的情况下为视频生成 DiT 提供 2.0×–5.1× 端到端加速

sparse-attention video-generation diffusion-transformer training-free online-softmax flashattention gpu-kernel inference-acceleration block-sparse approximation-correction