speculative-decoding

24 posts tagged with "speculative-decoding"

July 23, 2026

PRR: Predict, Reuse, and Repair

Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding via Speculative Execution and Incremental Repair

long-context-llm sparse-attention kv-cache speculative-decoding cuda-kernel flashattention inference-acceleration dynamic-sparse-attention

July 23, 2026

SpecLA: Efficient Speculative Decoding for Linear-Attention Models

A speculative decoding runtime for stateful linear-attention models that verifies chains and trees with topology-aware kernels, stores compact factors to recover accepted states, and uses confidence pruning plus a target-aligned EAGLE-style drafter.

speculative-decoding linear-attention gdn recurrent-state gpu-kernels inference-acceleration triton

July 22, 2026

WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning

面向同步 Agentic RL 的 workload-aware rollout 系统，通过 model-free speculative decoding (SuffixDecoding) 和 cache-aware scheduling 联合优化解码与调度

llm-inference system-optimization reinforcement-learning kv-cache speculative-decoding

July 6, 2026

SRT: Speculative Rollout with Tree-Structured Cache for Reinforcement Learning

Tree-structured speculative rollout cache for RL that eliminates redundant computation across rollouts, enabling faster policy optimization

llm-training speculative-decoding kv-cache system-optimization parallelism

July 5, 2026

Multi-Resolution Flow Matching: Training-Free Diffusion Acceleration via Staged Sampling

MrFlow achieves 10x+ training-free acceleration for flow-matching diffusion models via a four-stage low-to-high resolution pipeline

diffusion-model image-generation system-optimization speculative-decoding model-compression

July 4, 2026

Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

Nightjar 将投机长度选择建模为上下文多臂赌博机（Contextual MAB），按 batch 大小自适应选择 γ 或彻底关闭投机，并在高负载下把 draft 模型 offload 到 CPU 释放显存扩大 KV cache；在动态请求率下相比标准 SD 吞吐提升最高 14.76%、延迟降低最高 20.18%，相比无 SD 平均提升 27.29%。

speculative-decoding inference-serving kv-cache scheduling system-optimization

July 4, 2026

MiMo-V2-Flash Technical Report

小米 309B 总参 / 15B 激活的 MoE 推理与 Agent 大模型。混合注意力（5:1 SWA:GA，128 窗口 + 可学习 attention sink，KV/算力近 6× 缩减），27T token 预训练 + MTP，原生 32K 扩展到 256K。提出 Multi-Teacher On-Policy Distillation (MOPD)：多领域 RL 教师提供 token 级 KL 稠密奖励 + ORM 结果奖励，学生在自身分布上蒸馏多专家。以 1/2、1/3 的参数比肩 DeepSeek-V3.2 与 Kimi-K2；SWE-Bench Verified 73.4%。三层 MTP 做自投机解码，接受长度达 3.6、解码提速 2.6×。

mixture-of-experts sparse-attention speculative-decoding long-context rlhf reasoning

July 4, 2026

Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving

PASTE 通过 Pattern Tuple 抽象从历史执行中挖掘控制流与数据依赖，基于风险感知的机会式调度在 LLM 生成阶段就投机执行未来工具调用；跨深度研究、编码、科学 agent 工作负载将端到端任务完成时间平均降低 43.5%，工具延迟降低 1.8×，验证空闲达 10×。

inference-serving llm-inference scheduling speculative-decoding system-optimization

July 4, 2026

LUMEN: Coordinated Failure Recovery for Distributed LLM Serving

针对分布式 LLM 服务中 worker 故障同时丢失 KV cache 与服务容量的问题，提出以负载感知协调决策来做故障恢复的 LUMEN。三大机制：负载感知 KV checkpointing（用 $h(r)=\arg\min_w(q_w+\lambda p_w(r))$ 把 checkpoint 分散到低负载 worker）、局部性感知恢复调度（先按 checkpoint holder 路由再按 average-based 规则重平衡）、投机辅助渐进恢复（在恢复 worker 上加载 draft model，通过 fused batch 单次前向验证辅助最拥堵的存活 worker）。SGLang 原型上，Qwen3-32B/14B 相比 Stop-and-Restart 降低 TTFT 44.4%/29.6%、recovery time 50.0%/64.1%；大规模模拟中在 64 worker、25% 故障率下仍有 46.8-51.2% TTFT 降幅。

inference-serving llm-inference kv-cache speculative-decoding system-optimization

June 29, 2026

xLLM Technical Report

xLLM 是面向企业级大规模服务的 LLM 推理框架，采用解耦服务-引擎架构，通过动态 PD 分离、EPD 分离和多层流水线优化，吞吐量达到 MindIE 的 1.7x 和 vLLM-Ascend 的 2.2x

llm-serving inference-engine pd-disaggregation kv-cache speculative-decoding

June 29, 2026

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

DSpark 结合半自回归生成和置信度调度验证，统一高吞吐并行生成与自适应负载感知验证，在 DeepSeek-V4 生产系统中实现 60%-85% 生成加速

speculative-decoding llm-inference inference-optimization semi-autoregressive confidence-scheduling deepseek

June 27, 2026

Mellum2 Technical Report

JetBrains 开源 12B MoE 代码大模型，2.5B 激活参数，128K 上下文，支持代码生成/推理/工具调用

code-generation moe small-model ide-assistant speculative-decoding

June 22, 2026

SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference

通过复用闲置尾部模型服务作为远程草稿模型，实现资源高效的 LLM 推理服务

llm-serving speculative-decoding multi-tenant resource-efficiency inference-optimization

June 9, 2026

Better & Faster Large Language Models via Multi-token Prediction

通过多 token 预测训练提升 LLM 性能和推理速度

speculative-decoding llm-inference llm-training

May 28, 2026

MineDraft: A Framework for Batch Parallel Speculative Decoding

MineDraft：批量并行推测解码框架

llm-inference speculative-decoding parallelism-dp

April 29, 2026

Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decoding

前沿语言模型的 RL 后训练越来越多地被自回归 rollout 生成所瓶颈，使得 rollout 加速成为核心系统挑战。

speculative-decoding llm-inference llm-training

March 19, 2026

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative

大语言模型（LLM）的自回归推理存在高延迟问题。投机解码（Speculative Decoding）通过使用轻量级草稿模型提出多个token进行批量验证来缓解这一瓶颈。然而，投机解码的采用受到以下限制：

llm-inference speculative-decoding distributed-training

March 3, 2026

Speculative Speculative Decoding (SSD) / Saguaro

自回归解码因其顺序性而受到瓶颈限制。投机解码（Speculative Decoding, SD）已成为加速推理的标准方法，通过使用快速草稿模型预测慢速目标模型即将生成的 token，然后通过一次目标模型前向传播并行验证。

speculative-decoding llm-inference diffusion-model attention-mechanism parallelism-dp

June 4, 2025

LiteVLM: A Low-Latency Vision-Language Model Inference Pipeline for Resource-Constrained Environments

针对资源受限环境的低延迟视觉语言模型推理流水线

quantization speculative-decoding inference-serving reasoning vlm mllm

March 3, 2025

EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time

现代大语言模型（LLM）的自回归生成特性使其推理速度慢且成本高昂。投机解码（Speculative Sampling）已被证明是解决此问题的有效方案。EAGLE 系列方法通过在特征级进行自回归，复用目标模型的顶层特征，取得了比传统投机解码更好的效果。

speculative-decoding llm-inference diffusion-model

February 24, 2025

LongSpec: Long-Context Lossless Speculative Decoding

大语言模型（LLM）处理长上下文的能力对 LLM agents 和长推理任务等新兴应用至关重要，这些应用现在可以处理扩展到数百万 token 的上下文窗口。在这些高要求的长上下文场景中，标准自回归解码的高推理延迟成为一个显著的瓶颈。

efficient-attention long-context speculative-decoding llm-inference

February 5, 2025

Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free

在现代大语言模型（LLM）推理引擎中，改善首 token 延迟（Time-to-First-Token, TTFT）是一个至关重要的目标。优化 TTFT 可以直接提高最大 QPS，并满足许多关键应用的需求。

llm-inference speculative-decoding long-context

January 26, 2024

EAGLE 系列：大语言模型投机解码加速

EAGLE 系列是针对大语言模型（LLM）推理加速的投机解码（Speculative Decoding）方法，由 Yuhui Li 等人提出。该系列从特征级自回归、动态草稿树到训练时测试，逐步提升了投机解码的速度和效率。

speculative-decoding llm-inference parallelism diffusion-model architecture

November 30, 2022

Fast Inference from Transformers via Speculative Decoding

大型自回归模型（如 Transformer）的推理速度很慢——解码 K 个 token 需要 K 次串行运行模型。这成为了部署大型语言模型的主要瓶颈。

speculative-decoding architecture llm-inference diffusion-model inference-serving