Blog Tags GitHub

inference-optimization

6 posts tagged with "inference-optimization"

NVIDIA Nemotron 3: Efficient and Open Intelligence

NVIDIA Nemotron 3 系列（Nano/Super/Ultra）—— 混合 Mamba-Transformer MoE 架构，NVFP4 训练，LatentMoE，MTP，多环境 RL，1M token 上下文

llm-training moe mamba quantization nvfp4 latent-moe multi-token-prediction reinforcement-learning long-context inference-optimization

DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

DSpark 结合半自回归生成和置信度调度验证，统一高吞吐并行生成与自适应负载感知验证，在 DeepSeek-V4 生产系统中实现 60%-85% 生成加速

speculative-decoding llm-inference inference-optimization semi-autoregressive confidence-scheduling deepseek

Full Stack Optimization of Transformer Inference: a Survey

Transformer 推理全栈优化综述，涵盖硬件设计、软件优化、调度策略和神经架构搜索

survey transformer inference-optimization hardware-accelerator neural-architecture-search quantization pruning kernel-optimization

FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality

无需训练的视频扩散模型加速策略，通过动态特征复用和 CFG-Cache 实现 1.67x 加速同时保持高质量生成

video-generation diffusion-model inference-optimization cache-acceleration training-free

SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference

通过复用闲置尾部模型服务作为远程草稿模型，实现资源高效的 LLM 推理服务

llm-serving speculative-decoding multi-tenant resource-efficiency inference-optimization

ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inference

通过阶段式协调推理架构，实现流式 VideoLLM 在单 A100 GPU 上达到 134 FPS 吞吐量和低于 50ms TTFT

video-understanding videollm streaming-inference token-dropping inference-optimization real-time-processing