Blog Tags GitHub

efficient-inference

5 posts tagged with "efficient-inference"

OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

免训练的两阶段渐进式 token 压缩框架：LLM 前的模态特异结构压缩 + LLM 内的查询条件音视协同精炼

token-compression omni-modal multimodal-llm audio-visual-understanding efficient-inference training-free

SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training

首个 FP4 微缩放注意力推理加速与 INT8 可训练注意力探索

efficient-inference diffusion-model llm-inference sparse-attention efficient-attention

Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding

PRR 推理运行时，通过 EMA 预测器+增量修复内核将动态稀疏注意力的选择-注意力依赖瓶颈降低 30%+

efficient-inference llm-serving sparse-attention kv-cache cuda

Compute Only Once: UG-Separation for Efficient Large Recommendation Models

UG-Sep 在 TokenMixer 架构中解耦用户侧和物品侧信息流，实现用户侧计算复用，推理延迟降低 20%，已部署于字节跳动多个业务场景

efficient-inference recommendation-systems token-mixer computation-reuse industrial-deployment

Unlimited OCR Works

通过Reference Sliding Window Attention实现恒定KV缓存，支持单次前向传播解析数十页文档

ocr attention-mechanism long-context kv-cache document-understanding efficient-inference