Blog Tags GitHub

sglang

2 posts tagged with "sglang"

August 10, 2026

TensorCast: The Missing Tensor Management Layer in Large Language Model Infrastructure

首个将张量生命周期管理从计算栈中解耦的分布式服务层，通过 Artifact/Operation/Plan/Signal 四抽象实现 TaaS 范式，集成 vLLM/SGLang 后权重物化加速 60×、KV 缓存共享 TTFT 降低 87.5%

llm-infra tensor-management distributed-systems model-serving kv-cache taaS vllm sglang disaggregated-inference

Bole: Efficient Tree Speculation for Hybrid-Attention Language Models

内核-运行时协同设计：通过闭式树验证、值域分块GPU核和因子化推测状态，将混合注意力LLM的树投机解码吞吐量提升至自回归解码的4.72×

tree-speculative-decoding hybrid-attention linear-attention gdn llm-inference sglang gpu-kernel-optimization closed-form-parallelization