TileLink: Generating Efficient Compute-Communication Overlapping Kernels using Tile-Centric Primitives
基于Tile-Centric原语的编译器框架，高效生成计算-通信重叠内核
All tags
parallelism-sp
10 posts tagged with "parallelism-sp"
分布式并行训练必读经典论文
覆盖并行策略、序列并行、MoE并行、内存优化、通信优化、容错与系统框架
SwiftFusion: Scalable Sequence Parallelism for Distributed Inference of Diffusion Transformers on GPUs
面向扩散Transformer分布式推理的可扩展序列并行引擎
WLB-LLM: Workload-Balanced 4D Parallelism for Large Language Model Training
面向大语言模型训练的工作负载均衡4D并行
Transforming the Use of Earth Observation Data: Exascale Training of a Generative
利用历史先验的生成式压缩模型实现高达10000倍地球观测数据压缩
Striped Attention: Faster Ring Attention for Causal Transformers
针对因果 Transformer 的更快 Ring Attention 算法
Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
面向GPU集群长上下文注意力的拓扑感知解码
ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on
ByteScale：在超过 12,000 GPU 上高效扩展 2048K 上下文 LLM 训练
Scaling Llama 3 Training with Efficient Parallelism Strategies
Llama 3 405B 模型在 16K H100 GPU 上的四维并行训练系统设计，涵盖灵活流水线并行、上下文并行、大规模调试方法及硬件建议
xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism
面向扩散 Transformer 的大规模并行推理引擎，混合多种并行策略实现 16 GPU 扩展