分布式并行训练必读经典论文

一、并行策略基础

#	论文	会议/年份	核心贡献
1	GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism	NeurIPS 2019	流水线并行，微批次调度
2	PipeDream: Generalized Pipeline Parallelism for DNN Training	SOSP 2019	异步流水线并行，1F1B调度
3	Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism	2019	张量并行(TP)，行/列切分
4	Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM	SC 2021	3D并行(TP+PP+DP)，通信优化
5	ZeRO: Memory Optimizations Toward Training Trillion Parameter Models	SC 2020	ZeRO-1/2/3，分片优化器/梯度/参数
6	PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel	VLDB 2023	PyTorch原生FSDP实现

二、序列并行与长上下文

#	论文	会议/年份	核心贡献
7	Ring Attention with Blockwise Transformers for Near-Infinite Context	ICLR 2024	环形注意力，近无限上下文
8	Sequence Parallelism for Long-Range Vision Transformers	2021	序列维度并行
9	DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models	2023	注意力头维度序列并行
10	Context Parallelism for Scalable Million-Token Inference	2025	上下文并行推理

三、混合专家(MoE)并行

#	论文	会议/年份	核心贡献
11	GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding	ICLR 2021	MoE专家并行，Top-2路由
12	Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity	JMLR 2022	Top-1路由，简化MoE
13	MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs	NSDI 2024	万卡MoE训练，通信/容错优化
14	DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model	2024	MLA+MoE，236B参数/21B激活

四、优化器与内存优化

#	论文	会议/年份	核心贡献
15	ZeRO-Offload: Democratizing Billion-Scale Model Training	USENIX ATC 2021	CPU offload优化器状态
16	ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning	SC 2021	NVMe/CPU/GPU统一内存
17	Mixed Precision Training	ICLR 2018	FP16/BF16混合精度训练
18	Training Deep Nets with Sublinear Memory Cost	2016	梯度检查点(Activation Checkpointing)
19	ALMOST-ZERO MEMORY COST OPTIMIZERS FOR LARGE MODEL TRAINING	2024	8-bit优化器

五、通信优化

#	论文	会议/年份	核心贡献
20	BAMBU: Communication-Efficient Large Model Training	2022	梯度压缩+异步通信
21	FLUX: Fast Software-based Communication Overlap on GPUs through Kernel Fusion	2024	计算-通信重叠
22	ZeRO++: Extremely Efficient Collective Communication for Giant Model Training	2023	分层AllGather/ReduceScatter
23	Tesseract: Parallelize 100B-scale Model within a Few Minutes	2023	2D并行减少通信量

六、容错与弹性训练

#	论文	会议/年份	核心贡献
24	Varuna: Scalable, Low-cost Training of Massive Deep Learning Models	EuroSys 2022	弹性流水线并行
25	Oobleck: Resilient Distributed Training on Heterogeneous Clusters	SOSP 2023	异构集群弹性训练
26	FTPipe: Fault-Tolerant Pipeline Parallelism	2020	流水线容错

七、系统框架

#	论文	会议/年份	核心贡献
27	PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation	MLSys 2024	torch.compile，FSDP2
28	Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning	OSDI 2022	自动并行策略搜索
29	GSPMD: General and Scalable Parallelization for ML Computation Graphs	2021	JAX/XLA自动分片
30	COLA: Communication-Efficient Distributed Linear Algebra	2022	通信高效线性代数

八、推荐阅读路径

入门

1. GPipe - 流水线并行基础
2. PipeDream - 异步流水线
3. Megatron-LM - 张量并行基础
4. Mixed Precision Training - 混合精度基础

进阶

5. Megatron-LM (2021) - 3D并行完整方案
6. ZeRO - 内存优化核心
7. PyTorch FSDP - 工业级FSDP
8. Ring Attention - 长上下文并行
9. GShard - MoE并行
10. Switch Transformers - 简化MoE

前沿

11. DeepSpeed Ulysses - 极端长序列
12. MegaScale - 万卡训练
13. DeepSeek-V2 - MLA+MoE架构
14. ZeRO++ - 通信优化
15. Alpa - 自动并行

按技术栈

• Megatron系：3 → 4 → 22 → 23
• DeepSpeed系：5 → 15 → 16 → 19 → 22
• PyTorch系：6 → 27
• MoE系：11 → 12 → 13 → 14
• 长上下文系：7 → 8 → 9 → 10