[中文] 流水线调度:GPipe、1F1B与零气泡
[中文] 实现Device Mesh、DDP与FSDP
[中文] All-to-All token分发与合并
[英文] Model Architecture, Parameters and FLOPs
[英文] RoPE from First Principles
[英文] Implementing RoPE and YaRN
[英文] Building the Transformer
[英文] Attention, KV Cache and Arithmetic Intensity
[英文] Coding Multi-head Latent Attention
[英文] Block Matmul and MLA Internals
[英文] Deriving MLA and Decoupled RoPE
[英文] MLA Weight Absorption
[英文] Autograd and Distributed Training Math
[英文] Distributed Computation Graphs and DDP
[英文] Building the Training Loop
[英文] Pipeline Parallelism from First Principles
[英文] Pipeline Schedules: GPipe, 1F1B, Zero Bubble
[英文] Datasets, Tokenization and Data Parallelism
[英文] Coding Pipeline Parallelism
[英文] Device Meshes: Combining PP with DP
[英文] Distributed Communication Collectives
[英文] Implementing Device Meshes, DDP and FSDP
[英文] Tensor Parallelism from First Principles
[英文] Coding Tensor Parallelism
[英文] Context Parallelism and Ring Attention
[英文] Metrics, Optimizers, Schedulers, Checkpointing
[英文] Combining Parallelism in the Training Loop
[英文] Mixture of Experts from First Principles
[英文] Tensor Parallelism for MoE
[英文] All-to-All Token Dispatch and Combine
[英文] Expert Tensor Parallelism