Base Training & Distributed Compute
Large-scale pretraining, hardware execution, and distributed model sharding
Megatron-LM / PyTorch FSDP2
Backbone engines for Tensor Parallelism (TP), Pipeline Parallelism (PP), and Fully Sharded Data Parallelism (FSDP2) across multi-node GPU clusters. Eliminates inter-node bottlenecking and synchronizes gradients across tens of thousands of GPUs.
DeepSpeed
Pioneering ZeRO-stage (ZeRO-1, ZeRO-2, ZeRO-3) distributed memory optimization and CPU/NVMe parameter offloading engines. Enables training massive parameter models on constrained hardware budgets.
FlashAttention (v1–v4)
IO-aware exact attention kernels that dramatically reduce high-bandwidth memory (HBM) read/writes. Tailored specifically for NVIDIA Ampere, Hopper (H100/H200), and Blackwell (B200) architectures to unlock near-theoretical peak FLOPs.