FSDP vs DDP: when do you need sharded data parallel?
Use DDP until params, grads and AdamW state stop fitting on one GPU, about 3.75B params on an 80 GB H100. Then shard with FSDP. The memory math, worked.
Use DDP until params, grads and AdamW state stop fitting on one GPU, about 3.75B params on an 80 GB H100. Then shard with FSDP. The memory math, worked.