How do you train a PyTorch model on multiple GPUs?
Use DDP when the model fits on one GPU, FSDP when it doesn't, and launch with torchrun. Code, memory math and a cost check for going multi-node.
distributed
1 post with the tag “distributed”
Use DDP when the model fits on one GPU, FSDP when it doesn't, and launch with torchrun. Code, memory math and a cost check for going multi-node.