FSDP vs DeepSpeed: which should you use for LLM training?
Use FSDP by default. Use DeepSpeed ZeRO-3 when only CPU optimizer offload makes the model fit. Memory math for 8B and 70B, configs and a decision rule.
Use FSDP by default. Use DeepSpeed ZeRO-3 when only CPU optimizer offload makes the model fit. Memory math for 8B and 70B, configs and a decision rule.