How much VRAM do you need to run or fine-tune a 70B model?
A 70B model needs 40 to 141 GB for weights plus a KV cache to serve, and 50 GB to 1.1 TB to fine-tune. The math, GPU setups and a script to check yours.
inference
1 post with the tag “inference”