Hey all. Apologies in advance if this is the wrong category for this conversation.
My team is considering investing in a local workstation for model fine-tuning (both LLM and image generation) and inference (using various HuggingFace libraries - got some stuff going with diffusers, sentence-transformers, etc). I’m having a hard time finding good articles discussing multi-GPU hardware setups and their pros and cons - I found this article suggesting that a multi-4090 setup struggles with image model finetuning, but relatively little else.
Would a multi-GPU setup with multiple RTX 5000 Ada or RTX 6000 Ada GPUs still be worthwhile for improving training performance on fine-tuning LLMs or image generation models? If yes, does the overhead increase with additional GPUs beyond 2? (it seems like we could get a 4x RTX 5000 server for about the price of 2x RTX 6000). If no, what other options should we consider for better performance? H100s, older hardware that still supports nvlink, something else?