Hardware suggestions

Andrewberryman10101 · June 6, 2025, 1:21am

Hi,

Apologies if this is a common question. I have a test rig with a 8700 + 3060 12gb running at the moment, but in order to run larger models, is it feasible to run an X99 Xeon (16 or 18 core) along with 128gb RAM, plus 3 or 4 3060 12gb cards (perhaps 3-4 5060ti 16gb cards in future)?

Obviously bigger models could run in system ram but run slower, and anything needing GPU vram would load into that when needed. Is this possible? is the power draw going to be ridiculous?

Im down an AliExpress rabbit hole right now and need some sanity lol!

John6666 · June 6, 2025, 11:37am

For hardware purchase advice, I highly recommend Hugging Face Discord.

Obviously bigger models could run in system ram but run slower, and anything needing GPU vram would load into that when needed. Is this possible?

Multi-GPU is more difficult than single-GPU in terms of coding and environment settings, but I think many people use multi-GPU for that purpose. Just be careful with IOMMU settings in BIOS/UEFI… I hear it’s dangerous.

github.com/pytorch/pytorch

Multi-gpu example freeze and is not killable

opened 01:27PM - 09 Aug 19 UTC

Dubrzr

module: dependency bug module: multi-gpu module: multiprocessing module: cuda triaged module: deadlock has workaround module: data parallel quansight-nack

## 🐛 Bug Running pytorch with multiple P40 gpus freeze and is not killable (e…ven kill -9 by root). Only a reboot removes this process. Inside docker container (with nvidia-docker2) it freezes docker. https://github.com/NVIDIA/nvidia-docker/issues/1010 ## To Reproduce Steps to reproduce the behavior: 1. Install pytorch 1.0.2 2. Run the following code on multiple P40 Gpus ``` import os ###tutorial from https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html ###no error with only 1 gpu # os.environ['CUDA_VISIBLE_DEVICES'] = '0' #### to reproduce error allow multi gpu os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3' import torch torch.cuda.device_count() import torch.nn as nn from torch.utils.data import Dataset, DataLoader # Parameters and DataLoaders input_size = 5000 #increased input size (works with 500 on multi gpu) output_size = 2000 #increased output size (works with 200 on multi gpu) batch_size = 300 data_size = 100 device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") print(device) class RandomDataset(Dataset): def __init__(self, size, length): self.len = length self.data = torch.randn(length, size) def __getitem__(self, index): return self.data[index] def __len__(self): return self.len rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size), batch_size=batch_size, shuffle=True) class Model(nn.Module): # Our model def __init__(self, input_size, output_size): super(Model, self).__init__() self.fc = nn.Linear(input_size, output_size) def forward(self, input): output = self.fc(input) return output model = Model(input_size, output_size) if torch.cuda.device_count() > 1: print("Let's use", torch.cuda.device_count(), "GPUs!") # dim = 0 [30, xxx] -> [10, ...], [10, ...], [10, ...] on 3 GPUs model = nn.DataParallel(model) model.to(device) for i in range(10000): for data in rand_loader: input = data.to(device) output = model(input) ``` ## Expected behavior The training ## Environment Collecting environment information... PyTorch version: 1.0.1.post2 Is debug build: No CUDA used to build PyTorch: 10.0.130 OS: Ubuntu 18.04.2 LTS GCC version: (crosstool-NG fa8859cb) 7.2.0 CMake version: Could not collect Python version: 3.5 Is CUDA available: Yes CUDA runtime version: 10.0.130 GPU models and configuration: GPU 0: Tesla P40 GPU 1: Tesla P40 GPU 2: Tesla P40 GPU 3: Tesla P40 GPU 4: Tesla P40 GPU 5: Tesla P40 GPU 6: Tesla P40 GPU 7: Tesla P40 Nvidia driver version: 410.79 cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.2 Versions of relevant libraries: [pip3] numpy==1.15.2 [conda] mkl 2018.0.3 1 defaults [conda] mkl_fft 1.0.6 py35_0 conda-forge [conda] mkl_random 1.0.1 py35_0 conda-forge [conda] nomkl 2.0 0 defaults [conda] numexpr 2.6.5 py35_nomklhaa809a4_0 [nomkl] defaults [conda] pytorch 1.0.1 py3.5_cuda10.0.130_cudnn7.4.2_2 pytorch [conda] torch 0.4.1 <pip> [conda] torchvision 0.2.2 py_3 pytorch cc @ezyang @gchanan @zou3519 @ngimel

is the power draw going to be ridiculous?

With that spec, a single GeForce card seems to consume about 200 watts, which is an incredible amount of power, but it seems like it would be about the same as running an air conditioner at full power…

Topic		Replies	Views
If I have a small amount of VRAM compared to the model, will pytorch still use the CUDA accelerations? Beginners	0	449	April 21, 2023
Can't load huge model onto multiple GPU's Beginners	5	5130	June 15, 2023
How to load large model with multiple GPU cards? Beginners	8	42842	October 25, 2023
Running multiple instances on GPU Intermediate	0	176	June 1, 2024
Multi GPU Build Possible? Beginners	2	137	January 19, 2025

Hardware suggestions

Related topics