Cannot launch multi-gpu training?

Data-drone · September 14, 2023, 10:36am

I am trying to run training on multiple GPUs following this script here:

databricks/databricks-ml-examples/blob/master/llm-models/llamav2/llamav2-7b/06_fine_tune_qlora.py

# Databricks notebook source
# MAGIC %md
# MAGIC # Fine tune llama-2-7b-hf with QLORA
# MAGIC
# MAGIC [Llama 2](https://huggingface.co/meta-llama) is a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. It is trained with 2T tokens and supports context length window upto 4K tokens. [Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) is the 7B pretrained model, converted for the Hugging Face Transformers format.
# MAGIC
# MAGIC This is to fine-tune [llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) models on the [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) dataset.
# MAGIC
# MAGIC Environment for this notebook:
# MAGIC - Runtime: 13.2 GPU ML Runtime
# MAGIC - Instance: `g5.8xlarge` on AWS, `Standard_NV36ads_A10_v5` on Azure
# MAGIC
# MAGIC We leverage the PEFT library from Hugging Face, as well as QLoRA for more memory efficient finetuning.

# COMMAND ----------

from huggingface_hub import notebook_login

# Login to Huggingface to get access to the model
notebook_login()

This file has been truncated. show original

single gpu is fine. When I switch to multi-gpu I got:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

So I tried wrapping the dataloader in accelerate.prepare.
then I get:

AttributeError: 'DataLoaderShard' object has no attribute 'map'

What is the correct path forward?

Topic		Replies	Views
Fine tunning llama2 with multiple GPUs and Hugging face trainer 🤗Transformers	1	3048	November 3, 2023
Fine-tunning llama2 with multiple GPU hugging face trainer 🤗Transformers	8	2043	March 7, 2024
Multi-GPU inference with LLM produces gibberish 🤗Transformers	12	5390	March 28, 2024
Loading a HF Model in Multiple GPUs and Run Inferences in those GPUs 🤗Accelerate	9	4746	August 15, 2023
Multi-GPU LLM inference data parallelism (llama) Beginners	1	6495	October 25, 2023

Cannot launch multi-gpu training?

Related Topics