Error while fine tuning with peft, lora, accelerate, SFTConfig and SFTTrainer

luxNychta · November 5, 2024, 7:58pm

Hi,

I try to parallelize training on 4 GPU (v100 32GB VRAM). I have a working code for 1 GPU using lora, peft, SFTConfig and SFTTrainer. I tried to add some lines from accelerate (the lib) as I saw on some tutorials to achieve my goal without success.

This is the error I get (I get it 4 times due to the parallelization, but for more clarity, I put only one occurence):

 [rank0]: Traceback (most recent call last):
 [rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/peft/tuners/lora/model.py", line 360, in __getattr__
 [rank0]:     return super().__getattr__(name)  # defer to nn.Module's logic
 [rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^
 [rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1709, in __getattr__
 [rank0]:     raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
 [rank0]: AttributeError: 'LoraModel' object has no attribute 'prepare_inputs_for_generation'
 
 [rank0]: During handling of the above exception, another exception occurred:
 
 [rank0]: Traceback (most recent call last):
 [rank0]:   File "/project/6045847/user/project/env.py", line 179, in <module>
 [rank0]:     main(args.model_path, data_filename, args.output)
 [rank0]:   File "/project/6045847/user/project/env.py", line 136, in main
 [rank0]:     trainer = SFTTrainer(
 [rank0]:               ^^^^^^^^^^^
 [rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
 [rank0]:     return f(*args, **kwargs)
 [rank0]:            ^^^^^^^^^^^^^^^^^^
 [rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/trl/trainer/sft_trainer.py", line 268, in __init__
 [rank0]:     model = get_peft_model(model, peft_config)
 [rank0]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/peft/mapping.py", line 193, in get_peft_model
 [rank0]:     return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](
 [rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/peft/peft_model.py", line 1610, in __init__
 [rank0]:     self.base_model_prepare_inputs_for_generation = self.base_model.prepare_inputs_for_generation
 [rank0]:                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 [rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/peft/tuners/lora/model.py", line 364, in __getattr__
 [rank0]:     return getattr(self.model, name)
 [rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^
 [rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1709, in __getattr__
 [rank0]:     raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
 [rank0]: AttributeError: 'DistributedDataParallel' object has no attribute 'prepare_inputs_for_generation'

This is my code (I don’t put all, just the code about the model itself for more clarity):

def get_tokenizer(model_id):
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    tokenizer.pad_token = tokenizer.eos_token

    return tokenizer


def get_model(model_id):

    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype="float16", bnb_4bit_use_double_quant=True
    )

    device_index = Accelerator().process_index
    device_map = {"": device_index}

    model = AutoModelForCausalLM.from_pretrained(
        model_id, quantization_config=bnb_config, device_map=device_map
    )
    model.config.use_cache=False
    model.config.pretraining_tp=1
    return model



def main(model_id, data_file, dir_output):

    print("get tokenizer")
    tokenizer = get_tokenizer(model_id)

    print("data")
    raw_data = load_data(data_file)
    training_data, test_data = train_test_split(raw_data, test_size=0.2, random_state=12)

    data = prepare_train_datav2(training_data)

    print("model")
    model = get_model(model_id)

    print("lora")
    peft_config = LoraConfig(
        r=8, lora_alpha=16, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM"
    )
    model = get_peft_model(model, peft_config)

    print("preparation entrainement")
    model.train()

    model.gradient_checkpointing_enable()
    model.enable_input_require_grads()

    accelerator = Accelerator()                 # code added
    model = accelerator.prepare_model(model)    # code added

    training_arguments = SFTConfig(
        output_dir=dir_output,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=64,
        optim="paged_adamw_32bit",
        learning_rate=2e-4,
        lr_scheduler_type="cosine",
        save_strategy="epoch",
        logging_steps=10,
        num_train_epochs=3,
        max_steps=250,
        bf16=True,
        push_to_hub=False
    )

    trainer = SFTTrainer( # the error occurs here
        model=model,
        train_dataset=data,
        peft_config=peft_config,
        dataset_text_field="text",
        args=training_arguments,
        tokenizer=tokenizer,
        packing=False,
        max_seq_length=1024
    )

    print("train")

    trainer.train()

The only code I added between the 1 GPU and 4 GPU versions is:

 accelerator = Accelerator()
 model = accelerator.prepare_model(model)

And I run the code via a bash script:

 accelerate config
 accelerate launch --multi_gpu script.py --model_path $1 --output $2

The 1 GPU version of this script was:

 python script.py --model_path $1 --output $2

I also tried at the end of the main function (deleting ‘model = accelerator.prepare_model(model)’):

 accelerator = Accelerator()
 trainer = accelerator.prepare_model(trainer)

 trainer.train()

But this time I have this error:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/project/6045847/user/project/env.py", line 179, in <module>
[rank0]:     main(args.model_path, data_filename, args.output)
[rank0]:   File "/project/6045847/user/project/env.py", line 149, in main
[rank0]:     trainer = accelerator.prepare_model(trainer)
[rank0]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/accelerate/accelerator.py", line 1380, in prepare_model
[rank0]:     self.verify_device_map(model)
[rank0]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/accelerate/accelerator.py", line 3534, in verify_device_map
[rank0]:     for m in model.modules():
[rank0]:              ^^^^^^^^^^^^^
[rank0]: AttributeError: 'SFTTrainer' object has no attribute 'modules'
Map:   0%|          | 0/960 [00:00<?, ? examples/s]W1105 11:53:02.954000 23261697507392 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2716513 closing signal SIGTERM
W1105 11:53:02.955000 23261697507392 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2716514 closing signal SIGTERM
W1105 11:53:02.955000 23261697507392 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 2716515 closing signal SIGTERM
E1105 11:53:03.434000 23261697507392 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 2716512) of binary: /project/6045847/user/project/env/bin/python
Traceback (most recent call last):
  File "/project/6045847/user/project/env/bin/accelerate", line 8, in <module>
    sys.exit(main())
                     ^^^^^^
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1165, in launch_command
    multi_gpu_launcher(args)
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/accelerate/commands/launch.py", line 799, in multi_gpu_launcher
    distrib_run.run(args)
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
script.py FAILED

So my question is: how is it possible to use accelerate lib with all my previous code ? If you need more information, feel free to ask

John6666 · November 6, 2024, 10:06am

Multi-GPU, PEFT and quantization are a combination that seems quite likely to cause errors…
It’s probably not a simple problem, so I think it’s better to ask on HF Discord than on the forum.
I didn’t get much useful information from the search either.

github.com/huggingface/trl

SFTTrainer not using both GPUs

opened 02:46AM - 01 Feb 24 UTC

closed 10:42PM - 01 Feb 24 UTC

johnowhitaker

I am trying to fine-tune Llama 2 7B with QLoRA on 2 GPUs. From what I've read SF…TTrainer should support multiple GPUs just fine, but when I run this I see one GPU with high utilization and one with almost none: ![Screenshot 2024-01-31 182908](https://github.com/huggingface/trl/assets/6575163/159f6b97-6f9a-47ee-adbc-ef2064ef2f5c) Expected behaviour would be that both get used during training and it would be about 2x as fast as single-GPU training. I'm running this with `python train.py`, which I think means Trainer uses DP? I get an error launching with `python -m torch.distributed.launch train.py` (`RuntimeError: Expected to mark a variable ready only once...`) which makes me think DDP would need a bit more work... This is an older machine without any fast interconnect, but I saw similar usage on a cloud machine with 2xA5000s so I don't think it's that. Anyway, maybe someone can help by explaining why DP might be so slow in this case and/or how to test DDP instead :) Script: ```python from datasets import load_dataset import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AutoTokenizer from peft import LoraConfig from trl import SFTTrainer from transformers import TrainingArguments # Load the dataset dataset_name = "timdettmers/openassistant-guanaco" dataset = load_dataset(dataset_name, split="train") # Load the model + tokenizer model_name = "meta-llama/Llama-2-7b-hf" tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) tokenizer.pad_token = tokenizer.eos_token bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.float16, ) model = AutoModelForCausalLM.from_pretrained( model_name, quantization_config=bnb_config, trust_remote_code=True, use_cache = False ) # PEFT config lora_alpha = 16 lora_dropout = 0.1 lora_r = 64 peft_config = LoraConfig( lora_alpha=lora_alpha, lora_dropout=lora_dropout, r=lora_r, bias="none", task_type="CAUSAL_LM", target_modules=["k_proj", "q_proj", "v_proj", "up_proj", "down_proj", "gate_proj"] ) # Args max_seq_length = 512 output_dir = "./results" per_device_train_batch_size = 8 gradient_accumulation_steps = 2 optim = "adamw_hf" save_steps = 10 logging_steps = 1 learning_rate = 2e-4 max_grad_norm = 0.3 max_steps = 300 # Approx the size of guanaco at bs 8, ga 2, 2 GPUs. warmup_ratio = 0.1 lr_scheduler_type = "cosine" training_arguments = TrainingArguments( output_dir=output_dir, per_device_train_batch_size=per_device_train_batch_size, gradient_accumulation_steps=gradient_accumulation_steps, optim=optim, save_steps=save_steps, logging_steps=logging_steps, learning_rate=learning_rate, fp16=True, max_grad_norm=max_grad_norm, max_steps=max_steps, warmup_ratio=warmup_ratio, group_by_length=True, lr_scheduler_type=lr_scheduler_type, gradient_checkpointing=True, report_to="wandb", ) # Trainer trainer = SFTTrainer( model=model, train_dataset=dataset, peft_config=peft_config, dataset_text_field="text", max_seq_length=max_seq_length, tokenizer=tokenizer, args=training_arguments, ) # Not sure if needed but noticed this in https://colab.research.google.com/drive/1t3exfAVLQo4oKIopQT1SKxK4UcYg7rC1#scrollTo=7OyIvEx7b1GT for name, module in trainer.model.named_modules(): if "norm" in name: module = module.to(torch.float32) # Train :) trainer.train() ```

luxNychta · November 6, 2024, 8:44pm

I tried to do some fixes as discussed on your link (https://discuss.huggingface.co/t/multiple-gpu-in-sfttrainer/91899)

Unfortunately I still have some errors:

[rank1]: Traceback (most recent call last):
[rank1]:   File "/project/6045847/user/project/script.py", line 178, in <module>
[rank1]:     main(args.model_path, data_filename, args.output)
[rank1]:   File "/project/6045847/user/project/script.py", line 150, in main
[rank1]:     trainer.train()
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/trl/trainer/sft_trainer.py", line 434, in train
[rank1]:     output = super().train(*args, **kwargs)
[rank1]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/transformers/trainer.py", line 2052, in train
[rank1]:     return inner_training_loop(
[rank1]:            ^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/transformers/trainer.py", line 2388, in _inner_training_loop
[rank1]:     tr_loss_step = self.training_step(model, inputs)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/transformers/trainer.py", line 3518, in training_step
[rank1]:     self.accelerator.backward(loss, **kwargs)
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/accelerate/accelerator.py", line 2196, in backward
[rank1]:     loss.backward(**kwargs)
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/_tensor.py", line 525, in backward
[rank1]:     torch.autograd.backward(
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank1]:     _engine_run_backward(
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank1]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/autograd/function.py", line 301, in apply
[rank1]:     return user_fn(self, *args)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 320, in backward
[rank1]:     torch.autograd.backward(outputs_with_grad, args_with_grad)
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank1]:     _engine_run_backward(
[rank1]:   File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank1]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
[rank1]: Parameter at index 127 with name base_model.model.model.layers.31.self_attn.v_proj.lora_B.default.weight has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration.
W1106 12:34:58.957000 22647939338304 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3711548 closing signal SIGTERM
W1106 12:34:58.958000 22647939338304 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3711550 closing signal SIGTERM
W1106 12:34:58.958000 22647939338304 torch/distributed/elastic/multiprocessing/api.py:851] Sending process 3711551 closing signal SIGTERM
E1106 12:34:59.240000 22647939338304 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 1 (pid: 3711549) of binary: /project/6045847/user/project/env/bin/python
Traceback (most recent call last):
  File "/project/6045847/user/project/env/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1165, in launch_command
    multi_gpu_launcher(args)
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/accelerate/commands/launch.py", line 799, in multi_gpu_launcher
    distrib_run.run(args)
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/6045847/user/project/env/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================

This is my code now:


def get_model(model_id):

	....

	device_string = PartialState().process_index

    model = AutoModelForCausalLM.from_pretrained(
        model_id, quantization_config=bnb_config, device_map=device_string
    )

    ....


.....

def main(model_id, data_file, dir_output):

	....

	training_arguments = SFTConfig(
		....
		gradient_checkpointing_kwargs={'use_reentrant':False},
        gradient_checkpointing=False
    )

    ....

I removed


	accelerator = Accelerator()
	model = accelerator.prepare_model(model)

and I modified my bash script:


accelerate config
accelerate launch --multi_gpu --num_processes 4 script.py --model_path $1 --output $2

John6666 · November 7, 2024, 1:02am

I found an issue that is in progress by searching for error messages.

github.com/pytorch/torchtune

torch.distributed.elastic.multiprocessing.errors.ChildFailedError

opened 02:32AM - 29 Sep 24 UTC

Vattikondadheeraj

bug

Context :- I am trying to run distributed training on 2 A-100 gpus with 40GB of …VRAM. The batch size is 3 and gradient accumulation=1. I have attached the config file below for more details and the error as well. I thing is I am not able to pinpoint the problem here because the error message itself is unclear. Is this because of CUDA memory issue? Or am I missing something else? ``` INFO:torchtune.utils._logging:Running FullFinetuneRecipeDistributed with resolved config: batch_size: 3 checkpointer: _component_: torchtune.training.FullModelMetaCheckpointer checkpoint_dir: /home/toolkit/scratch/LLMcode/Train/llama-3.1-instruct/original checkpoint_files: - consolidated.00.pth model_type: LLAMA3 output_dir: /home/toolkit/scratch/LLMcode/Checkpoints/full_finetuning_results-1 recipe_checkpoint: null custom_sharded_layers: - tok_embeddings - output dataset: _component_: torchtune.datasets.alpaca_dataset device: cuda dtype: bf16 enable_activation_checkpointing: true epochs: 10 gradient_accumulation_steps: 1 log_every_n_steps: 1 log_peak_memory_stats: true loss: _component_: torchtune.modules.loss.CEWithChunkedOutputLoss max_steps_per_epoch: null metric_logger: _component_: torchtune.training.metric_logging.DiskLogger log_dir: /home/toolkit/scratch/LLMcode/Checkpoints/full_finetuning_results-1 model: _component_: torchtune.models.llama3_1.llama3_1_8b optimizer: _component_: torch.optim.AdamW fused: true lr: 2.0e-05 output_dir: /home/toolkit/scratch/LLMcode/Checkpoints/full_finetuning_results-1 resume_from_checkpoint: false seed: null shuffle: true tokenizer: _component_: torchtune.models.llama3.llama3_tokenizer max_seq_len: null path: /home/toolkit/scratch/LLMcode/Train/llama-3.1-instruct/original/tokenizer.model DEBUG:torchtune.utils._logging:Setting manual seed to local seed 4088951359. Local seed is seed + rank = 4088951359 + 0 Writing logs to /home/toolkit/scratch/LLMcode/Checkpoints/full_finetuning_results-1/log_1727576992.txt INFO:torchtune.utils._logging:FSDP is enabled. Instantiating model and loading checkpoint on Rank 0 ... INFO:torchtune.utils._logging:Instantiating model and loading checkpoint took 17.02 secs INFO:torchtune.utils._logging:Memory stats after model init: GPU peak memory allocation: 8.52 GiB GPU peak memory reserved: 8.67 GiB GPU peak memory active: 8.52 GiB INFO:torchtune.utils._logging:Optimizer is initialized. INFO:torchtune.utils._logging:Loss is initialized. INFO:torchtune.utils._logging:Dataset and Sampler are initialized. WARNING:torchtune.utils._logging: Profiling disabled. INFO:torchtune.utils._logging: Profiler config after instantiation: {'enabled': False} 441 aux1 aux2 0%| | 0/70 [00:00<?, ?it/s]765 aux1 aux1 /home/toolkit/.conda/envs/torch/lib/python3.11/site-packages/torch/utils/checkpoint.py:1399: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead. with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context: # type: ignore[attr-defined] /home/toolkit/.conda/envs/torch/lib/python3.11/site-packages/torch/utils/checkpoint.py:1399: FutureWarning: `torch.cpu.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cpu', args...)` instead. with device_autocast_ctx, torch.cpu.amp.autocast(**cpu_autocast_kwargs), recompute_context: # type: ignore[attr-defined] W0929 02:30:29.754000 139982797317952 torch/distributed/elastic/multiprocessing/api.py:858] Sending process 2171 closing signal SIGTERM E0929 02:30:36.777000 139982797317952 torch/distributed/elastic/multiprocessing/api.py:833] failed (exitcode: -9) local_rank: 1 (pid: 2172) of binary: /home/toolkit/.conda/envs/torch/bin/python Traceback (most recent call last): File "/home/toolkit/.conda/envs/torch/bin/tune", line 8, in <module> sys.exit(main()) ^^^^^^ File "/home/toolkit/scratch/LLMcode/Train/torchtune/torchtune/_cli/tune.py", line 49, in main parser.run(args) File "/home/toolkit/scratch/LLMcode/Train/torchtune/torchtune/_cli/tune.py", line 43, in run args.func(args) File "/home/toolkit/scratch/LLMcode/Train/torchtune/torchtune/_cli/run.py", line 183, in _run_cmd self._run_distributed(args) File "/home/toolkit/scratch/LLMcode/Train/torchtune/torchtune/_cli/run.py", line 89, in _run_distributed run(args) File "/home/toolkit/.conda/envs/torch/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in run elastic_launch( File "/home/toolkit/.conda/envs/torch/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 133, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/toolkit/.conda/envs/torch/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /home/toolkit/scratch/LLMcode/Train/torchtune/recipes/full_finetune_distributed.py FAILED ------------------------------------------------------------ Failures: <NO_OTHER_FAILURES> ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2024-09-29_02:30:29 host : 92b61c39-4dd9-4911-be6f-522a27802a4a rank : 1 (local_rank: 1) exitcode : -9 (pid: 2172) error_file: <N/A> traceback : Signal 9 (SIGKILL) received by PID 2172 ============================================================ ``` Config File :- ``` tokenizer: _component_: torchtune.models.llama3.llama3_tokenizer path: /home/toolkit/scratch/LLMcode/Train/llama-3.1-instruct/original/tokenizer.model max_seq_len: null # Dataset dataset: _component_: torchtune.datasets.alpaca_dataset seed: null shuffle: True # Model Arguments model: _component_: torchtune.models.llama3_1.llama3_1_8b checkpointer: _component_: torchtune.training.FullModelMetaCheckpointer checkpoint_dir: /home/toolkit/scratch/LLMcode/Train/llama-3.1-instruct/original checkpoint_files: [ consolidated.00.pth, ] recipe_checkpoint: null output_dir: /home/toolkit/scratch/LLMcode/Checkpoints/full_finetuning_results-1 model_type: LLAMA3 resume_from_checkpoint: False # Fine-tuning arguments batch_size: 3 epochs: 10 optimizer: _component_: torch.optim.AdamW lr: 2e-5 fused: True loss: _component_: torchtune.modules.loss.CEWithChunkedOutputLoss max_steps_per_epoch: null gradient_accumulation_steps: 1 # Training env device: cuda # Memory management enable_activation_checkpointing: True custom_sharded_layers: ['tok_embeddings', 'output'] # Reduced precision dtype: bf16 # Logging metric_logger: _component_: torchtune.training.metric_logging.DiskLogger log_dir: /home/toolkit/scratch/LLMcode/Checkpoints/full_finetuning_results-1 output_dir: /home/toolkit/scratch/LLMcode/Checkpoints/full_finetuning_results-1 log_every_n_steps: 1 log_peak_memory_stats: True ```

Topic		Replies	Views
Issue with LoRA Adapter Loading on Multiple GPUs during Fine-Tuning with Accelerate and SFTTrainer 🤗Accelerate	3	150	September 18, 2024
Using evaluate.evaluator on a PEFT model Beginners	1	184	April 10, 2024
Accelerator() causes Error Beginners	2	253	April 12, 2024
Errors when using gradient accumulation with FSDP + PEFT LoRA + SFTTrainer 🤗Accelerate	1	93	September 6, 2024
Error when training with `peft` + `lora` 🤗Transformers	1	1258	August 25, 2023

Error while fine tuning with peft, lora, accelerate, SFTConfig and SFTTrainer

Related Topics