ValueError: Using fsdp only works in distributed training

🤗Transformers
1

I am getting an error when launching the TRL trainer with fsdp. Attaching complete configuration below:

from transformers import TrainingArguments

output_dir = model_output_dir
per_device_train_batch_size = 6
gradient_accumulation_steps = 8
optim = "paged_adamw_32bit"
save_steps = 50
save_total_limit=3
logging_steps = 10
learning_rate = 2e-4
max_grad_norm = 0.3
warmup_ratio = 0.03
lr_scheduler_type = "cosine_with_restarts"
max_steps = 8000
group_by_length = True
do_fsdp_training = "full_shard"

training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=per_device_train_batch_size,
    gradient_accumulation_steps=gradient_accumulation_steps,
    optim=optim,
    save_steps=save_steps,
    save_total_limit=save_total_limit,
    logging_steps=logging_steps,
    learning_rate=learning_rate,
    fp16=True,
    max_grad_norm=max_grad_norm,
    max_steps=max_steps,
    warmup_ratio=warmup_ratio,
    # group_by_length=group_by_length,
    lr_scheduler_type=lr_scheduler_type,
    report_to = "tensorboard",
    fsdp= do_fsdp_training
)

math_qa_data_collator = DataCollatorWithPadding(
    tokenizer,
    return_tensors = "pt"
)

from trl import SFTTrainer

max_seq_length = 1024

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    args=training_arguments,
    peft_config=peft_config,
    train_dataset=math_qa_dataset["train"],
    formatting_func = dataset_formatting_func,
    max_seq_length=max_seq_length,
    data_collator=math_qa_data_collator,
    packing=True,
)

Stacktrace Snippet

File /usr/local/lib/python3.8/dist-packages/transformers/trainer.py:455, in Trainer.__init__(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics)
    451     raise ValueError(
    452         "Using --fsdp xxx together with --deepspeed is not possible, deactivate one of those flags."
    453     )
    454 if not args.fsdp_config["xla"] and args.parallel_mode != ParallelMode.DISTRIBUTED:
--> 455     raise ValueError("Using fsdp only works in distributed training.")
    457 # dep_version_check("torch>=1.12.0")
    458 # Would have to update setup.py with torch>=1.12.0
    459 # which isn't ideally given that it will force people not using FSDP to also use torch>=1.12.0
    460 # below is the current alternative.
    461 if version.parse(version.parse(torch.__version__).base_version) < version.parse("1.12.0"):

ValueError: Using fsdp only works in distributed training.
2

@nielsr @muellerzr can you please help us here? I’m also facing the same issue.

I’m trying gemma model by following this example https://huggingface.co/google/gemma-7b/blob/main/examples/example_fsdp.py

However, I’m using Nvidia GPU

import torch
from datasets import load_dataset
from peft import LoraConfig, get_peft_model
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments
from trl import SFTTrainer

model_id = "google/gemma-7b"

# Load the pretrained model and tokenizer.
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map={"":0})

# Set up PEFT LoRA for fine-tuning.
lora_config = LoraConfig(
    r=8,
    target_modules=["k_proj", "v_proj"],
    task_type="CAUSAL_LM",
)

# Load the dataset and format it for training.
data = load_dataset("Abirate/english_quotes", split="train")
max_seq_length = 1024

# Set up the FSDP config. To enable FSDP via SPMD, set xla_fsdp_v2 to True.
fsdp_config = {"fsdp_transformer_layer_cls_to_wrap": [
        "GemmaDecoderLayer"
    ]}

# Finally, set up the trainer and train the model.
trainer = SFTTrainer(
    model=model,
    train_dataset=data,
    args=TrainingArguments(
        per_device_train_batch_size=64,  # This is actually the global batch size for SPMD.
        num_train_epochs=100,
        max_steps=-1,
        output_dir="./output",
        optim="adafactor",
        logging_steps=1,
        dataloader_drop_last = True,  # Required for SPMD.
        fsdp="full_shard",
        fsdp_config=fsdp_config,
    ),
    peft_config=lora_config,
    dataset_text_field="quote",
    max_seq_length=max_seq_length,
    packing=True,
)

trainer.train()
3

Hi,

How many GPUs do you have? You need at least 2 as FSDP is meant for distributed training.

4

Hello, I’m also trying with Gemma and I have 4 GPUs, and it still wont work for me

5

@LidorPrototype I think you need to init a config file first, as explained in the docs for FSPD training. Also there is an example script here

6

I followed those docs, and created my fsdp config file, and again when I tried to run my trainer with trainer.train() I got that error ValueError: Using fsdp only works in distributed training.

Here is my TrainingArguments:

training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    optim="paged_adamw_8bit", # paged_adamw_32bit
    learning_rate=0.0002, # 0.001
    fp16=True,
    max_grad_norm=0.3,
    warmup_ratio=0.03,
    lr_scheduler_type="constant",
    logging_steps=LOGS,
    save_steps=SAVES,
    max_steps=EPOCHS,
    disable_tqdm=True,
    fsdp="full_shard",
    fsdp_config="fsdp_config.json"
)
7

Having the same issue with Gemma and 8x Nvidia H100 GPUs. Code is based on the same example_fsdp.py

Accelerator test returns success:

> accelerate test
...
Test is a success! You are ready for your distributed training!

The same script works on TPUs using GCP, with added

fsdp_config = {
    "fsdp_transformer_layer_cls_to_wrap": [
        "GemmaDecoderLayer"
    ],
    "xla": True,
    "xla_fsdp_v2": True,
    "xla_fsdp_grad_ckpt": True
}