laurb
November 5, 2024, 7:39pm
1
I can’t find any examples using the num_items_in_batch parameter to the loss function that appears in the latest transformer version. I want to implement a custom loss_function that biases my model towards precision. Do i need to downgrade tranformers to 4.25.2, or are there examples anywhere of how to use this new parameter? If I just ignore it, I end up with mismatched tensor sizes.
1 Like
It is possible that the problem has already been solved in the dev version.
opened 08:07PM - 24 Oct 24 UTC
🐛 bug
🏋 DPO
### System Info
System Info
Python version: 3.11.0
PyTorch vers… ion: 2.4.1 or 2.5.0
Transformers version: 4.46.0
TRL version: 0.11.4
PEFT version: 0.13.2
### Information
- [ ] The official example scripts
- [X] My own modified scripts
### Tasks
- [ ] An officially supported task in the `examples` folder
- [X] My own task or dataset (give details below)
### Reproduction
Code to reproduce the error (with transformers==4.46.0 and trl==0.11.4)
```python
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen2.5-0.5B",
device_map="auto",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
config = LoraConfig(
r = 16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type= "CAUSAL_LM"
)
model = get_peft_model(model, config)
data_dpo = load_dataset("CultriX/llama70B-dpo-dataset")
def preprocess_data_dpo(data_point):
system = data_point['system']
question = data_point['question']
chosen = data_point['chosen']
rejected = data_point['rejected']
features = {
"prompt": f"{question}",
"chosen": f"{chosen}",
"rejected": f"{rejected}"
}
return features # fill the gap, using dpo format
data_dpo = data_dpo['train'].shuffle(seed=42).map(preprocess_data_dpo)
training_args = DPOConfig(
per_device_train_batch_size=1,
gradient_accumulation_steps=4,
num_train_epochs=1,
learning_rate=2e-4,
fp16=True,
save_total_limit=3,
logging_steps=1,
output_dir="output_dir",
max_steps=200,
optim="paged_adamw_8bit",
lr_scheduler_type="cosine",
warmup_ratio=0.05,
report_to="tensorboard",
)
trainer = DPOTrainer(
model=model,
args = training_args,
train_dataset = data_dpo,
tokenizer=tokenizer,
)
trainer.train()
```
This error is raised:
```
Traceback (most recent call last):
File "dpo_lora.py", line 70, in <module>
trainer.train()
^^^^^^^^^^^^^^^
File ".venv/env/lib/python3.11/site-packages/transformers/trainer.py", line 2122, in train
return inner_training_loop(
^^^^^^^^^^^^^^^^^^^^
File ".venv/env/lib/python3.11/site-packages/transformers/trainer.py", line 2426, in _inner_training_loop
batch_samples, num_items_in_batch = self.get_batch_samples(epoch_iterator, num_batches)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".venv/env/lib/python3.11/site-packages/trl/trainer/dpo_trainer.py", line 1508, in get_batch_samples
policy_output = model.generate(
^^^^^^^^^^^^^^
AttributeError: 'generator' object has no attribute 'generate'
```
### Expected behavior
It seems to be due to the recent commit 6ba31a8 on Oct 17, 2024 "Enable users to use their own loss functions + deal with prefetching for grad accum (https://github.com/huggingface/transformers/pull/34198)" on Transformers 4.46.0
where a new method get_batch_samples is defined and used by _inner_training_loop
But the subclass DPOTrainer overwrites the method get_batch_samples with a different signature (and output).
Error can be avoided with Transformers==4.45.2 and trl==0.11.4
pip install git+https://github.com/huggingface/transformers
laurb
November 6, 2024, 12:51pm
3
thanks, but I don’t think this resolves my question. That new ‘num_items_in_batch’ parameter must have to be used somehowwhen I define my loss function, but there are no indications of how to use it.
1 Like
I couldn’t find any other information…
if self.use_apex:
with amp.scale_loss(loss, self.optimizer) as scaled_loss:
scaled_loss.backward()
else:
self.accelerator.backward(loss, **kwargs)
# Finally we need to normalize the loss for reporting
if num_items_in_batch is None:
return loss.detach() / self.args.gradient_accumulation_steps
return loss.detach()
def compute_loss(self, model, inputs, return_outputs=False, num_items_in_batch=None):
"""
How the loss is computed by Trainer. By default, all models return the loss in the first element.
Subclass and override for custom behavior.
"""
if (self.label_smoother is not None or self.compute_loss_func is not None) and "labels" in inputs:
labels = inputs.pop("labels")
else:
labels = None
if self.model_accepts_loss_kwargs: