In the Finetuning chapter, the article on Full-training isn’t available on Tensorflow.
Try it out! Replicate the preprocessing on the GLUE SST-2 dataset. It’s a little bit different since it’s composed of single sentences instead of pairs, but the rest of what we did should look the same. For a harder challenge, try to write a preprocessing function that works on any of the GLUE tasks.
How many types do the GLUE tasks have? single sentences/ pairs/three?
from transformers import TrainingArguments
training_args = TrainingArguments(“test-trainer”)
ImportError Traceback (most recent call last)
in <cell line: 3>()
1 from transformers import TrainingArguments
----> 3 training_args = TrainingArguments(“test-trainer”)
/usr/local/lib/python3.10/dist-packages/transformers/training_args.py in _setup_devices(self)
1770 if not is_sagemaker_mp_enabled():
1771 if not is_accelerate_available(min_version = “0.20.1”):
→ 1772 raise ImportError(
1773 “Using the
accelerate>=0.20.1: Please run
pip install transformers[torch] or
pip install accelerate -U”
ImportError: Using the
accelerate>=0.20.1: Please run
pip install transformers[torch] or
pip install accelerate -U
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
“Open Examples” button below.
I want to fine tune a summarization model on a custom dataset. Are there any guidelines around how much data I would need, will data from a different domain help, etc.?
I am trying to summarize conversations. In most cases, these conversations will involve just two people. I finetuned google/flan-t5-base and facebook/bart-large-cnn on about 1000 examples, results are good but not as good as GPT-3.5.
Do I need to gather and train on more data? If I don’t have access to data for my use case, can I use data from any other domain as long as they are conversations? Say, from podcasts? For how long do I train the model for? Are there any best practices around choosing number of epochs, etc.?
I am looking to improve the performance of my model and can really use some help! I have looked online but can’t find a clear answer. I understand that in a lot of cases, you need to experiment what works for you but there are so many possibilities and I am looking for a starting point, as a beginner in this field.
Thank you for your help!
When i run the trainer.train(), it comes with the following error:
TypeError: ‘NoneType’ object is not callable.
The attention_masks returned by tokenize_function will be equal to the sequence length passed. Also, it will all be 1’s. DataCollatorWithPadding will add 0’s to the attention_mask based on the longest sequence in the batch.
- based on the example in the chapter.
I am working on the last ‘Try out!’ in Chapter 3 section ‘Fine-tuning a model with the Trainer API’. Everything goes fine until the code line ‘trainer.train()’ as it is showed in the image.
Please help me to solve this error.
In Chapter 3, section '‘Fine-tuning a model with the Trainer API’. ’ when I run into the following error when I istantiate the TrainingArguments Class.
The accelerate module is already installed in the requested version.
The issue happens in the linked colab exercise notebook.
Any idea how to fix it?
--------------------------------------------------------------------------- ImportError Traceback (most recent call last) <ipython-input-3-11170ce17e38> in <cell line: 3>() 1 from transformers import TrainingArguments 2 ----> 3 training_args = TrainingArguments("test-trainer") 4 frames /usr/local/lib/python3.10/dist-packages/transformers/training_args.py in _setup_devices(self) 1799 if not is_sagemaker_mp_enabled(): 1800 if not is_accelerate_available(min_version="0.20.1"): -> 1801 raise ImportError( 1802 "Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U`" 1803 ) ImportError: Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U` --------------------------------------------------------------------------- NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt. To view examples of installing some common dependencies, click the "Open Examples" button below.
Have you finish this?
Hi! I am not understanding why we should tokenize the dataset using map and also using it as the trainer argument tokenizer. What is the behavior of the tokenizer argument in the Trainer?
I am trying to do the transformers course, and am running into trouble in the lesson “Fine-tuning a model with the Trainer API.” I am running it on a free Google Colab instance with a T4 GPU. All of the provided code works (I had to add a
!pip install transformers torch after the
!pip install datasets evaluate transformers[sentencepiece] to make it work), but at the bottom, we were told to " Fine-tune a model on the GLUE SST-2 dataset, using the data processing you did in section 2." Here, I ran into a strange error.
Below is the code I was trying to use for this exercise. Each code block is put into its own block here.
single_sentence_dataset = load_dataset("glue", "sst2") single_sentence_dataset
checkpoint = "bert-base-uncased" tokenizer = AutoTokenizer.from_pretrained(checkpoint)
def tokenize_single_sentence_function(example): return tokenizer(example["sentence"], truncation=True) tokenized_single_sentence_datasets = single_sentence_dataset.map(tokenize_single_sentence_function, batched=True) single_sentence_data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
single_sentence_training_args = TrainingArguments("sst2-trainer", evaluation_strategy="epoch")
from transformers import AutoModelForSequenceClassification single_sentence_model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
def compute_metrics_single_sentence(eval_preds): metric = evaluate.load("glue", "sst2") logits, labels = eval_preds predictions = np.argmax(logits, axis=-1) return metric.compute(predictions=predictions, references=labels)
trainer = Trainer( single_sentence_model, single_sentence_training_args, train_dataset=tokenized_single_sentence_datasets["train"], eval_dataset=tokenized_single_sentence_datasets["validation"], data_collator=single_sentence_data_collator, tokenizer=tokenize_single_sentence_function, compute_metrics=compute_metrics_single_sentence )
Everything runs, except for
trainer.train(). When I call that, it gets to
[ 501/25257 00:46 < 38:17, 10.77 it/s, Epoch 0.06/3] (what is the 25257 from, anyways? There are 67349 training examples here) before crashing with the following error:
/usr/local/lib/python3.10/dist-packages/transformers/trainer.py in _maybe_log_save_evaluate(self, tr_loss, model, trial, epoch, ignore_keys_for_eval) 2280 2281 if self.control.should_save: -> 2282 self._save_checkpoint(model, trial, metrics=metrics) 2283 self.control = self.callback_handler.on_save(self.args, self.state, self.control) 2284 /usr/local/lib/python3.10/dist-packages/transformers/trainer.py in _save_checkpoint(self, model, trial, metrics) 2348 run_dir = self._get_output_dir(trial=trial) 2349 output_dir = os.path.join(run_dir, checkpoint_folder) -> 2350 self.save_model(output_dir, _internal_call=True) 2351 if self.is_deepspeed_enabled: 2352 # under zero3 model file itself doesn't get saved since it's bogus! Unless deepspeed /usr/local/lib/python3.10/dist-packages/transformers/trainer.py in save_model(self, output_dir, _internal_call) 2841 2842 elif self.args.should_save: -> 2843 self._save(output_dir) 2844 2845 # Push to the Hub when `save_model` is called by the user. /usr/local/lib/python3.10/dist-packages/transformers/trainer.py in _save(self, output_dir, state_dict) 2904 2905 if self.tokenizer is not None: -> 2906 self.tokenizer.save_pretrained(output_dir) 2907 2908 # Good practice: save your training arguments together with the trained model AttributeError: 'function' object has no attribute 'save_pretrained'
compute_metrics argument does not change anything.
Does anyone know what is going on? I am not explicitly telling it to save anything. Why is this failing when the provided code works?
I was able to train a model for SST2 successfully without using the trainer API in the next lesson. Why does it not work here?
I fixed the issue. You need to pass the tokenizer, and not the tokenizer function, into Trainer. Also, I had to set
save_steps=0.25 in the
TrainingArguments to prevent the training process from filling up my Google Colab’s disk by storing weights every 500 steps.
When running the pytorch version of the “Fine-Tuning a model notebook”(link)[Google Colab], I’m getting this error:
"Using the `Trainer` with `PyTorch` requires `accelerate>=0.20.1`: Please run `pip install transformers[torch]` or `pip install accelerate -U`"
I tried pip installing as the error suggests but no luck. Any ideas would be appreciated!
Awesome courses, btw! Thanks!