Training on 'free' Googe Colab

🤗Accelerate
1

I am pretty sure this question has been answered before, but I could not find it.

Basically, I am learning how to train a Bert classifier from scratch to classify a set of e-mails as ‘spam’ or ‘not spam’ on Google colab using T4 GPU

Most of the code is pretty basic and standard, i.e., loading ‘sms_spam’ data set, tokening loading the model (“distilbert-base-uncased”), and Unfreezing all the model parameters.

And here is the instance of the trainer class:

import numpy as np
from transformers import DataCollatorWithPadding, Trainer, TrainingArguments
from accelerate import Accelerator

accelerator = Accelerator()

def compute_metrics(eval_pred):
predictions, labels = eval_pred
predictions = np.argmax(predictions, axis=1)
return {“accuracy”: (predictions == labels).mean()}

The HuggingFace Trainer class handles the training and eval loop for PyTorch for us.

Read more about it here Trainer

trainer = Trainer(
model=model,
args=TrainingArguments(
output_dir=“/content/drive/My Drive/DATA/spam_not_spam”,
learning_rate=2e-3,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
evaluation_strategy=“epoch”,
save_strategy=“epoch”,
num_train_epochs=2,
weight_decay=0.01,
load_best_model_at_end=True,
),

train_dataset=tokenized_dataset["train"],
eval_dataset=tokenized_dataset["test"],
tokenizer=tokenizer,
data_collator=DataCollatorWithPadding(tokenizer=tokenizer),
compute_metrics=compute_metrics,

accelerator=accelerator,
)

trainer.train()

Note: First time, I did not have ‘accelerate’. Colab threw an error and suggested I install accelerator and create an instance which I did with “!pip install accelerate -U”

But even after that, I keep getting the following error:

<<

ImportError Traceback (most recent call last)
in <cell line: 18>()
18 trainer = Trainer(
19 model=model,
—> 20 args=TrainingArguments(
21 # output_dir=“./data/spam_not_spam”,
22 output_dir=“/content/drive/My Drive/DATA/spam_not_spam”,

4 frames
/usr/local/lib/python3.10/dist-packages/transformers/training_args.py in _setup_devices(self)
1785 if not is_sagemaker_mp_enabled():
1786 if not is_accelerate_available(min_version=“0.20.1”):
→ 1787 raise ImportError(
1788 “Using the Trainer with PyTorch requires accelerate>=0.20.1: Please run pip install transformers[torch] or pip install accelerate -U”
1789 )

ImportError: Using the Trainer with PyTorch requires accelerate>=0.20.1: Please run pip install transformers[torch] or pip install accelerate -U

Can anyone offer any suggestions on what I am doing wrong and how to fix this? Cheers

2

You need to restart your runtime as well

3

@muellerzr, Can you elaborate please? At what point do I need to restart the runtime?

4

After installing accelerate (Or just make it the first thing you do)

5

Hi,

https://huggingface.co/docs/accelerate/basic_tutorials/notebook

https://huggingface.co/docs/accelerate/v0.27.2/en/package_reference/accelerator#accelerate.Accelerator.prepare