Training the t5

Beginners
1

Hello, on the T5 page it says to train it using just

loss = model(input_ids=input_ids, lm_labels=labels).loss
This did nothing but give me a loss value.

However when i do this Summarization
I get the error text, and summary are not recognized.

2

Could you please show your code and the respective error?

3

from datasets import load_dataset
from transformers import T5Tokenizer, T5ForConditionalGeneration
from transformers import AutoModelForSeq2SeqLM, Seq2SeqTrainingArguments, Seq2SeqTrainer

billsum = load_dataset(“billsum”, split=“ca_test”)
billsum = billsum.train_test_split(test_size=0.2)
from transformers import AutoTokenizer

tokenizer = T5Tokenizer.from_pretrained(“google/t5-efficient-xxl-nl4”)
model = AutoModelForSeq2SeqLM.from_pretrained(“/media/samuel/New Volume/models/t5-efficient-xxl-nl4”)

prefix = "summarize: "

def preprocess_function(examples):
inputs = [prefix + doc for doc in examples[“text”]]
model_inputs = tokenizer(inputs, max_length=1024, truncation=True)

with tokenizer.as_target_tokenizer():
    labels = tokenizer(examples["summary"], max_length=128, truncation=True)

model_inputs["labels"] = labels["input_ids"]
return model_inputs

tokenized_billsum = billsum.map(preprocess_function, batched=True)

from transformers import DataCollatorForSeq2Seq

data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)

training_args = Seq2SeqTrainingArguments(
output_dir=“./results”,
evaluation_strategy=“epoch”,
learning_rate=2e-5,
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
weight_decay=0.01,
save_total_limit=3,
num_train_epochs=1,
fp16=True,
)

trainer = Seq2SeqTrainer(
model=model,
args=training_args,
train_dataset=tokenized_billsum[“train”],
eval_dataset=tokenized_billsum[“test”],
tokenizer=tokenizer,
data_collator=data_collator,
)

trainer.train()

Errior:

The following columns in the training set don’t have a corresponding argument in T5ForConditionalGeneration.forward and have been ignored: title, summary, text. If title, summary, text are not expected by T5ForConditionalGeneration.forward, you can safely ignore this message.

4

I believe that error is fine but it doesn’t change my model when i train on this.
the training loss is 0.0
and the eval loss is nan

5

I figured it out. you get the loss value from the model and call

loss = model(input_ids=input_ids, lm_labels=labels).loss
optimizer.zero_grad()
loss.backward()
optimizer.step()

I just figure it would do this part for you