Fine tune Transformers for text generation

🤗Transformers
1

Hello

Is there an example like this one (Fine-tune a pretrained model) for fine tuning HF transformers for text generation?

1 Like
2

@mwitiderrick Hello :hugs:
You can check out this link for all example notebooks.

3

Hello @merve, thanks for the response, indeed I found the notebooks very useful. One follow-up question.

When I run predictions like this for a binary problem

import tensorflow as tf
predicted_class_id = int(tf.math.argmax(logits, axis=-1)[0])
bert.config.id2label[predicted_class_id]

I get the result as LABEL_1, How do I know if this is the prediction for class 0 or 1.

Thanks.

4

Hello Derrick,

Can you send me the model repo so that I can see the config file?

5

Hello @merve

Not sure about the repo but the model is

from transformers import TFAutoModelForSequenceClassification
model = TFAutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
6

Hello Derrick,

Sorry it’s my fault, label is already a label :sweat_smile: I meant, which dataset is the model fine-tuned on?

7

Hello @merve
It’s imdb

dataset = load_dataset("imdb")
8

Any update on this @merve ?

9

@mwitiderrick In the page of the dataset you can see the label for 1 is positive.

10

Wanted to clarify that LABEL_1 means label 1 and not 0

11

hi @mwitiderrick, which HF notebook did you use for fine-tuning a model for text generation? thank you, Tom

12

Hi @mwitiderrick I am a beginner I was trying to fine-tune the model to generate text I will be a great help if you will be able to provide your notebook to me.
I am ready to use any model of LlamaForCasualLM architecture any dataset will be fine for me I just want to understand the Concept and practical Implementation Behind that.