Fine-Tuning AutoModelWithLMHead Model

🤗Transformers
1

Hi everyone,

I want to fine-tune the AutoModelWithLMHead model from this repository, which is a German GPT-2 model.

I have prepocessed a bunch of text passages for the fine-tuning, but when beginning training, I receive the following error (copied with a little context):

File "GPT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "GPT\lib\site-packages\transformers\models\gpt2\modeling_gpt2.py", line 774, in forward
    raise ValueError("You have to specify either input_ids or inputs_embeds")
ValueError: You have to specify either input_ids or inputs_embeds

It’s asking for either input ids or embeddings, which I thought I provided by instantiating the trainer. Here’s my code for the preparation of the model:

# Load data
with open("Fine-Tuning Dataset/train.txt", "r", encoding="utf-8") as train_file:
    train_data = train_file.read().split("--")

with open("Fine-Tuning Dataset/test.txt", "r", encoding="utf-8") as test_file:
    test_data = test_file.read().split("--")

# Load pre-trained tokenizer and prepare input
tokenizer = AutoTokenizer.from_pretrained('dbmdz/german-gpt2')

tokenizer.pad_token = tokenizer.eos_token
train_input = tokenizer(train_data, padding="longest")
test_input = tokenizer(test_data, padding="longest")

# Define model

model = AutoModelWithLMHead.from_pretrained("dbmdz/german-gpt2")
training_args = TrainingArguments("test_trainer")

# Evaluation

metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = numpy.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

# Train
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_input,
    eval_dataset=test_input,
    compute_metrics=compute_metrics,
)
trainer.train()
trainer.evaluate()

Does anyone know the cause for this? Any help is gladly appreciated! Thank you.

2

It looks like your train_input is not a dataset containing the "input_ids", as expected by the model. Look at train_input[0] for instance to see which keys it contains.