AttributeError: 'NoneType' object has no attribute 'dtype'

Intermediate
1

Hello!

I’ve been trying to fine-tune a german gpt-2 Modell with a dataset from huggingface via TensorFlow.
When trying to start the Training, I get the following Error: AttributeError: ‘NoneType’ object has no attribute ‘dtype’.

You can find the code on this Google Colab Notebook: Google Colab

Any help would be appreciated and thanks for reading!

1 Like
2

Same boat here. If it helps, I was getting my error at loss and so I printed it and got None. This is my case. Perhaps it’s not computing for some reason.

`use_cache = True` is incompatible with gradient checkpointing`. Setting `use_cache = False`...
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_3879877/1790787526.py in <cell line: 7>()
     14         loss = outputs.loss
     15         print(loss)
---> 16         loss.backward()
     17 
     18         optimizer.step()

AttributeError: 'NoneType' object has no attribute 'backward'
3

Sometimes, this happens in the case that the function does not return anything. You can try to check whether forget to return something, such as return loss

1 Like
4

Hm, yes. In my case, there was no 'label' column so there was no output, and thus no loss. I’m wondering if these features (label, target…) are task-dependent and why they would be needed for generation tasks, as supposed to classification?

5

Generation task training also needs the label as generation task training is supervised learning here.

1 Like
6

I see. Thank you! So when training, would that be responsible in part of my dataset (to have “labels”) or the tokenizer to create a column called [‘labels’] alongside the others (attn_mask…)?

7

Attribute errors in Python are generally raised when you try to access or call an attribute that a particular object type doesn’t possess. It’s simply because there is no attribute with the name you called, for that Object. This means that you got the error when the “module” does not contain the method you are calling. But it is evident that the method is there, which leads to believe that may be the method was added by you in the source code after you had already imported the file (module). Or, some times packages get deprecated and they rename some functions. If that is true, then you may want to exit and reimport the module once again to be able to access the new method .

You can do it in another way to reimport the module with changes without having to exit the interpreter is to do the following:

reload(myModule)

If you are using python 3.2 or 3.3 you should:

import imp
imp.reload(myModule)

If running Python 3.4 and up, do import importlib, then do:

import importlib
importlib.reload(myModule)

The importlib.reload() method reload a previously imported module. The argument must be a module object, so it must have been successfully imported before .

8

Hi @jhaeub,
I encountered the same issue (refer to Fine-tune a German GPT-2 Model with Tensorflow in Transformers for text generation (NLP part 4) - Data Dive for the original source).

After a lot of trial an error, the issue comes from the loss function. Basically the model.compute_loss function does not have the same behavior as when the article was written, and we fallback on Keras’ implementation instead of HuggingFace’s.

The following PullRequest shows that the method was renamed to hf_compute_loss.
https://github.com/huggingface/transformers/pull/15207

In a nutshell, change:
model.compile(optimizer=optimizer, loss=model.compute_loss)
by:
model.compile(optimizer=optimizer, loss=model.hf_compute_loss)

And enjoy :slight_smile:
Sadlig.

3 Likes
9

This is awesome - thank you @Sadlig I’ve been using that guide as a learning resource, but struggling with this bit! Much appreciated - I knew it was to do with the loss function!