I’ve been trying to fine-tune a german gpt-2 Modell with a dataset from huggingface via TensorFlow.
When trying to start the Training, I get the following Error: AttributeError: ‘NoneType’ object has no attribute ‘dtype’.
You can find the code on this Google Colab Notebook: Google Colab
Any help would be appreciated and thanks for reading!
Same boat here. If it helps, I was getting my error at loss and so I printed it and got None. This is my case. Perhaps it’s not computing for some reason.
`use_cache = True` is incompatible with gradient checkpointing`. Setting `use_cache = False`...
AttributeError Traceback (most recent call last)
/tmp/ipykernel_3879877/1790787526.py in <cell line: 7>()
14 loss = outputs.loss
---> 16 loss.backward()
AttributeError: 'NoneType' object has no attribute 'backward'
Sometimes, this happens in the case that the function does not return anything. You can try to check whether forget to return something, such as return loss
Hm, yes. In my case, there was no
'label' column so there was no output, and thus no loss. I’m wondering if these features (label, target…) are task-dependent and why they would be needed for generation tasks, as supposed to classification?
Generation task training also needs the label as generation task training is supervised learning here.
I see. Thank you! So when training, would that be responsible in part of my dataset (to have “labels”) or the tokenizer to create a column called [‘labels’] alongside the others (attn_mask…)?
Attribute errors in Python are generally raised when you try to access or call an attribute that a particular object type doesn’t possess. It’s simply because there is no attribute with the name you called, for that Object. This means that you got the error when the “module” does not contain the method you are calling. But it is evident that the method is there, which leads to believe that may be the method was added by you in the source code after you had already imported the file (module). Or, some times packages get deprecated and they rename some functions. If that is true, then you may want to exit and reimport the module once again to be able to access the new method .
You can do it in another way to reimport the module with changes without having to exit the interpreter is to do the following:
If you are using python 3.2 or 3.3 you should:
If running Python 3.4 and up, do import importlib, then do:
The importlib.reload() method reload a previously imported module. The argument must be a module object, so it must have been successfully imported before .
I encountered the same issue (refer to Fine-tune a German GPT-2 Model with Tensorflow in Transformers for text generation (NLP part 4) - Data Dive for the original source).
After a lot of trial an error, the issue comes from the loss function. Basically the model.compute_loss function does not have the same behavior as when the article was written, and we fallback on Keras’ implementation instead of HuggingFace’s.
The following PullRequest shows that the method was renamed to hf_compute_loss.
In a nutshell, change: