TypeError: compute_loss() got an unexpected keyword argument 'labels'

Beginners
1

Hi there,

I’ve been trying to further pre-train BERT on my own data using the example script run_mlm.py.

The code I’m using to train is the following:

python3 transformers/examples/tensorflow/language-modeling/run_mlm.py \
--model_name_or_path bert-base-cased \
--validation_split_percentage 20 \
--line_by_line \
--learning_rate 2e-5 \
--do_train \
--do_eval \
--per_device_train_batch_size 128 \
--per_device_eval_batch_size 256 \
--num_train_epochs 4 \
--output_dir output \ 
--train_file text.txt

The full output can be seen here.

In particular, I’m getting an error TypeError: compute_loss() got an unexpected keyword argument 'labels' and I’m not really sure how to fix it.

I also ran the same code in colab using the same parameters, data, etc and everything works as expected and training runs without issues.

OS Version Ubuntu 20.04.4 LTS
Tensorflow version 2.8.0
Python version 3.8.10

Any help is much appreciated.
Thank you!

2

cc @Rocketknight1 and @joaogante

3

Hi @silkski, we changed what some of the compute_loss functions do internally in a relatively recent version of Transformers. As a result, if you’re using an older version, the example scripts for the current version won’t work for you. Try updating on your local machine (e.g. pip install --upgrade transformers) and let us know if that fixes it!

4

Complementing @Rocketknight1’s answer – TF/Keras 2.8 defined a new compute_loss function, which clashed with one of our internal functions. Because of that, updating transformers should fix it :slight_smile:

5

Thank you so much @Rocketknight1 and @joaogante. Upgrading transformers did the trick.