Hi there,
I’ve been trying to further pre-train BERT on my own data using the example script run_mlm.py.
The code I’m using to train is the following:
python3 transformers/examples/tensorflow/language-modeling/run_mlm.py \
--model_name_or_path bert-base-cased \
--validation_split_percentage 20 \
--line_by_line \
--learning_rate 2e-5 \
--do_train \
--do_eval \
--per_device_train_batch_size 128 \
--per_device_eval_batch_size 256 \
--num_train_epochs 4 \
--output_dir output \
--train_file text.txt
The full output can be seen here.
In particular, I’m getting an error TypeError: compute_loss() got an unexpected keyword argument 'labels'
and I’m not really sure how to fix it.
I also ran the same code in colab using the same parameters, data, etc and everything works as expected and training runs without issues.
OS Version Ubuntu 20.04.4 LTS
Tensorflow version 2.8.0
Python version 3.8.10
Any help is much appreciated.
Thank you!