Hello fine people of huggingface!
I’ve finetuned a BERT model on Sagemaker, roughly following the outline from this notebook
The training process itself completes without issues, but when the trainer.push_to_hub() (this line from train.py) method is called I’m getting the following error:
Error for Training job bert-target-sample-2022-05-02-08-35-04-2022-05-02-08-35-04-695: Failed. Reason: AlgorithmError: ExecuteUserScriptError:
ExitCode 1
ErrorMessage "raise EnvironmentError(exc.stderr)
OSError: remote: error: cannot lock ref 'refs/heads/main': is at bb0adcff5079411402a28f80eaa92ec8ae6ccbbd but expected 681a8a1fa9dd7b8bd61aecca1f5da47994f293e0 To https://huggingface.co/thusken/nb-bert-base-target-group ! [remote rejected] main -> main (failed to update ref) error: failed to push some refs to 'https://user:{HF_TOKEN_REDACTED}@huggingface.co/thusken/nb-bert-base-target-group'"
Command "/opt/conda/bin/python3.8 train.py --epochs 1 --eval_batch_size 64 --hub_model_id nb-bert-base-target-group --hub_strategy every_save --hub_token {HF_TOKEN_REDACTED} --learning_rate 3e-05 --model_id NbAiLab/nb-bert-base --model_name_or_path NbAiLab/nb-bert-base --output_dir /opt/ml/model --push_to_hub True --train_batch_size 16", exit code: 1
Any idea what’s happening here?