Also discussed here: https://github.com/huggingface/evaluate/issues/https://github.com/huggingface/evaluate/issues/542542
I’m facing this issue when running my code in a distributed setting (using accelerate on 3 nodes with 4 GPUs each)
I can see my lock files being generated and present in the cache directory, but I still get this error -
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/rm6418/projects/GEMT5/gemt5-mul.py", line 517, in <module>
[rank0]: gemt5_train(args.model, args.src, args.tgt, args.g, args.bsize, args.maxlen)
[rank0]: File "/home/rm6418/projects/GEMT5/gemt5-mul.py", line 498, in gemt5_train
[rank0]: trainer.train()
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer.py", line 1932, in train
[rank0]: return inner_training_loop(
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer.py", line 2345, in _inner_training_loop
[rank0]: self._maybe_log_save_evaluate(tr_loss, grad_norm, model, trial, epoch, ignore_keys_for_eval)
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer.py", line 2793, in _maybe_log_save_evaluate
[rank0]: metrics = self._evaluate(trial, ignore_keys_for_eval)
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer.py", line 2750, in _evaluate
[rank0]: metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 180, in evaluate
[rank0]: return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer.py", line 3641, in evaluate
[rank0]: output = eval_loop(
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/transformers/trainer.py", line 3923, in evaluation_loop
[rank0]: metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
[rank0]: File "/home/rm6418/projects/GEMT5/gemt5-mul.py", line 478, in <lambda>
[rank0]: compute_metrics=lambda eval_pred: compute_metrics(eval_pred, tokenizer, accelerator),
[rank0]: File "/home/rm6418/projects/GEMT5/gemt5-mul.py", line 119, in compute_metrics
[rank0]: return metric.compute(predictions=decoded_preds, references=references)
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/evaluate/module.py", line 455, in compute
[rank0]: self.add_batch(**inputs)
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/evaluate/module.py", line 515, in add_batch
[rank0]: self._init_writer()
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/evaluate/module.py", line 661, in _init_writer
[rank0]: self._check_all_processes_locks() # wait for everyone to be ready
[rank0]: File "/ext3/miniconda3/envs/venv/lib/python3.8/site-packages/evaluate/module.py", line 355, in _check_all_processes_locks
[rank0]: raise ValueError(
[rank0]: ValueError: Expected to find locked file /scratch/rm6418/exp_cache/sacrebleu/default/exp_exp1-12-4.arrow.lock from process 0 but it doesn't exist."
[rank0]: ValueError: Expected to find locked file /scratch/rm6418/exp_cache/sacrebleu/default/exp_exp1-12-4.arrow.lock from process 0 but it doesn't exist."
Does anyone know any workarounds for this issue?