Make compute_metric / load_metric distributed and use multiple test sets on do_predict

Beginners
#1

I am using transformer PyTorch examples to do Machine Translation. I am computing multiple metrics on large validation sets.

  1. how is it possible to Integrate distributed compute_metrics with with seq2seqTrainer? (with or without predict_with_generate( just for during do_eval) ) I took a look at distributed usage of load_metric, however not sure if it is possible to integrate it without implementing custom train/evaluation loop.

  2. how is it possible to have two sets on do_predict?

Thanks,

#2

As long as you launch your script with the torch.distributed launcher, the evaluation will automatically be distributed (each GPU will see a fraction of your dataset and make the predictions on it).

To pass along two datasets for predictions, you will need to tweak the example script to run the predict method on those two datasets.

1 Like