Questions about pseudolabels

sshleifer · October 8, 2020, 2:34am

huggingface/transformers/blob/master/examples/seq2seq/precomputed_pseudo_labels.md

### Precomputed pseudolabels
+ decompress with tar -xzvf. The produced directory name may differ from the filename.

| Dataset | Model                       | Rouge Scores       | Notes                                                                                                       | Link                                                                                   |
|---------|-----------------------------|--------------------|-------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|
| XSUM    | facebook/bart-large-xsum    | 49.8/28.0/42.5     |                                                                                                             | [download](https://s3.amazonaws.com/datasets.huggingface.co/pseudo/xsum/bart_xsum_pl.tgz)          |
| XSUM    | google/pegasus-xsum         | 53.3/32.7/46.5     |                                                                                                             | [download](https://s3.amazonaws.com/datasets.huggingface.co/pseudo/xsum/pegasus_xsum.tgz)          |
| XSUM    | facebook/bart-large-xsum    | ?                  | Bart pseudolabels filtered to those with Rouge2 > 10.0 w GT                                                 | [download](https://s3.amazonaws.com/datasets.huggingface.co/pseudo/xsum/xsum_pl2_bart.tgz)         |
|         |                             |                    |                                                                                                             | [download](https://s3.amazonaws.com/datasets.huggingface.co/pseudo/xsum/pegasus_xsum_on_cnn.tgz)   |
| CNN/DM  | sshleifer/pegasus-cnn-ft-v2 | 47.316/26.65/44.56 | do not worry about the fact that train.source is one line shorter.                                          | [download](https://s3.amazonaws.com/datasets.huggingface.co/pseudo/cnn_dm/pegasus_cnn_cnn_pls.tgz) |
| CNN/DM  | facebook/bart-large-cnn     |                    | 5K (2%) are missing, there should be 282173                                                                 | [download](https://s3.amazonaws.com/datasets.huggingface.co/pseudo/cnn_dm/cnn_bart_pl.tgz)         |
| CNN/DM  | google/pegasus-xsum         | 21.5/6.76/25       | extra labels for xsum distillation  Used max_source_length=512, (and all other pegasus-xsum configuration). | [download](https://s3.amazonaws.com/datasets.huggingface.co/pseudo/cnn_dm/pegasus_xsum_on_cnn.tgz) |
| EN-RO   | Helsinki-NLP/opus-mt-en-ro  |       |  | [download](https://s3.amazonaws.com/datasets.huggingface.co/pseudo/wmt_en_ro/opus_mt_en_ro.tgz) |
| EN-RO   | facebook/mbart-large-en-ro  |       |  | [download](https://s3.amazonaws.com/datasets.huggingface.co/pseudo/wmt_en_ro/mbart_large_en_ro.tgz) |

### Generating Pseudolabels
+ These command takes a while to run. For example,  pegasus_cnn_cnn_pls.tgz took 8 hours on 8 GPUs.
+ Pegasus does not work in fp16 :(, Bart, mBART and Marian do.

```

This file has been truncated. show original

Feel free to contribute your own pseudolabels via google drive link!

boscof · April 4, 2021, 2:25pm

Thanks for all the update @sshleifer.

Can I ask which dataset is the google/pegasus-large trained on?