Questions on distilling [from] T5

[sshleifer]

Exactly.
Complicated version of that:

Diverse Generations for pseudolabeling

I read the FAIR self-training paper and it is very relevant, but I am struggling to understand the specifics of the approach. Things I understood: “use beam search decoding (beam size 5) to create the pseudo targets and to report BLEU on test set.” This outperforms sampling. They train with dropout rate at 0.3. This helps. They don’t do any cleaning or checking of the pseudolabels against the ground truth target. all experiments are run on 8 GPUs with an effective batch size of 33K tokens. T…