BART pre-training?

Beginners
1

Hi,
How can I pre-train Bart with our own dataset? It seems that the script examples/language-modeling/run_language_modeling.py doesn’t support it yet. Thanks.

1 Like
2

Hi @cahya, BART pre-training is not yet available in transformers. You can find the denoising dataset here in the fairseq repo, and try to use it;

3

Thanks for the info and the link to the denoising dataset. Maybe @sshleifer can tell us also his experience with BART and transformers?
And how about T5? I see that you have already several T5 models, can we pre-train T5 with our own dataset using transformers?

4

Adding both of these tasks (T5 and BART pre-training) is in my todo list. Might take some time though.
If you are able to create the span masking code for T5 then you can easily pre-train T5 with Transformers.

1 Like
5

Great, I’ll try it

6

Any updated on this?