Roberta Swedish The project idea is identical to the one for Pretraining Roberta in Spanish but instead using the Swedish dataset. The idea is to use the Swedish portion of mC4 (which roughly amounts for 100GB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length …

Pretrain RoBERTa-large from scratch in Swedish

saattrupdan June 28, 2021, 5:38pm 2

Just a suggestion: How about we pretrain a common Scandinavian model instead, as our languages are so similar? I’ve made a project suggestion here: Scandinavian RoBERTa.

2 Likes

Topic		Replies	Views
Pretrain RoBERTa-large from scratch in Finnish Flax/JAX Projects	1	315	June 29, 2021
Scandinavian RoBERTa Flax/JAX Projects	30	1934	July 15, 2021
RobIt : PreTrain RoBERTa-base from scratch in Italian Flax/JAX Projects	4	422	June 29, 2021
PreTrain RoBERTa from scratch in Indonesian Flax/JAX Projects	6	462	June 28, 2021
Pre-train RoBERTa from Scratch for Georgian Language Flax/JAX Projects	1	1173	July 7, 2021

Pretrain RoBERTa-large from scratch in Swedish

Related Topics