Roberta Swedish The project idea is identical to the one for Pretraining Roberta in Spanish but instead using the Swedish dataset. The idea is to use the Swedish portion of mC4 (which roughly amounts for 100GB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length …

Pretrain RoBERTa-large from scratch in Swedish

saattrupdan June 28, 2021, 5:38pm 2

Just a suggestion: How about we pretrain a common Scandinavian model instead, as our languages are so similar? I’ve made a project suggestion here: Scandinavian RoBERTa.

2 Likes

Topic		Replies	Views
Pretrain RoBERTa-large from scratch in Finnish Flax/JAX Projects	1	351	June 29, 2021
Scandinavian RoBERTa Flax/JAX Projects	30	2032	July 15, 2021
BERTIN: PreTrain RoBERTa-large from scratch in Spanish Flax/JAX Projects	23	1998	July 19, 2021
RobIt : PreTrain RoBERTa-base from scratch in Italian Flax/JAX Projects	4	471	June 29, 2021
PreTrain RoBERTa from scratch in Portuguese Flax/JAX Projects	16	2360	October 4, 2021

Pretrain RoBERTa-large from scratch in Swedish

Related topics