[Open-to-the-community] Community week using JAX/Flax for NLP & CV :jax:

Flax/JAX Projects
18

Great! Created some of my project ideas here.

  1. PreTrain GPT2 from scratch in Bengali
  2. PreTrain T5 from scratch in Bengali
  3. PreTrain RoBERTa (MLM model) from scratch for Programming Languages

Not sure what is the condition of the T5 pre-training script. Would love to contribute and adapt the given MLM and CLM script to T5 if it’s not done yet.

3 Likes