Is there a possibility to use MLM modelling for pretraining for autocasualLM model like MPT or falcon? If yes, Has someone tried it? Are there any relevant code bases which I can use?

uzumakiusa · August 16, 2023, 1:41pm

I wanted to use Masked language modelling as method to pretrain text on domain specific tasks. But for open source LLMs like MPT - They only work on autocasualLM. Is there a way to do MLM on MPT 30B?

Topic	Replies	Views
Pretraining T5 from scratch using MLM Models	161	February 15, 2024
Best LLM to pretrain? 🤗Transformers	380	February 29, 2024
How to repurpose a domain specific MLM model for Q&A? Models	182	January 12, 2023
Seeking clarification on MPNet fine-tuning objectives 🤗Transformers	294	February 17, 2021
Retrain T5 using unsupervised learning with MLM 🤗Transformers	179	May 21, 2023

Is there a possibility to use MLM modelling for pretraining for autocasualLM model like MPT or falcon? If yes, Has someone tried it? Are there any relevant code bases which I can use?

Related Topics