Dataset for fake news detection, fine tune or pre-train

[valhalla]

You could try to get baseline with fine-tuning before going for pre-training and then make decision based on the results.

This thread has nice pointer for pre-training

Pre-Train BERT (from scratch)

BERT has been trained on MLM and NSP objective. I wanted to train BERT with/without NSP objective (with NSP in case suggested approach is different). I haven’t performed pre-training in full sense before. Can you please share how to obtain the data (crawl and tokenization details which were used) on which BERT was trained on ?. Since it takes a lot of time, I am looking for well tested code that can yield the BERT with/without NSP in one go. Any suggestions will be helpful. I know about some pr…