Pre-Train BERT (from scratch)

Research
44

Any progress here ? I would be so convenient to train a Bert from scratch using datasets and transformers. Does anyone achieve this with comparable results as original Bert ?

45

Hi @BramVanroy is there an example for pretraining bert on NSP tasks with dataset.map? Thanks!

46

Hi @vblagoje , I found the file_path param of TextDatasetForNextSentencePrediction is only one file. Does it mean that I need to convert all datasets into one file when splitting sentences? But this file will be too big.

47

To chunk the articles you can check https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset

new link is https://huggingface.co/docs/datasets/process#data-augmentation