Hello HF Team,
I am familiar with how to pretrain BERT and I have a Dataloader that reads an input CSV file line by line and every time it reads a line, it tokenizes it and sends back the tokens for training to the training code. My question is whether is it ok for this input CSV file to contain more than one sentence on every line when pretraining BERT for masked language modelling?
Otherwise, is it important for it to contain only one meaningful sentence only? I am thinking if self attention will still continue to work and the model train properly even if every single line in the input CSV file (single training sample) is actually more than one sentence, each of it separated with a ‘.’ delimiter of course.
Thanks