Why Text Dataset For Next SentencePrediction get “Run out of input” error?

🤗Transformers
1

I’m training BERT model using NSP (TextDatasetForNextSentencePrediction).

VOCAB_NAME = “bert-base-uncased"
MODEL_MAX_LEN = 512

tokenizer = AutoTokenizer.from_pretrained(VOCAB_NAME, max_len=MODEL_MAX_LEN)
# tokenizer = BertTokenizerFast.from_pretrained(VOCAB_NAME, max_len=MODEL_MAX_LEN)

dataset = TextDatasetForNextSentencePrediction(
    tokenizer=tokenizer,
    file_path=NSP_DATESET_PATH,
    block_size=MODEL_MAX_LEN
)
# NSP_DATESET_PATH is a text file processed from wikipedia following the requirement

But it keeps getting the EOFError: Ran out of input :

__init__(self, tokenizer, file_path, block_size, overwrite_cache, short_seq_probability, nsp_probability)
    401                 start = time.time()
    402                 with open(cached_features_file, "rb") as handle:
--> 403                     self.examples = pickle.load(handle)
    404                 logger.info(
    405                     f"Loading features from cached file {cached_features_file} [took %.3f s]", time.time() - start

EOFError: Ran out of input