This is my tokenizer method. I found that no matter how much batch_size is set, the speed is the same. Tokenizer Spend time even longer than training. How cloud I do. Thanks very much.
def tokenize_function(example): return tokenizer(example["sentence1"], truncation=True, max_length = 512) tokenized_datasets = raw_datasets.map(tokenize_function, batched=True, batch_size = 8) tokenized_datasets = tokenized_datasets.remove_columns(["sentence1"])