Hello, Given the following code: tokenizer = GPT2Tokenizer.from_pretrained(xxx) def encode(batch): return tokenizer(batch['text'], padding="max_length", add_special_tokens=True, truncation=True, max_length=64) dataset = load_dataset("text", data_files = ['text.txt']) dataset = dataset['train']…

How did the dataset manages long sentences?

lhoestq February 15, 2022, 4:09pm 2

Hi ! That’s correct. You can use .map() to do so, see how to split long sentences for example

1 Like