Map with tokenize function stuck in the beginning

[sleepywalker]

Hello. I have a function

block_size = 8192

def tokenize_and_chunk(examples):
    tokenized_inputs = tokenizer(examples['content'], return_tensors='pt', add_special_tokens=False)
    
    input_ids = tokenized_inputs['input_ids'][0]
    total_length = len(input_ids)
    
    if total_length >= block_size:
        input_ids = input_ids[:(total_length // block_size) * block_size]
    
    input_chunks = [input_ids[i:i + block_size] for i in range(0, len(input_ids), block_size)]
    
    return {'input_ids': input_chunks}

and a dataset with 25000 samples where every sample is a string with ~50000 symbols. When I try to tokenize this dataset using tokenized_dataset = concatenated_data.map(tokenize_and_chunk, batched=True, num_proc=200, remove_columns=concatenated_data.column_names) it stucks in the beginning. I tried to change num_proc to 1 to use tokenizer parallelism but it didn’t help. When I try to tokenize with this function only one sample like tokenize_and_chunk(concatenated_data[0]) it succesfully tokenized in a moment. How can I fix that problem with dataset.map for tokenization?

[John6666]

Maybe try IterableDataset?

Training a Tokenizer on a Streamed Dataset

Hi, I’m trying to train a tokenizer on a dataset that uses streaming. I followed the instructions provided here, with the addition of streaming=True during the dataset loading step. However, it quickly failed as the IterableDataset class does not have a length property (unlike the normal Dataset class). How can I work around this issue without having to download the dataset file entirely, i.e. purely relying on dataset streaming? Here’s the Colab link to reproduce the error. Any help would be …

[sleepywalker]

With IterableDataset like ConstnatLengthDataset model will train longer than with predefined tokenized dataset. Moreover, with IterableDataset multigpu training is more complex. It needs something like DistributedSample, DistributedLoader and so on.

[John6666]

As you say, I don’t want to recommend it either. The slowdown and difficulty of implementation are unavoidable overheads, and this is a last resort. However, your dataset is probably too large even in an environment with a lot of RAM…
There may be a way to train it by slicing the dataset itself in advance simply and feeding it in small amounts without streaming.

Keeping IterableDataset node-wise split fixed during DDP

Thanks, yes, the dataset has many shards. But could you give the example how to keep the dataset split fixed for each node during the whole DDP run? The documentation had the following example for split_dataset_by_node: from datasets.distributed import split_dataset_by_node ids = ds.to_iterable_dataset(num_shards=512) ids = ids.shuffle(buffer_size=10_000) # will shuffle the shards order and use a shuffle buffer when you start iterating ids = split_dataset_by_node(ids, world_size=8, rank=0) …

Stream

[sleepywalker]

I mean why function with tokenization takes so much time to implement it relaitively to other functions. I understand that tokenization is not a simple transformation but this tokenizer is fast and as I know tokenization has multiprocessing by itself and I can map with only 1 num_proc and anyway it will parallelizae tokenization by itself. But I can’t understand problem that mapping with batch_size=100 for example just stuck in the beginning and not working. Due to that behaviour, other mappings takes like 1-2 minute for implementation while tokenization may take 15-20 minutes with batched=False to not to stuck in the beginnging