NLP dataset for ByteLevelTokenizer Training

tillfurger · February 15, 2021, 9:47pm

Hi I would like to train my own ByteLevelBPETokenizer using an nlp dataset.

tokenizer = ByteLevelBPETokenizer()

tokenizer.train(files=???, vocab_size=52000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

The dataset is from:

from datasets import load_dataset
dataset = load_dataset('wikicorpus', 'raw_en')

How can I process this dataset to input it in the tokenizer.train() function?

Thanks

lhoestq · February 16, 2021, 2:10pm

You can take a look at the example script here:

github.com

huggingface/tokenizers/blob/master/bindings/python/examples/train_with_datasets.py

import datasets
from tokenizers import normalizers, pre_tokenizers, Tokenizer, models, trainers

# Build a tokenizer
bpe_tokenizer = Tokenizer(models.BPE())
bpe_tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
bpe_tokenizer.normalizer = normalizers.Lowercase()

# Initialize a dataset
dataset = datasets.load_dataset("wikitext", "wikitext-103-raw-v1")

# Build an iterator over this dataset
def batch_iterator():
    batch_length = 1000
    for i in range(0, len(dataset["train"]), batch_length):
        yield dataset["train"][i : i + batch_length]["text"]


# And finally train
bpe_tokenizer.train_from_iterator(batch_iterator(), length=len(dataset["train"]))

Topic		Replies	Views
Does the ByteLevelBPETokenizer need to be wrapped in a normal Tokenizer? 🤗Tokenizers	0	1769	March 18, 2023
HuggingFace BPE Trainer Error - Training Tokenizer 🤗Tokenizers	1	2920	July 14, 2022
Building a Custom Tokenizer with HGF Dataset: Batch Iterator Best Practices 🤗Datasets	4	527	December 8, 2023
Help understanding how to build a dataset for language as with the old TextDataset 🤗Datasets	7	12484	October 6, 2021
Use dataset.map for ngrams and Word2Vec style data pipeline Beginners	0	877	April 26, 2021

NLP dataset for ByteLevelTokenizer Training

Related topics