Trying to use AutoTokenizer with TensorFlow gives: `ValueError: text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).`

🤗Tokenizers
13

def tokenize(batch):
texts = [str(text) for text in batch[“text”]] # convert all to str
return tokenizer(texts, padding=True, truncation=True)

emotions_encoded = emotions.map(tokenize, batched=True, batch_size=None)

IT WORKS!!

1 Like