Big dataset when being tokenized using map function gives type error as TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

Beginners
1

As stated above , I have a big dataset that is taking 2 days to tokenize . I am finetuning llama 3.1 and around 6 hrs in run into following error

TypeError Traceback (most recent call last)
in <cell line: 1>()
----> 1 tokenized_datasets = ds.map(tokenize_function, batched=True)

11 frames
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py in _batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, split_special_tokens)
526 self._tokenizer.encode_special_tokens = split_special_tokens
527
→ 528 encodings = self._tokenizer.encode_batch(
529 batch_text_or_text_pairs,
530 add_special_tokens=add_special_tokens,

TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

here is the code :
from datasets import load_dataset

ds = load_dataset(“talkmap/telecom-conversation-corpus”)
def tokenize_function(examples):
return tokenizer(examples[“text”],padding=“max_length”,truncation=True,is_split_into_words=True)
tokenized_datasets = ds.map(tokenize_function, batched=True)

any help is appreciated