0
Hard time understanding the working of tokenizer
from transformers import AutoModelForSequenceClassification,AutoTokenizer #hugging face libraries tkz = AutoTokenizer.from_pretrained(model)
The function: def tkz_func(x): return tkz(x[‘input’]) works perfectly when we apply it to the datasets, returns updated dataset with input_ids, token_type_ids, attention_masks
When we apply it to the dataframe df.apply(tkz_func,axis=1) it just returns the list of row names for all the row values [input_ids,token_type_ids,attention_masks]
Why?