How to tokenize using map

[lewtun]

hey @Neel-Gupta, could you share a minimal example of the Dataset object you’re working with?

@Neel-Gupta

The output of this last command produces text that is completely raw (basically untokenized) string- (like: ‘lorem ipsum…’) which is expected since I didn’t call tokenizer.tokenize

So does anyone have any idea how to get the model to tokenized as well? I tried a few obvious ways, but it didn’t yield anything

just so i understand, you’re saying that the tokenizer is not tokenizing the strings in the src field right?