Damn that’s sneaky! I just spent a whole day debugging this thing. I mean, tokenizer does return pytorch tensors, but not if called within Map, Map just ignores that argument! Well, turns out it was some internal implementation thing that’s supposed to be “expected”. I really wish that it was explicitly stated in the docs, not hidden like that
2 Likes