Hi, I want to run multiple models on the same batch, but it seems each model has a different tokenizer. Is it possible to tokenize a batch?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Help defining tokenizer | 0 | 272 | April 28, 2023 | |
| Tokenize iterable dataset | 0 | 259 | June 7, 2023 | |
| Mix and Match models and tokenizers for different operations over the same data? | 0 | 204 | October 31, 2023 | |
| Can i use a tokenizer x for a model y | 1 | 1695 | April 20, 2023 | |
| Do we have to tokenize the question and context together for Q&A models? | 0 | 225 | March 13, 2022 |