Hi, I want to run multiple models on the same batch, but it seems each model has a different tokenizer. Is it possible to tokenize a batch?
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Help defining tokenizer | 0 | 279 | April 28, 2023 | |
| Tokenize iterable dataset | 0 | 262 | June 7, 2023 | |
| Best way to pass multiple pipelines over the same dataset | 0 | 161 | September 6, 2023 | |
| Mix and Match models and tokenizers for different operations over the same data? | 0 | 210 | October 31, 2023 | |
| Can i use a tokenizer x for a model y | 1 | 1823 | April 20, 2023 |