Is there a version of `prepare_for_model` that works on a `List[List[int]]`?

wgpubs · February 4, 2022, 12:29am

Is there a version of prepare_for_model that works on a List[List[int]]?

For example, I’ve pre-processed a dataset to build my input_ids and now I want to get the token_type_ids, attention_mask, associated to them given a list of them. What is the recommended approach for doing so?

Thanks - wg

See: Utilities for Tokenizers

Topic		Replies	Views
How can we pass a list of strings to a fine tuned bert model? 🤗Transformers	0	510	August 18, 2022
Do any models support 3 types of token_type_ids? 🤗Transformers	0	177	June 21, 2021
Do I need token_type_ids for BertForSequenceClassification? 🤗Transformers	2	219	October 12, 2020
Set dataset to pytorch tensors produce class list making the model unable to process the data 🤗Datasets	3	2470	July 20, 2021
Use of "input_ids,token_type_ids and lm_labels" in BERT Language model 🤗Transformers	1	1060	September 20, 2020

Is there a version of `prepare_for_model` that works on a `List[List[int]]`?

Related topics