when i use the map on Dataset object the 'input_ids return as list instead of tensors def tokenize(batch): return tokenizer(batch['text'],padding=True, return_tensors='pt', truncation=True).to(DEVICE) data.map(tokenize, batched=False, batch_size=None) → return list tokenizer(data['text'],paddin…

Dataset map return only list instead torch tensors

sd3ntato June 27, 2023, 11:01am 6

I ended up using a data collator to take the lists and turn them back into tensors

1 Like

Topic		Replies	Views
The datasets.map() method doesn't keep tensor format from `tokenizer` 🤗Datasets	1	1928	November 4, 2022
Map() with a tokenizer does not return pytorch tensors Beginners	2	1159	August 23, 2023
Applying `.map` results in getting `List` type on `input_values` 🤗Datasets	1	380	November 9, 2023
TypeError when applying map after set_format(type='torch') 🤗Datasets	3	1349	September 13, 2022
Unable to properly map tensors to examples 🤗Datasets	6	1287	December 15, 2022