when i use the map on Dataset object the 'input_ids return as list instead of tensors def tokenize(batch): return tokenizer(batch['text'],padding=True, return_tensors='pt', truncation=True).to(DEVICE) data.map(tokenize, batched=False, batch_size=None) → return list tokenizer(data['text'],paddin…

Dataset map return only list instead torch tensors

kenfus March 1, 2024, 4:12pm 7

Yep, I also got burned by this. What is the point here to return list of lists when it’s a torch tensor? Inside the map function, it’s a tensor, then after it’s a list?

2 Likes

Topic		Replies	Views
The datasets.map() method doesn't keep tensor format from `tokenizer` 🤗Datasets	1	1928	November 4, 2022
Map() with a tokenizer does not return pytorch tensors Beginners	2	1159	August 23, 2023
Applying `.map` results in getting `List` type on `input_values` 🤗Datasets	1	380	November 9, 2023
TypeError when applying map after set_format(type='torch') 🤗Datasets	3	1349	September 13, 2022
Unable to properly map tensors to examples 🤗Datasets	6	1287	December 15, 2022

Dataset map return only list instead torch tensors

Related topics