Applying `.map` results in getting `List` type on `input_values`

panigrah · November 9, 2023, 6:02am

see here… dataset returns pure python objects.

Dataset.map() turns tensors into lists?

opened 11:43AM - 03 Dec 20 UTC

closed 12:12PM - 05 Oct 22 UTC

I apply `Dataset.map()` to a function that returns a dict of torch tensors (like… a tokenizer from the repo transformers). However, in the mapped dataset, these tensors have turned to lists! ```import datasets import torch from datasets import load_dataset print("version datasets", datasets.__version__) dataset = load_dataset("snli", split='train[0:50]') def tokenizer_fn(example): # actually uses a tokenizer which does something like: return {'input_ids': torch.tensor([[0, 1, 2]])} print("First item in dataset:\n", dataset[0]) tokenized = tokenizer_fn(dataset[0]) print("Tokenized hyp:\n", tokenized) dataset_tok = dataset.map(tokenizer_fn, batched=False, remove_columns=['label', 'premise', 'hypothesis']) print("Tokenized using map:\n", dataset_tok[0]) print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids'])) dataset_tok = dataset.map(tokenizer_fn, batched=False, remove_columns=['label', 'premise', 'hypothesis']) print("Tokenized using map:\n", dataset_tok[0]) print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids'])) ``` The output is: ``` version datasets 1.1.3 Reusing dataset snli (/home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c) First item in dataset: {'premise': 'A person on a horse jumps over a broken down airplane.', 'hypothesis': 'A person is training his horse for a competition.', 'label': 1} Tokenized hyp: {'input_ids': tensor([[0, 1, 2]])} Loading cached processed dataset at /home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c/cache-fe38f449fe9ac46f.arrow Tokenized using map: {'input_ids': [[0, 1, 2]]} <class 'torch.Tensor'> <class 'list'> ``` Or am I doing something wrong?

here is one possible approach but it has other side effects.

eval_dataset = eval_dataset.with_format('tf')

Topic		Replies	Views
Datasets map modifying audio array to list? 🤗Datasets	1	1267	November 29, 2021
Dataset map return only list instead torch tensors Beginners	8	5469	March 17, 2025
Missing ['array] when using map() method Beginners	2	357	November 25, 2022
The datasets.map() method doesn't keep tensor format from `tokenizer` 🤗Datasets	1	1882	November 4, 2022
About dataset map 🤗Datasets	5	401	August 20, 2023

Applying `.map` results in getting `List` type on `input_values`

Related topics