I am trying to map a dataset based on the stack smol xs that is transformed using this map function: def tokenize(code_file): code_file["tokens"] = tokenizer.encode(code_file["content"]).tokens code_file["token_ids"] = tokenizer.encode(code_file["content"]).ids return code_file data =…

Dataset map() raises value error when mapping list to dict-like class

mikehemberger August 13, 2024, 8:31pm 6

What I meant was: maybe code_file has a datasets.features setup that gets conflicted during your (re-)assignment.
Hope someone can help!
Best,
Mike

1 Like

Topic		Replies	Views
Dataset.map returns error: pyarrow.lib.ArrowInvalid: cannot mix list and non-list, non-null values 🤗Datasets	3	1776	November 21, 2025
Dataset map occasionally throws error 🤗Datasets	1	297	January 15, 2024
TypeError: Provided `function` which is applied to all elements of table returns a variable of type <class 'list'> 🤗Datasets	2	6562	February 28, 2024
Map method to tokenize raises index error 🤗Datasets	9	4317	June 9, 2021
TypeError: Couldn't cast array of type int64 to null 🤗Datasets	3	174	February 6, 2025