How to edit outside variables in mapping function

Beginners
1

I am trying to find a workaround for my issue and to do so I would like to continue using the mapping function as I have a very large dataset (much bigger than the one in the linked post). I would like to store the multisets in a list of dicts like so:

list_data = []
def _multisetify(code_file):
    list_data.append(Multiset(code_file["tokens"]))
    return code_file
data.shuffle() # just to make the tqdm output more accurate
print(data.map(_multisetify, num_proc=32))

However this just leaves list_data empty? Does anybody know someway I would be able to do this using the map() function or someway I can do it as quickly as the map function?

2

Hi again,
I suspect that this here and your other issue have the same solution.
Issue

3

It doesn’t raise any error in this one it just doesn’t allow me to edit or create data that can be accessed outside of the function, like if I made a variable in _multisetify() it wouldn’t be able to be accessed outside (as it should), but if I make a variable scoped outside of _multisetify() and edit it inside (with append) then it changes inside the function (I checked), but does not get changed outside _multisetify().

4

In this example case you would have to also return the list_data

return code_file, list_data

But that doesn’t solve your problem.
To help debug this issue you might just run a loop over the data.
Sorry, hope someone can help,
M