The code fails when multiprocessing is called. I suspect this is because you didn’t use
if __name__ == "__main__":
in your script. Therefore the subprocess would re-run your script and call load_dataset in the subprocess again.
1 Like