Pickling issue using map

MagicLuke · April 6, 2025, 5:44pm

I am mapping my dataset with the following compute_metrics method which give me a pickling issue.

    metric_cfg_list = config["metric_list"]
    metrics = [evaluate.load(metric_cfg["path"]) for metric_cfg in metric_cfg_list]

    # Placeholder for a tokenizer or normalizer class if needed.
    tokenizer = None

    def compute_metrics(sample):
        for metric in metrics:
            sample[metric.name] = metric.compute(
                predictions=[sample["clean_prediction"]],
                references=[sample["clean_label"]]
            )
        return sample

the following is the error message

Parameter 'function'=<function main.<locals>.compute_metrics at 0x7aa60a95f0a0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mec
hanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.                                                                                                                                                                                                               
Map (num_proc=16):   0%|                                                                                                                                                                                                                                                                                                              | 0/2116 [00:00<?, ? examples/s]                 
Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                     
  File "/ws/ifp-54_2/hasegawa/haolong2/AI4EE/CSR4RSR/evaluation.py", line 207, in <module>  
...
    StockPickler.save(self, obj, save_persistent_id)                                         
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 578, in save                                                                                
    rv = reduce(self.proto)                                                                  
TypeError: cannot pickle 'ThreadLocalFileContext' object

I saw a relevant post about the nonpicklable issue with some tokenizer and ppl solved it by implementing the getstate method or so. In my case, it’s an object from the evaluate package. I wonder how I should modify them to avoid this error.

John6666 · April 6, 2025, 6:31pm

Hmm… unless it’s a problem with dill, multiprocessing, or the cache, it’s better to call lhonestq…

github.com/huggingface/datasets

Failure to hash function when using .map()

opened 03:12AM - 16 Feb 23 UTC

closed 02:56PM - 16 Feb 23 UTC

venzen

### Describe the bug _Parameter 'function'=<function process at 0x7f1ec4388af…0> of the transform datasets.arrow_dataset.Dataset.\_map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed._ This issue with `.map()` happens for me consistently, as also described in closed issue #4506 Dataset indices can be individually serialized using dill and pickle without any errors. I'm using tiktoken to encode in the function passed to map(). Similarly, indices can be individually encoded without error. ### Steps to reproduce the bug ```py from datasets import load_dataset import tiktoken dataset = load_dataset("stas/openwebtext-10k") enc = tiktoken.get_encoding("gpt2") tokenized = dataset.map( process, remove_columns=['text'], desc="tokenizing the OWT splits", ) def process(example): ids = enc.encode(example['text']) ids.append(enc.eot_token) out = {'ids': ids, 'len': len(ids)} return out ``` ### Expected behavior Should encode simple text objects. ### Environment info Python versions tried: both 3.8 and 3.10.10 `PYTHONUTF8=1` as env variable Datasets tried: - stas/openwebtext-10k - rotten_tomatoes - local text file OS: Ubuntu Linux 20.04 Package versions: - torch 1.13.1 - dill 0.3.4 (if using 0.3.6 - same issue) - datasets 2.9.0 - tiktoken 0.2.0

github.com/huggingface/datasets

`_pickle.PicklingError: logger cannot be pickled` in multiprocessing `map`

opened 11:51PM - 03 Oct 22 UTC

closed 02:43PM - 21 Jul 23 UTC

ZhaofengWu

bug

## Describe the bug When I `map` with multiple processes, this error occurs. Th…e `.name` of the `logger` that fails to pickle in the final line is `datasets.fingerprint`. ``` File "~/project/dataset.py", line 204, in <dictcomp> split: dataset.map( File ".../site-packages/datasets/arrow_dataset.py", line 2489, in map transformed_shards[index] = async_result.get() File ".../site-packages/multiprocess/pool.py", line 771, in get raise self._value File ".../site-packages/multiprocess/pool.py", line 537, in _handle_tasks put(task) File ".../site-packages/multiprocess/connection.py", line 214, in send self._send_bytes(_ForkingPickler.dumps(obj)) File ".../site-packages/multiprocess/reduction.py", line 54, in dumps cls(buf, protocol, *args, **kwds).dump(obj) File ".../site-packages/dill/_dill.py", line 620, in dump StockPickler.dump(self, obj) File ".../pickle.py", line 487, in dump self.save(obj) File ".../pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File ".../pickle.py", line 902, in save_tuple save(element) File ".../pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File ".../site-packages/dill/_dill.py", line 1963, in save_function _save_with_postproc(pickler, (_create_function, ( File ".../site-packages/dill/_dill.py", line 1140, in _save_with_postproc pickler.save_reduce(*reduction, obj=obj) File ".../pickle.py", line 717, in save_reduce save(state) File ".../pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File ".../pickle.py", line 887, in save_tuple save(element) File ".../pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File ".../site-packages/dill/_dill.py", line 1251, in save_module_dict StockPickler.save_dict(pickler, obj) File ".../pickle.py", line 972, in save_dict self._batch_setitems(obj.items()) File ".../pickle.py", line 998, in _batch_setitems save(v) File ".../pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File ".../site-packages/dill/_dill.py", line 1963, in save_function _save_with_postproc(pickler, (_create_function, ( File ".../site-packages/dill/_dill.py", line 1140, in _save_with_postproc pickler.save_reduce(*reduction, obj=obj) File ".../pickle.py", line 717, in save_reduce save(state) File ".../pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File ".../pickle.py", line 887, in save_tuple save(element) File ".../pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File ".../site-packages/dill/_dill.py", line 1251, in save_module_dict StockPickler.save_dict(pickler, obj) File ".../pickle.py", line 972, in save_dict self._batch_setitems(obj.items()) File ".../pickle.py", line 998, in _batch_setitems save(v) File ".../pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File ".../site-packages/dill/_dill.py", line 1963, in save_function _save_with_postproc(pickler, (_create_function, ( File ".../site-packages/dill/_dill.py", line 1154, in _save_with_postproc pickler._batch_setitems(iter(source.items())) File ".../pickle.py", line 998, in _batch_setitems save(v) File ".../pickle.py", line 578, in save rv = reduce(self.proto) File ".../logging/__init__.py", line 1774, in __reduce__ raise pickle.PicklingError('logger cannot be pickled') _pickle.PicklingError: logger cannot be pickled ``` ## Steps to reproduce the bug Sorry I failed to have a minimal reproducible example, but the offending line on my end is ```python dataset.map( lambda examples: self.tokenize(examples), # this doesn't matter, lambda e: [1] * len(...) also breaks. In fact I'm pretty sure it breaks before executing this lambda batched=True, num_proc=4, ) ``` This does work when `num_proc=1`, so it's likely a multiprocessing thing. ## Expected results `map` succeeds ## Actual results The error trace above. ## Environment info - `datasets` version: 1.16.1 and 2.5.1 both failed - Platform: Ubuntu 20.04.4 LTS - Python version: 3.10.4 - PyArrow version: 9.0.0

You can also provide your own unique hash in map if you want, with the new_fingerprint argument.
Or disable caching using

import datasets
datasets.disable_caching()

MagicLuke · April 7, 2025, 2:12am

I tried both new_fingerprint and disable_cache(), but all still gave the same bug.

the complete error is as follow:

Map (num_proc=16):   0%|                                                                                                                                                                                                                                                                                                                               | 0/2116 [00:00<?, ? examples/s]
Traceback (most recent call last):                                                                                                                                                                                                                                                                                                                                                     
  File "/ws/ifp-54_2/hasegawa/haolong2/AI4EE/CSR4RSR/evaluation.py", line 213, in <module>                                                                                                                                                                                                                                                                                             
    main()                                                                                                                                                                                                                                                                                                                                                                             
  File "/ws/ifp-54_2/hasegawa/haolong2/AI4EE/CSR4RSR/evaluation.py", line 178, in main                                                                                                                                                                                                                                                                                                 
    ds[split] = ds[split].map(                                                                                                                                                                                                                                                                                                                                                         
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 557, in wrapper                                                                                                                                                                                                                                           
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)                                                                                                                                                                                                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3166, in map                                                                                                                                                                                                                                              
    for rank, done, content in iflatmap_unordered(                                                                                                                                                                                                                                                                                                                                     
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 720, in iflatmap_unordered                                                                                                                                                                                                                               
    [async_result.get(timeout=0.05) for async_result in async_results]                                                                                                                                                                                                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 720, in <listcomp>                                                                                                                                                                                                                                       
    [async_result.get(timeout=0.05) for async_result in async_results]                                                                                                                                                                                                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get                                                                                                                                                                                                                                                    
    raise self._value                                                                                                                                                                                                                                                                                                                                                                  
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/multiprocess/pool.py", line 540, in _handle_tasks                                                                                                                                                                                                                                          
    put(task)                                                                                                                                                                                                                                                                                                                                                                          
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/multiprocess/connection.py", line 209, in send                                                                                                                                                                                                                                             
    self._send_bytes(_ForkingPickler.dumps(obj))                                                                                                                                                                                                                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/multiprocess/reduction.py", line 54, in dumps                                                                                                                                                                                                                                              
    cls(buf, protocol, *args, **kwds).dump(obj)                                                                                                                                                                                                                                                                                                                                        
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 420, in dump                                                                                                                                                                                                                                                          
    StockPickler.dump(self, obj)                                                                                                                                                                                                                                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 487, in dump                                                                                                                                                                                                                                                                            
    self.save(obj)                                                                                                                                                                                                                                                                                                                                                                     
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                                                                                                                                                                                                                          
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                                                                                                                                                                                                                   
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save                                                                                                                                                                                                                                                                            
    f(self, obj)  # Call unbound method with explicit self                                                                                                                                                                                                                                                                                                                             
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 902, in save_tuple                                                                                                                                                                                                                                                                      
    save(element)
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 887, in save_tuple
    save(element)
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 902, in save_tuple                                                                          
    save(element)                                                                                                                                                                          
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                              
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save                                                                                
    f(self, obj)  # Call unbound method with explicit self                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 887, in save_tuple                                                                          
    save(element)                                                                                                                                                                          
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                              
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save                                                                                
    f(self, obj)  # Call unbound method with explicit self                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict                                                 
    StockPickler.save_dict(pickler, obj)                                                                                                                                                   
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 972, in save_dict                                                                           
    self._batch_setitems(obj.items())                                                                                                                                                      
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 998, in _batch_setitems                                                                     
    save(v)                                                                                                                                                                                
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                              
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save                                                                                
    f(self, obj)  # Call unbound method with explicit self                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1985, in save_function                                                    
    _save_with_postproc(pickler, (_create_function, (                                                                                                                                      
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1117, in _save_with_postproc                                              
    pickler.save_reduce(*reduction)                                                                                                                                                        
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 692, in save_reduce                                                                         
    save(args)                                                                                                                                                                             
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                              
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save                                                                                
    f(self, obj)  # Call unbound method with explicit self                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 887, in save_tuple                                                                          
    save(element)                                                                                                                                                                          
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                              
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save                                                                                
    f(self, obj)  # Call unbound method with explicit self                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 932, in save_list                                                                           
    self._batch_appends(obj)                                                                                                                                                               
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 956, in _batch_appends                                                                      
    save(x)                                                                                                                                                                                
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                              
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 603, in save                                                                                
    self.save_reduce(obj=obj, *rv)                                                                                                                                                         
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 717, in save_reduce                                                                         
    save(state)                                                                                                                                                                            
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                              
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save                                                                                
    f(self, obj)  # Call unbound method with explicit self                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict                                                 
    StockPickler.save_dict(pickler, obj)                                                                                                                                                   
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 972, in save_dict                                                                           
    self._batch_setitems(obj.items())                                                                                                                                                      
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 998, in _batch_setitems                                                                     
    save(v)                                                                                                                                                                                
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                              
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save                                                                                
    f(self, obj)  # Call unbound method with explicit self                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 932, in save_list                                                                           
    self._batch_appends(obj)                                                                                                                                                               
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 959, in _batch_appends                                                                      
    save(tmp[0])                                                                                                                                                                           
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                              
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                                                                                                                                                                                                                   
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 603, in save                                                                                
    self.save_reduce(obj=obj, *rv)                                                                                                                                                         
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 717, in save_reduce                                                                                                                                                                                                                                                                     
    save(state)                                                                                                                                                                            
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                                                                                                                                                                                                                   
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self                                                                                                                                 
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict
    StockPickler.save_dict(pickler, obj)                                                                                                                                                   
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())                                                                                                                                                      
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)                                                                                                                                                                                
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save                                                                                                                                                                                                                                                          
    StockPickler.save(self, obj, save_persistent_id)                                                                                                                                       
  File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 578, in save                                                                                                                                                                                                                                                                            
    rv = reduce(self.proto)                             
TypeError: cannot pickle 'ThreadLocalFileContext' object

John6666 · April 7, 2025, 4:00am

Hmm… @lhoestq map function or PyArrow issue…?

lhoestq · April 7, 2025, 9:51am

It looks like the ThreadLocalFileContext from filelock is not picklable, and therefore can’t be used with .map() with num_proc=...

Apparently thid can be fixed using thread_local=False, see the docs at filelock

Can you modify evaluate to pass thread_local=False to all FileLock objects and try again to see if it works ?

MagicLuke · April 7, 2025, 9:05pm

I am not sure if I do it right.

I modify the function get_from_cache in the file_utils located
…/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/evaluate/utils/file_utils.py
from

with FileLock(lock_path): # Origin

to

with FileLock(lock_path, thread_local=False): # Modified

but the problem persist.

MagicLuke · April 7, 2025, 9:30pm

By adding this code chunck before importing evaluating seems solved the problem.

from filelock import FileLock as OriginalFileLock

class PatchedFileLock(OriginalFileLock):
    def __init__(self, *args, **kwargs):
        kwargs["thread_local"] = False  # Force it every time
        super().__init__(*args, **kwargs)

import filelock
filelock.FileLock = PatchedFileLock

Thanks for the insight @lhoestq.
Would you mind telling where you find the clue for the error if it’s not too much trouble
In this way, I might be able to fix it the same way in the future.

lhoestq · April 8, 2025, 8:56am

Great ! Let me know if you think we should make this the default in datasets and evaluate, apparently this logic appears with python >= 3.11

Would you mind telling where you find the clue for the error if it’s not too much trouble
In this way, I might be able to fix it the same way in the future.

The dill error says “TypeError: cannot pickle ‘ThreadLocalFileContext’ object”, so it means that in the function you pass to map() there is an object that contains a ThreadLocalFileContext that is not supported by dill for multiprocessing.

I searched on google for ThreadLocalFileContext on github.com to look for packages that have such objects and figured it came from filelock which is a dependency of evaluate. Finally the filelock changelog they mention ThreadLocalFileContext as a recent addition for FileLock

MagicLuke · April 8, 2025, 4:54pm

Thanks for the explanation!

I think it would be great to set it as the default in my case, which is several metrics that need to be computed for a dataset. For me, I just want to avoid using multiple rounds of map. Or maybe there is a better way to do it that I haven’t figured out.

Topic		Replies	Views
Using "load_metric" offline in datasets 🤗Datasets	2	4315	August 24, 2021
How to deal with unpickable objects in map 🤗Datasets	9	4482	October 23, 2020
How to force caching of previously tokenized data? (run_clm.py) Beginners	3	643	November 21, 2023
PyTorchBenchmark pickle local object error 🤗Transformers	0	317	June 3, 2023
`datasets.map` calls a function that requires a `transformers.PreTrainedModel` object - unpickable object 🤗Datasets	2	1874	December 2, 2022

Pickling issue using map

Related topics