Using multiple processes causes errors when retrieving active_run from MLflowTracker

🤗Accelerate
1

I’m currently using MLflowTracker to track my experiments and wanted to save the metadata for each run as follows:

info = tracker.active_run.to_dictionary()
with open(file=$PATH, mode="w") as f:
    json.dump(obj=info, fp=f, indent=2)

This works fine when I’m using 1 process, but when I try to run more than 1 process then I get an error message AttributeError: 'MLflowTracker' object has no attribute 'active_run'. What’s funny is that when I add a print(tracker.active_run) I get the information printed out to stdout just fine.

2

Turns out that only one of the processes’ trackers has the attribute active_run. I added a print(dir(tracker)) and got the following:

['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'finish', 'log', 'main_process_only', 'name', 'requires_logging_directory', 'store_init_configuration', 'tracker']

['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'active_run', 'finish', 'log', 'main_process_only', 'name', 'requires_logging_directory', 'store_init_configuration', 'tracker']

After doing some more investigation it seems like there’s a main_process and non-main_processs. I changed my code to:

if accelerator.is_main_process:
    info = tracker.active_run.to_dictionary()
    with open(file=$PATH, mode="w") as f:
        json.dump(obj=info, fp=f, indent=2)

and it works now.

1 Like
3

That’s the exact right solution, nice work! Accelerate will log/create loggers only on the main process

1 Like