Limiting print and log statements

[aclifton314]

Accelerate: 0.9.0
Python: 3.9.7

I have written a class that handles printing and logging when going between cpu and gpu training configurations. Here is the class:

import logging
from accelerate import Accelerator
import wandb
from typing import (List, Tuple, Any, Union)
import torch
logging.basicConfig(level=logging.DEBUG, format='%(levelname)s: %(message)s')


class TrainRun:
    def __init__(
        self,
        tracker_project_name,
        tracker_run_name, 
        is_accelerate:bool=False,
        tracker_name:str=None) -> None:
        
        self.tracker_project_name = tracker_project_name
        self.tracker_run_name = tracker_run_name
        self.is_accelerate = is_accelerate
        self.tracker_name = tracker_name
        
        if self.is_accelerate:
            self.accelerator = Accelerator(log_with=[self.tracker_name])
            self.accelerator.init_trackers(self.tracker_project_name)
            self.accelerator.trackers[0].run.name = self.tracker_run_name
            self.device=self.accelerator.device
            
        else:
            wandb.init(project=self.tracker_project_name)
            wandb.run.name = self.tracker_run_name
            self.device = torch.device("cpu")


    def log_it(self, str_to_log: Union[str, List[str]]):
        if type(str_to_log) is str:
            if self.is_accelerate and self.accelerator.is_local_main_process:
                logging.info(str_to_log)
            else:
                logging.info(str_to_log)
        else:
            if self.is_accelerate and self.accelerator.is_local_main_process:
                for log_str in str_to_log:
                    logging.info(log_str)
            else:
                for log_str in str_to_log:
                    logging.info(log_str)


    def print_it(self, str_to_print: Union[str, List[str]]):
        if type(str_to_print) is str:
            if self.is_accelerate and self.accelerator.is_local_main_process: 
                print(str_to_print)
            else:
                print(str_to_print)
        else:
            if self.is_accelerate and self.accelerator.is_local_main_process: 
                for print_str in str_to_print:
                    print(print_str)
            else: 
                for print_str in str_to_print:
                    print(print_str)

I have a machine that has 4 GPUs on it. If I’ve done things correctly, I’d expect a single print out or log statement in the terminal. Here is some sample code to try:

tracker_project_name = 'my_project'
tracker_name = 'wandb'
tracker_run_name = 'full_dataset'
train_run = TrainRun(tracker_project_name, tracker_run_name, is_accelerate=True, tracker_name=tracker_name)

train_run.log_it('CREATED TrainRun OBJECT')
train_run.log_it('is_accelerate = {}'.format(train_run.is_accelerate))
train_run.log_it('device = {}'.format(train_run.device))
train_run.print_it('-'*100)

However, when I run accelerate launch tmp.py from the command line, I get several repeated log and print out statements:

INFO: Added key: store_based_barrier_key:1 to store for rank: 3
INFO: Added key: store_based_barrier_key:1 to store for rank: 2
INFO: Added key: store_based_barrier_key:1 to store for rank: 0
INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Added key: store_based_barrier_key:1 to store for rank: 1
INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
DEBUG: git repository is invalid
DEBUG: git repository is invalid
DEBUG: git repository is invalid
DEBUG: git repository is invalid
wandb: Tracking run with wandb version 0.12.17
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
INFO: CREATED TrainRun OBJECT
INFO: is_accelerate = True
INFO: device = cuda:1
----------------------------------------------------------------------------------------------------
wandb: Tracking run with wandb version 0.12.17
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
wandb: Tracking run with wandb version 0.12.17
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
INFO: Initialized WandB project my_project
INFO: Make sure to log any initial configurations with `self.store_init_configuration` before training!
INFO: CREATED TrainRun OBJECT
INFO: is_accelerate = True
INFO: device = cuda:0
----------------------------------------------------------------------------------------------------
INFO: CREATED TrainRun OBJECT
----------------------------------------------------------------------------------------------------INFO: is_accelerate = True
INFO: device = cuda:3

wandb: Tracking run with wandb version 0.12.17
wandb: W&B syncing is set to `offline` in this directory.  
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
INFO: CREATED TrainRun OBJECT
----------------------------------------------------------------------------------------------------INFO: is_accelerate = True
INFO: device = cuda:2

wandb: Waiting for W&B process to finish... (success).
wandb: Waiting for W&B process to finish... (success).ed)
wandb:                                                                                
wandb:                                                                                
wandb: Waiting for W&B process to finish... (success).
wandb: Waiting for W&B process to finish... (success).ed)
wandb:                                                                                
wandb:                                                                                
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/aclifton/rf_fp/wandb/offline-run-20220705_125143-3cosslzv
wandb: Find logs at: ./wandb/offline-run-20220705_125143-3cosslzv/logs
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/aclifton/rf_fp/wandb/offline-run-20220705_125143-3x7h5hfi
wandb: Find logs at: ./wandb/offline-run-20220705_125143-3x7h5hfi/logs
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/aclifton/rf_fp/wandb/offline-run-20220705_125143-2wkg3sg7
wandb: Find logs at: ./wandb/offline-run-20220705_125143-2wkg3sg7/logs
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /home/aclifton/rf_fp/wandb/offline-run-20220705_125143-1x53nj8b
wandb: Find logs at: ./wandb/offline-run-20220705_125143-1x53nj8b/logs

Is there a way to limit the number of print and log statements to 1 when using accelerate? For isntance, I’d like to have the following printed out once:

INFO: CREATED TrainRun OBJECT
INFO: is_accelerate = True
INFO: device = cuda:1
----------------------------------------------------------------------------------------------------

Thanks in advance for your help! Please let me know if there is any other information I can provide.

[muellerzr]

Thanks for all these questions! It’s made us realize that there should be some better guides out there + some key discussion posts here on the forum in regards to features in Accelerate

What you want right now is the Logging capability inside Accelerate:

from accelerate.logging import get_logger

logger = get_logger(__name__)

This provides the same functionality as logging.Logger but will also make special care for if you want something posted on only the main process or across all processes.

For example, if it should be done only on the main process you can pass that in:

# Will be logged n times (where n = number of processes)
logger.info("My log", main_process_only=False)
# Will be logged once on the main process
logger.debug("My log", main_process_only=True)

Can you try using that and see if you still have issues with what you’re trying to accomplish? We can work towards that further and see if our own logging API needs improvement!

[aclifton314]

@muellerzr I’m immensely grateful for the Accelerate library. It has made my large model and dataset experimentation much more streamlined and faster. Thank you for all your efforts!

Incorporating your suggestions into my log_it function works great! Here is what I have now:

def log_it(self, str_to_log: Union[str, List[str]]):
        if type(str_to_log) is str:
            if self.is_accelerate:
                self.accelerate_logger.info(str_to_log, main_process_only=True)
            else:
                logging.info(str_to_log)
        else:
            if self.is_accelerate:
                for log_str in str_to_log:
                    self.accelerate_logger.info(str_to_log, main_process_only=True)
            else:
                for log_str in str_to_log:
                    logging.info(log_str)

Do you have any suggestions for the print_it() function?

[muellerzr]

@aclifton314

Here is what I have now:

I’d actually simplify this further potentially by declaring a Logging.logger and store that in your class. E.g.:

# Somewhere in init or before your init and pass it to the class constructor
self.logger = logging.getLogger(__name__)

And then when you declare your is_accelerate:

if is_accelerate:
    self.logger = get_logger(__name__)

(Since one is get_logger and the other is getLogger you can just import from accelerate import get_logger at the top I think)

This reduces your log_it to be:

def log_it(self, str_to_log: Union[str, List[str]]):
    if isinstance(str_to_log, str):
        # Note that the accelerate logger has main_process_only=True by default
        self.logger.info(str_to_log)
    else:
        for log_str in str_to_log:
            self.logger.info(str_to_log)

For print_it just use accelerator.print():

Accelerator

You could do something similar to what I did early by setting something like a printer attribute in your class, e.g.:

# somewhere in init
self.printer = print

# when is_accelerate gets made
if is_accelerate:
    self.printer = self.accelerator.print

and now our print_it function:

def print_it(self, str_to_print: Union[str, List[str]]):
    if isinstance(str_to_print, str):
        self.printer(str_to_print)
    else:
        for print_str in str_to_print:
            self.printer(print_str )

This will only print once per server for you

[aclifton314]

@muellerzr thank you for the response and suggestion. One question. In the following:

def log_it(self, str_to_log: Union[str, List[str]]):
    if isinstance(str_to_log, str):
        # Note that the accelerate logger has main_process_only=True by default
        self.logger.info(str_to_log)
    else:
        for log_str in str_to_log:
            self.logger.info(str_to_log)

if is_accelerate = True then self.logger.info(str_to_log) will print str_to_log for each process, which I can fix with the keyword argument main_process_only=True. But if is_accelerate = False then won’t main_process_only=True cause an error since self.logger = logging.getLogger(__name__). I don’t know for sure, but I assume the logger from the python module logging doesn’t have that keyword argument.

[muellerzr]

@aclifton314

rint str_to_log for each process, which I can fix with the keyword argument main_process_only=True

It won’t, because main_process_only=True is the default.

[aclifton314]

@muellerzr

hat the accelerate logger has main_process_only=True by default
        s

Ah yes, I see. I’ll give these two a shot and see how they work.

[aclifton314]

Works great. Thank you so much!

[aclifton314]

The above solutions worked when I implemented my own print_it and log_it methods. However, it seems that there are some default logging statements within the HF framework that still get printed several times. Here are some examples:

INFO: Added key: store_based_barrier_key:1 to store for rank: 3
INFO: Added key: store_based_barrier_key:1 to store for rank: 2
INFO: Added key: store_based_barrier_key:1 to store for rank: 1
INFO: Added key: store_based_barrier_key:1 to store for rank: 0
INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
DEBUG: git repository is invalid
DEBUG: git repository is invalid
DEBUG: git repository is invalid
DEBUG: git repository is invalid

and

DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
Flattening the indices:  20%|██████████████████████████▊                                                                                                           | 1/5 [00:00<00:01,  3.55ba/s]DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
Flattening the indices:  40%|█████████████████████████████████████████████████████▌                                                                                | 2/5 [00:00<00:00,  3.48ba/s]DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
Flattening the indices:  60%|████████████████████████████████████████████████████████████████████████████████▍                                                     | 3/5 [00:00<00:00,  3.53ba/s]DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
Flattening the indices: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00,  3.92ba/s]
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
Flattening the indices:   0%|                                                                                                                                              | 0/1 [00:00<?, ?ba/s]DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
Flattening the indices: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 14.17ba/s]
INFO: Saved dataset to /home/aclifton/rf_fp/proto_data/rffp_dataset
INFO: time to create rffp_data: 4.077894926071167 s
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/accuracy/accuracy.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/f1/f1.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/recall/recall.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): s3.amazonaws.com:443
DEBUG: https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/metrics/precision/precision.py HTTP/1.1" 200 0
DEBUG: Starting new HTTPS connection (1): raw.githubusercontent.com:443
DEBUG: https://raw.githubusercontent.com:443 "HEAD /huggingface/datasets/2.1.0/metrics/precision/precision.py HTTP/1.1" 200 0

These are not outputs that I explicitly ask to be printed out. It seems they come from the datasets library or somewhere else. Is it possible to have these statements printed out only in the local main process?

[aclifton314]

Here’s another good example that occurs during a training loop:

   EPOCH 1/1:   0%|                                                                                                                                                     | 0/4456 [00:00<?, ?it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   0%|▎                                                                                                                                            | 8/4456 [00:00<02:15, 32.95it/s]INFO: Reducer buckets have been rebuilt in this iteration.
INFO: Reducer buckets have been rebuilt in this iteration.
INFO: Reducer buckets have been rebuilt in this iteration.
INFO: Reducer buckets have been rebuilt in this iteration.
INFO: Successfully logged to WandB
   EPOCH 1/1:   0%|▌                                                                                                                                           | 16/4456 [00:00<01:55, 38.46it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|▊                                                                                                                                           | 24/4456 [00:00<01:58, 37.52it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|█                                                                                                                                           | 32/4456 [00:00<01:58, 37.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|█▎                                                                                                                                          | 40/4456 [00:01<01:59, 36.95it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|█▌                                                                                                                                          | 48/4456 [00:01<01:59, 36.81it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|█▊                                                                                                                                          | 56/4456 [00:01<01:59, 36.73it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   1%|██                                                                                                                                          | 64/4456 [00:01<01:59, 36.70it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|██▎                                                                                                                                         | 72/4456 [00:01<01:59, 36.69it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|██▌                                                                                                                                         | 80/4456 [00:02<01:59, 36.67it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|██▊                                                                                                                                         | 88/4456 [00:02<01:59, 36.61it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|███                                                                                                                                         | 96/4456 [00:02<01:59, 36.55it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   2%|███▏                                                                                                                                       | 104/4456 [00:02<01:59, 36.53it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|███▍                                                                                                                                       | 112/4456 [00:03<01:58, 36.54it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|███▋                                                                                                                                       | 120/4456 [00:03<01:58, 36.55it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|███▉                                                                                                                                       | 128/4456 [00:03<01:58, 36.57it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|████▏                                                                                                                                      | 136/4456 [00:03<01:58, 36.61it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|████▍                                                                                                                                      | 144/4456 [00:03<01:57, 36.59it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   3%|████▋                                                                                                                                      | 152/4456 [00:04<01:57, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|████▉                                                                                                                                      | 160/4456 [00:04<01:57, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|█████▏                                                                                                                                     | 168/4456 [00:04<01:57, 36.59it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|█████▍                                                                                                                                     | 176/4456 [00:04<01:56, 36.60it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|█████▋                                                                                                                                     | 184/4456 [00:05<01:56, 36.55it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|█████▉                                                                                                                                     | 192/4456 [00:05<01:56, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   4%|██████▏                                                                                                                                    | 200/4456 [00:05<01:56, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|██████▍                                                                                                                                    | 208/4456 [00:05<01:56, 36.56it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|██████▋                                                                                                                                    | 216/4456 [00:05<01:56, 36.51it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|██████▉                                                                                                                                    | 224/4456 [00:06<01:55, 36.49it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|███████▏                                                                                                                                   | 232/4456 [00:06<01:55, 36.46it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   5%|███████▍                                                                                                                                   | 240/4456 [00:06<01:55, 36.49it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|███████▋                                                                                                                                   | 248/4456 [00:06<01:55, 36.52it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|███████▉                                                                                                                                   | 256/4456 [00:06<01:55, 36.50it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|████████▏                                                                                                                                  | 264/4456 [00:07<01:54, 36.51it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|████████▍                                                                                                                                  | 272/4456 [00:07<01:54, 36.50it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|████████▋                                                                                                                                  | 280/4456 [00:07<01:54, 36.46it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   6%|████████▉                                                                                                                                  | 288/4456 [00:07<01:54, 36.49it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|█████████▏                                                                                                                                 | 296/4456 [00:08<01:53, 36.50it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|█████████▍                                                                                                                                 | 304/4456 [00:08<01:53, 36.50it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|█████████▋                                                                                                                                 | 312/4456 [00:08<01:53, 36.46it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|█████████▉                                                                                                                                 | 320/4456 [00:08<01:53, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   7%|██████████▏                                                                                                                                | 328/4456 [00:08<01:53, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|██████████▍                                                                                                                                | 336/4456 [00:09<01:52, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|██████████▋                                                                                                                                | 344/4456 [00:09<01:52, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|██████████▉                                                                                                                                | 352/4456 [00:09<01:52, 36.47it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|███████████▏                                                                                                                               | 360/4456 [00:09<01:52, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|███████████▍                                                                                                                               | 368/4456 [00:10<01:52, 36.48it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   8%|███████████▋                                                                                                                               | 376/4456 [00:10<01:51, 36.45it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|███████████▉                                                                                                                               | 384/4456 [00:10<01:51, 36.44it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|████████████▏                                                                                                                              | 392/4456 [00:10<01:51, 36.41it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|████████████▍                                                                                                                              | 400/4456 [00:10<01:51, 36.44it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|████████████▋                                                                                                                              | 408/4456 [00:11<01:51, 36.43it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:   9%|████████████▉                                                                                                                              | 416/4456 [00:11<01:50, 36.45it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|█████████████▏                                                                                                                             | 424/4456 [00:11<01:50, 36.41it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|█████████████▍                                                                                                                             | 432/4456 [00:11<01:50, 36.36it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|█████████████▋                                                                                                                             | 440/4456 [00:12<01:50, 36.37it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|█████████████▉                                                                                                                             | 448/4456 [00:12<01:50, 36.37it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|██████████████▏                                                                                                                            | 456/4456 [00:12<01:49, 36.40it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  10%|██████████████▍                                                                                                                            | 464/4456 [00:12<01:49, 36.43it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|██████████████▋                                                                                                                            | 472/4456 [00:12<01:49, 36.44it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|██████████████▉                                                                                                                            | 480/4456 [00:13<01:49, 36.43it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|███████████████▏                                                                                                                           | 488/4456 [00:13<01:48, 36.44it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|███████████████▍                                                                                                                           | 496/4456 [00:13<01:48, 36.39it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|███████████████▋                                                                                                                           | 504/4456 [00:13<01:48, 36.39it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  11%|███████████████▉                                                                                                                           | 512/4456 [00:14<01:48, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|████████████████▏                                                                                                                          | 520/4456 [00:14<01:48, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|████████████████▍                                                                                                                          | 528/4456 [00:14<01:48, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|████████████████▋                                                                                                                          | 536/4456 [00:14<01:47, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|████████████████▉                                                                                                                          | 544/4456 [00:14<01:47, 36.33it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  12%|█████████████████▏                                                                                                                         | 552/4456 [00:15<01:47, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|█████████████████▍                                                                                                                         | 560/4456 [00:15<01:47, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|█████████████████▋                                                                                                                         | 568/4456 [00:15<01:46, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|█████████████████▉                                                                                                                         | 576/4456 [00:15<01:46, 36.38it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|██████████████████▏                                                                                                                        | 584/4456 [00:16<01:46, 36.37it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|██████████████████▍                                                                                                                        | 592/4456 [00:16<01:46, 36.35it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  13%|██████████████████▋                                                                                                                        | 600/4456 [00:16<01:46, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|██████████████████▉                                                                                                                        | 608/4456 [00:16<01:45, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|███████████████████▏                                                                                                                       | 616/4456 [00:16<01:45, 36.34it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|███████████████████▍                                                                                                                       | 624/4456 [00:17<01:45, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|███████████████████▋                                                                                                                       | 632/4456 [00:17<01:45, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  14%|███████████████████▉                                                                                                                       | 640/4456 [00:17<01:45, 36.30it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|████████████████████▏                                                                                                                      | 648/4456 [00:17<01:44, 36.29it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|████████████████████▍                                                                                                                      | 656/4456 [00:17<01:44, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|████████████████████▋                                                                                                                      | 664/4456 [00:18<01:44, 36.25it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|████████████████████▉                                                                                                                      | 672/4456 [00:18<01:44, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|█████████████████████▏                                                                                                                     | 680/4456 [00:18<01:44, 36.27it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  15%|█████████████████████▍                                                                                                                     | 688/4456 [00:18<01:43, 36.27it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|█████████████████████▋                                                                                                                     | 696/4456 [00:19<01:43, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|█████████████████████▉                                                                                                                     | 704/4456 [00:19<01:43, 36.31it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|██████████████████████▏                                                                                                                    | 712/4456 [00:19<01:43, 36.29it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|██████████████████████▍                                                                                                                    | 720/4456 [00:19<01:42, 36.31it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  16%|██████████████████████▋                                                                                                                    | 728/4456 [00:19<01:42, 36.33it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|██████████████████████▉                                                                                                                    | 736/4456 [00:20<01:42, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|███████████████████████▏                                                                                                                   | 744/4456 [00:20<01:42, 36.32it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|███████████████████████▍                                                                                                                   | 752/4456 [00:20<01:42, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|███████████████████████▋                                                                                                                   | 760/4456 [00:20<01:41, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|███████████████████████▉                                                                                                                   | 768/4456 [00:21<01:41, 36.26it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  17%|████████████████████████▏                                                                                                                  | 776/4456 [00:21<01:41, 36.25it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|████████████████████████▍                                                                                                                  | 784/4456 [00:21<01:41, 36.26it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|████████████████████████▋                                                                                                                  | 792/4456 [00:21<01:40, 36.28it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|████████████████████████▉                                                                                                                  | 800/4456 [00:21<01:40, 36.24it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|█████████████████████████▏                                                                                                                 | 808/4456 [00:22<01:40, 36.18it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|█████████████████████████▍                                                                                                                 | 816/4456 [00:22<01:40, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  18%|█████████████████████████▋                                                                                                                 | 824/4456 [00:22<01:40, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|█████████████████████████▉                                                                                                                 | 832/4456 [00:22<01:40, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|██████████████████████████▏                                                                                                                | 840/4456 [00:23<01:39, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|██████████████████████████▍                                                                                                                | 848/4456 [00:23<01:39, 36.19it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|██████████████████████████▋                                                                                                                | 856/4456 [00:23<01:39, 36.19it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  19%|██████████████████████████▉                                                                                                                | 864/4456 [00:23<01:39, 36.19it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|███████████████████████████▏                                                                                                               | 872/4456 [00:23<01:38, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|███████████████████████████▍                                                                                                               | 880/4456 [00:24<01:38, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|███████████████████████████▋                                                                                                               | 888/4456 [00:24<01:38, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|███████████████████████████▉                                                                                                               | 896/4456 [00:24<01:38, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|████████████████████████████▏                                                                                                              | 904/4456 [00:24<01:38, 36.19it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  20%|████████████████████████████▍                                                                                                              | 912/4456 [00:25<01:37, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|████████████████████████████▋                                                                                                              | 920/4456 [00:25<01:37, 36.24it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|████████████████████████████▉                                                                                                              | 928/4456 [00:25<01:37, 36.22it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|█████████████████████████████▏                                                                                                             | 936/4456 [00:25<01:37, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|█████████████████████████████▍                                                                                                             | 944/4456 [00:25<01:36, 36.21it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  21%|█████████████████████████████▋                                                                                                             | 952/4456 [00:26<01:36, 36.23it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|█████████████████████████████▉                                                                                                             | 960/4456 [00:26<01:36, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▏                                                                                                            | 968/4456 [00:26<01:36, 36.20it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▍                                                                                                            | 976/4456 [00:26<01:36, 36.18it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▋                                                                                                            | 984/4456 [00:27<01:35, 36.18it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▉                                                                                                            | 992/4456 [00:27<01:35, 36.17it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  22%|██████████████████████████████▉                                                                                                           | 1000/4456 [00:27<01:35, 36.15it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|███████████████████████████████▏                                                                                                          | 1008/4456 [00:27<01:35, 36.13it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|███████████████████████████████▍                                                                                                          | 1016/4456 [00:27<01:35, 36.12it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|███████████████████████████████▋                                                                                                          | 1024/4456 [00:28<01:35, 36.12it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|███████████████████████████████▉                                                                                                          | 1032/4456 [00:28<01:34, 36.13it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  23%|████████████████████████████████▏                                                                                                         | 1040/4456 [00:28<01:34, 36.14it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|████████████████████████████████▍                                                                                                         | 1048/4456 [00:28<01:34, 36.12it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|████████████████████████████████▋                                                                                                         | 1056/4456 [00:29<01:34, 36.12it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|████████████████████████████████▉                                                                                                         | 1064/4456 [00:29<01:33, 36.14it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|█████████████████████████████████▏                                                                                                        | 1072/4456 [00:29<01:33, 36.14it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|█████████████████████████████████▍                                                                                                        | 1080/4456 [00:29<01:33, 36.16it/s]INFO: Successfully logged to WandB
   EPOCH 1/1:  24%|█████████████████████████████████▋                                                                                                        ```

[muellerzr]

@aclifton314

These are not outputs that I explicitly ask to be printed out. It seems they come from the datasets library or somewhere else. Is it possible to have these statements printed out only in the local main process?

Yep, take a peek in the transformers examples to see how this is done here

@aclifton314

Here’s another good example that occurs during a training loop:

That one I’m unsure of

[aclifton314]

That works! Thanks @muellerzr!!