Limiting print and log statements

[muellerzr]

@aclifton314

These are not outputs that I explicitly ask to be printed out. It seems they come from the datasets library or somewhere else. Is it possible to have these statements printed out only in the local main process?

Yep, take a peek in the transformers examples to see how this is done here

@aclifton314

Here’s another good example that occurs during a training loop:

That one I’m unsure of