How to prevent "filter" in vit.Finetune


I am trying to finetune the vit-base-patch16-224-in21k.

When I have a limited amount of images training starts immediately with ok results after 1 dy (local GPU).

When I add more images (say double it) training DOES NOT start but the script is "filter"ing. Takes 4 days.

Is it correct that the execution of this step depends on the amount of images provided? Can I jump this step even with more images provided?