Loading checkpoint shards is very slow. Same model and same machine, sometimes it takes less than 1 minute, but sometimes it takes more than 10 minutes. May I know what’s the reason and how to speed up the process?
check out your RAM vs VRAM usage… is it loading to your ram or GPU vram. typical vram is faster.
try quantization… it may be faster.