Do you have a way to estimate the costs for various kinds of training asks (ie, PT & FT) by inputing the coarse configuration like model type, dataset size, and epochs?
Thats hard to estimate since you don’t know which batch size fits. Additionally is training time depended on more factors, like if you eval during training or use mixed precision etc.
The best is to give it a try and see what the throughput is
Thanks @philschmid. These are great points.
Let’s say you’ve wall-timed one training “step” and determined the right batch size and FP precision that works for you. And let’s say you validate after each epoch. Is that closer to something you could ballpark a formula for?