Hi everyone, i’am starting a research project for arabic and the idea is to finetune multiple SOTA models then compare them , the thing is we are short in terms of budget so i would like to know if there is a way to estimate the cost giving the model size and the dataset size (assuming it will be done on an A100 80GB)
i.e : is there a magical formula to estimate the cost ?
Hugging Face has a model memory usage calculator that I guess can be used in calculating the number of A100 80Gs needed. https://huggingface.co/spaces/hf-accelerate/model-memory-usage
Not very useful in my case ! I’m more interested to know that given a model size let’s 560M params (8% of 7B since we are using LoRA) and a dataset size of let’s say 2B tokens, how much would it take to fine-tune that model for 1 epoch on a single A100 80gb ? I don’t even know if it’s possible to do such calculation but the HF AutoTrain (https://ui.autotrain.huggingface.co) gives a cost estimation before launching the training so i assumed it is possible after all 
@merve can you guide to who from the team can help with that ?