You can read the LoRa paper : https://arxiv.org/pdf/2106.09685.pdf, at the beginning they said that using lora for finetuning by 3 because you don’t have to store the gradient and gradient momentum of the optimizer.
You can read the LoRa paper : https://arxiv.org/pdf/2106.09685.pdf, at the beginning they said that using lora for finetuning by 3 because you don’t have to store the gradient and gradient momentum of the optimizer.