The closest answer I got to this question is here. Anyone know if this means that LoRA does Q and V matrices, while QLoRA only does linear layers? Or does it mean that QLoRA also does Q and V on top of linear layers?
Hi,
Q and V are linear layers, so those are included when you train LoRa or QLoRa. QLoRa is the same as LoRa except that you also quantize the base model (turn it into 4 bits rather than 16 per parameter).