In order to fine tune prompt + completion tasks in LLM, it is usual to give less weight to the prompt tokens. For instance, the default usage in GPT3 training is 0.01 of the weight of the completion tokens. In other trainings I have seen just 0.0, and it seems that it is the only possibility in the …

Prompt loss weight instead of masking in generative models

SantoshScienceIO June 18, 2023, 2:39am 2

Did you ever figure it out?

2 Likes