Task_type parameter of LoraConfig

I am training a fine-tune of codellama using PEFT but not sure how to use the task_type parameter of LoraConfig. Should it be CAUSAL_LM or SEQ_2_SEQ_LM or something else? Does it have any affect?

The goal of my model is to parse an input for independent clauses in a sentence. For example, it would insert a delimiter, such as in this sentence: “the tea was on the stove and was at high temperature” , separating the independent clause from the subordinate clause. My training data is all in a single col and each row looks like this (where the → and are custom tokens I add to the tokenizer vocab and the is the EOS token):

“the tea was on the stove and was at high temperature → the tea was on the stove and was at high temperature ”

1 Like

Does task_type matters in LoraConfig, and if so, in what way?

I have the same question. Does task_type matter?

In the source code, task_type is not even there:

The task_type parameter is used in the superclass PeftConfig

3 Likes