I am training a fine-tune of codellama using PEFT but not sure how to use the task_type parameter of LoraConfig. Should it be CAUSAL_LM or SEQ_2_SEQ_LM or something else? Does it have any affect?
The goal of my model is to parse an input for independent clauses in a sentence. For example, it would insert a delimiter, such as in this sentence: “the tea was on the stove and was at high temperature” , separating the independent clause from the subordinate clause. My training data is all in a single col and each row looks like this (where the → and are custom tokens I add to the tokenizer vocab and the is the EOS token):
“the tea was on the stove and was at high temperature → the tea was on the stove and was at high temperature ”