Decoder only model - how to have it not include the prompt in its output?

[amperie]

Using decoder only models it seems that they always output the prompt first with the answer after.

Is there any way to supress the model from spitting out the prompt first and just have it output the answer?

T5, which is an encoder-decoder model has no problem doing this and giving out only the answer.

Possible? Am I missing something obvious?

An example of what I’m talking about is below. (I shortened the context). Every time, it outputs all of the input, with the Yes/No at the very end.

Answer the following question with a yes or no with the context given. Answer with only one word - yes or no. Question: Do obese patients with idiopathic pulmonary fibrosis have a higher 90-day mortality risk with bilateral lung transplantation?
Context: (‘Obese patients with idiopathic pulmonary fibrosis (IPF) … BMI of 18.5 to 30 kg/m(2).’)
Remember to only answer with one word - yes or no.

Yes

Thanks in advance!

[RaushanTurganbay]

Hi!

If you want only the generated text for for decoder-only models, you can crop the output manually as follows:

input_length = inputs.input_ids.shape[-1]
output = model.generate(**inputs)
output_only_answer = output[:, input_length: ]

[amperie]

Ok yeah I guess there’s no way to supress it since this is what decoders do, put one word at a time at the end of the input… thanks!