pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=200)
# Run text generation pipeline with our next model
while True:
user_input = "write an sql statement"
prompt = f"<s>[INTS]{user_input}[/INST]"
result = pipe(prompt)
print(result[0]['generated_text'])
It outputs:
<s>[INTS]write an sql statement[/INST] SELECT * FROM table_name WHERE name = "mario"</s>\n\nWhat does the SQL statement mean?\n\nIn this SQL...
You can see that that the correct stop token is set in every single config and the pipeline still ignores the </s> and continues.
Ah sorry, I thought you want to generate even if there is stop token. Then you shouldn’t be setting them to None as I showed above.
If the model is still generating beyond eos_token set in the model.generation_config.eos_token_id, can you check if the generation config has forced_eos_tokens or min_length. Also it would be easier if you provide a reproducible code with the model checkpoint you’re using
No, I’m sorry, this was on me!
The script is about a fine tuned LLM (with LoRA). It tries to predict the SQL instruction from a user question and the database table.
Thanks a lot for sharing the notebook. Unfortunately I still can’t reproduce it on ‘main’. I am using an arbitrary adapter as I don’t have access to the one you had in Colab, so my wild guess is that there’s smth in the adapter config as the version in Colab is the latest one.
Let me know if you can provide the whole script with the adapter, and I will try to find the bug!
No, thank you so much for even trying to help me.
But please don’t spend too much time on it but if you can not reproduce it with these script I will go crazy.
The entire training script: Google Colab
(If you want to try out the training script you just have to let it run for maybe 3-5min and then you can stop the cell. By then the model will have learned the rough desired structure and will already fail to stop at the stop token)
@VitalContribution hey, I ran the notebook with your adapter and I found the error. The generated “” tokens are not EOS, i.e. the model is generating ids: [829, 29879, 29958]`which are together decoded as “”.
I can’t say right now what went wrong with training, or how to train properly since I’m prioritizing other tasks. A workaround here will be to use a stop_strings as a kwarg to pipeline, which stops when a custom string is generated. See docs for more details