Is there a way to control the beginning of sentence, end of sentence tokens through the inference api? I could not find it in the documentation.
Hi @cristivlad ,
Currently there is no way to override those within the API.
We are adding and end_sequence
parameter to enable stopping the generation when using prompt-like generation (for GPT-Neo for instance).
For BOS, what did you want to do with it ?
Cheers,
Nicolas
1 Like
I tried end_sequence and indeed works, I would suggest updating the doc, thanks in any case!
data = json.dumps(
{
"inputs": "1+1=2\n2+2=",
"parameters": {
"max_length": 50,
"num_return_sequences": 1,
"return_text": False,
"return_full_text": False,
"do_sample": True,
"top_k": 50,
"top_p": 0.95,
"end_sequence": "\n"
},
"options": {
"wait_for_model": True,
},
}
)