Combining tokenizer.decode and model.generate scores for probability prediction

Intermediate
1

I want to use the input_id level scores provided by model.generate(return_scores=True) with the decode function of the Tokenizer.

The logit scores are input_id level and the decoder decodes to token level.
My desired output for a prompt of “hello, how are you” would be "doing (p=0.8) today (p=0.8 * 0.7) sir (p=0.8 * 0.7 * 0.7) for example.

How can I achieve this?

2

Hey @Shaike04

Have a look at our documentation here. I believe the first example does precisely what you want! :hugs:

3

Thank you so much, I am looking into it.