Novel, provably optimal, lossy variant of speculative decoding

Show and Tell
1

Hi!

Speculative decoding is a hot topic right now!

Here is a blog post I wrote proposing mentored decoding, a new variant of speculative decoding. It maximizes the probability to accept the draft tokens while maintaining the Kullback-Leibler divergence between the resulting distribution and the target distribution below a constant D. Speculative decoding can be seen as a special case of mentored decoding for D = 0.

Feedback welcome!

Thanks to @joaogante who got me interested in speculative decoding with his blog post.

2 Likes
2

I am a bit late, but want to praise that you wrote an excellent blog, which is highly appreciate to many researchers …

Would love to follow every blog you write!

3

Hi @Jung. Many thanks, that’s very kind of you. When I publish a new blog post (quite soon hopefully), I’ll mention it on Twitter, so you can follow me (@vivien000000) there to be notified.