Speculative decoding is a hot topic right now!
Here is a blog post I wrote proposing mentored decoding, a new variant of speculative decoding. It maximizes the probability to accept the draft tokens while maintaining the Kullback-Leibler divergence between the resulting distribution and the target distribution below a constant D. Speculative decoding can be seen as a special case of mentored decoding for D = 0.
Thanks to @joaogante who got me interested in speculative decoding with his blog post.
I am a bit late, but want to praise that you wrote an excellent blog, which is highly appreciate to many researchers …
Would love to follow every blog you write!
Hi @Jung. Many thanks, that’s very kind of you. When I publish a new blog post (quite soon hopefully), I’ll mention it on Twitter, so you can follow me (@vivien000000) there to be notified.