Thanks for the answer @RaushanTurganbay !
I will start with that and try it out.
Regarding use case -
The use case is that we have a lot of long context examples that we repeatedly query, and so it doesn’t make sense to recalculate everything for every query. It makes more sense to reload the KV values for every query.
Does that make sense?