I have an input to an LLM (I am using GPT2 but any causal model should work), X, and a response string Y that I manually created. I would like to get P(Y|X), the probability that the LLM would output the response Y. How do I do this? .generate() , compute_transition_scores provides functionality for getting the probabilities of a generated output, but in this case my output is not generated by the LLM.
3 Likes
Did you discover a way to do this?