Following the example here, I can create compute perplexity for a model I have previously saved like this:
perplexity = load("perplexity", module_type="metric")
results = perplexity.compute(predictions=dataset,model_id='my-saved-model')
But this only lets me specify a model_id, which is then loaded. I can’t do something like this:
model = AutoModelForCausalLM.from_pretrained('my-saved-model')
perplexity = load("perplexity", module_type="metric")
results = perplexity.compute(predictions=dataset,model_id=model)
Is there an alternative way to use evaluate’s perplexity metric that doesn’t require me to point to a saved model on disk?
For larger context, I’m trying to follow the quantization pipeline here and want to use perplexity as my criterion. But this requires computing perplexity on the model as it’s being updated in memory. If evaluate’s perplexity metric is not the correct tool for this job, is there something else I should be using?