Hi! I am new to the transformers and evaluate libraries but I am noticing that when trying to calculate perplexity my notebook will randomly fail with the error:
It seems to happen randomly. When I call model.generate I am passing the following parameters:
inputs, min_new_tokens=200, max_length=350, do_sample=do_sample, top_p=top_p, top_k=top_k
And here is my function for calculating perplexity:
from evaluate import load
def calc_perplexity(model_name, text):
perplexity = load("perplexity", module_type="metric")
results = perplexity.compute(predictions=text, model_id=model_name)
return round(results.get('mean_perplexity'),2)
The exact same parameters being passed on different runs will sometimes fail and sometimes not. Normally the first call to perplexity will work, but then the second call does not. Any ideas what could be happening here?
EDIT: It fails after a few consecutive calls, generally about 3
Solved: Well I am a goober. A string cleaning function I had was not returning in some edge scenarios and that caused a bunch of down stream issues. I’ll leave thi post up though in case anyone runs into something similiar.
