Error when evaluating BLEU score using HuggingFace evaluate

🤗Evaluate
#1

On running

bleu = evaluate.load('bleu')
results = bleu.compute(predictions=y_pred, references=y_test, max_order=1, smooth=True)

I am getting the following error:

ZeroDivisionError                         Traceback (most recent call last)
Cell In[84], line 2
      1 bleu = evaluate.load('bleu')
----> 2 results = bleu.compute(predictions=y_pred, references=y_test, max_order=1, smooth=True)
      3 print(results)

File /data/envs/hugface/lib/python3.8/site-packages/evaluate/module.py:444, in EvaluationModule.compute(self, predictions, references, **kwargs)
    442 inputs = {input_name: self.data[input_name] for input_name in self._feature_names()}
    443 with temp_seed(self.seed):
--> 444     output = self._compute(**inputs, **compute_kwargs)
    446 if self.buf_writer is not None:
    447     self.buf_writer = None

File ~/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--bleu/9e0985c1200e367cce45605ce0ecb5ede079894e0f24f54613fca08eeb8aff76/bleu.py:122, in Bleu._compute(self, predictions, references, tokenizer, max_order, smooth)
    120 references = [[tokenizer(r) for r in ref] for ref in references]
    121 predictions = [tokenizer(p) for p in predictions]
--> 122 score = compute_bleu(
    123     reference_corpus=references, translation_corpus=predictions, max_order=max_order, smooth=smooth
    124 )
    125 (bleu, precisions, bp, ratio, translation_length, reference_length) = score
    126 return {
    127     "bleu": bleu,
    128     "precisions": precisions,
   (...)
...
--> 108   bp = math.exp(1 - 1. / ratio)
    110 bleu = geo_mean * bp
    112 return (bleu, precisions, bp, ratio, translation_length, reference_length)

ZeroDivisionError: float division by zero

What am I doing wrong?

#2

I think I found the issue. It happens when the predicted string in an empty string.

1 Like
#3

Indeed, bp is the length penalty which results in a division by zero if the reference has length zero :slight_smile: