` transformers` '4.57.0.dev0' is not compatible with `evaluate`?

🤗Transformers
1

Hi, I try to use transformers ‘4.57.0.dev0’, but when I import evaluate'：

File “”, line 241, in _call_with_frames_removed
File “/home/–/dev_env/lib/python3.10/site-packages/evaluate/init.py”, line 29, in
from .evaluation_suite import EvaluationSuite
File “/home/–/dev_env/lib/python3.10/site-packages/evaluate/evaluation_suite/init.py”, line 10, in
from ..evaluator import evaluator
File “/home/–/dev_env/lib/python3.10/site-packages/evaluate/evaluator/init.py”, line 17, in
from transformers.pipelines import SUPPORTED_TASKS as SUPPORTED_PIPELINE_TASKS
File “/home/–/transformers/src/transformers/pipelines/init.py”, line 63, in
from .document_question_answering import DocumentQuestionAnsweringPipeline
File “/home/–/transformers/src/transformers/pipelines/document_question_answering.py”, line 30, in
from .question_answering import select_starts_ends
File “/home/–/transformers/src/transformers/pipelines/question_answering.py”, line 9, in
from ..data import SquadExample, SquadFeatures, squad_convert_examples_to_features

1 Like
2

You’ve run into a version mismatch problem between transformers and evaluate.

Here’s what’s happening:

  • You’re on a dev version of transformers (4.57.0.dev0).

  • That version moved/removed some internals (like SquadExample, SquadFeatures, and squad_convert_examples_to_features).

  • But your installed evaluate package still imports those old classes → import error.

:white_check_mark: How to Fix

  1. Use matching stable versions
    The easiest fix: install stable releases of both packages that are tested together. For example:

    pip install --upgrade transformers evaluate

    This will give you the latest released pair (e.g., transformers 4.44.x and evaluate 0.4.x).

  2. If you must stay on dev branch (4.57.0.dev0)

    • Pull evaluate from its latest GitHub main branch too:

      pip install git+https://github.com/huggingface/evaluate

    • This way both repos are in sync with ongoing refactors.

  3. Temporary workaround
    If you only need evaluation metrics (BLEU, accuracy, etc.), you can skip the broken imports by using:

    import evaluate
metric = evaluate.load("accuracy")

    That doesn’t rely on the pipelines imports.

:high_voltage: Recommendation

Unless you’re contributing to Hugging Face core dev, avoid the dev0 builds — they’re unstable and will break like this. Stick with the latest PyPI release.

2 Likes
3

Make sure you select “always use the latest environment”

1 Like