Cleaup_tokenization_spaces error

Beginners
1

I have set up ComfyUI on Ubuntu and am attempting to run the workflow downloaded from here (ComfyUI workflow for Flux (simple) | ComfyUI Workflow) for the first time. However, I am getting an warning clean_up_tokenization_spaces and then ComfyUI is shutdown - full console output below:

got prompt
model weight dtype torch.bfloat16, manual cast: None
model_type FLUX
/home/garrett/AI/ComfyUI/venv/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
./launch.sh: line 7: 47378 Killed                  python3 main.py

The Github bug pointed to says a parameter needs to be set to false but where do I change this?

2

Got the same error :smiling_face_with_tear:

3

You can add the flag clean_up_tokenization_spaces intro the

AutoTokenizer 
tokenizer = AutoTokenizer.from_pretrained(model_name,clean_up_tokenization_spaces=False)

Before, It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default.
The clean_up_tokenization_spaces wether to cleanup spaces after decoding, cleanup consists in removing potential artifacts like extra spaces. E.g., If you have state-of-the-art it will be encoded as state - of - the - art.