How can I specify `stop_strings` in `generation_config.json`?

suhara · November 14, 2024, 6:01pm

Apparently, it’s not supported yet.

Feature to configure `stop_strings` in `generation_config.json` or other config files

opened 01:43AM - 02 Nov 24 UTC

Feature request Generation

### Feature request The transformer library should offer a way to configure `st…op_strings` and the tokenizer for it. `model.generate()` can take a `stop_strings` argument to use custom stop tokens for generation, but a tokenizer object needs to be passed as well. ``` model.generate(..., stop_strings=["<stop token>"], tokenizer=tokenizer) ``` If we add `stop_strings` to `generation_config.json`, which can be loaded correctly [code](https://github.com/huggingface/transformers/blob/33868a057c02f0368ba63bd1edb746be38fe3d90/src/transformers/generation/configuration_utils.py#L144-L145), it will return the following error, as it requires a tokenizer object, which cannot be defined in the config file. ``` >>> from transformers import AutoTokenizer, AutoModelForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained(model_path) >>> model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto") >>> model.generate(**tokenizer("Hi how are you?", return_tensors="pt", return_token_type_ids=False)) ... ValueError: There are one or more stop strings, either in the arguments to `generate` or in the model's generation config, but we could not locate a tokenizer. When generating with stop strings, you must pass the model's tokenizer to the `tokenizer` argument of `generate`. ``` ### Motivation The user shouldn't be bothered by adding extra arguments to `generate()` or `pipeline`. For example, [nvidia/Mistral-NeMo-Minitron-8B-Instruct](https://huggingface.co/nvidia/Mistral-NeMo-Minitron-8B-Instruct) needs to use `stop_strings` but so many people simply calls `generate()` without `stop_strings` and share complaints. ### Your contribution I'd be happy to create a PR but need guidance for the design choice.

Topic		Replies	Views
How does the text-generation pipeline know the special stop token? Beginners	8	3585	June 10, 2024
How to stop after generating "###" in transformers? Beginners	0	855	May 3, 2023
How to set stopping criteria in model.generate() when a certain word appears 🤗Transformers	3	3799	February 18, 2024
Implimentation of Stopping Criteria List Beginners	24	30564	January 24, 2025
How to add a stop sequence to a Pipeline? 🤗Transformers	0	302	July 29, 2024

How can I specify `stop_strings` in `generation_config.json`?

Related topics