Token restriction via the Huggin face API

[Locon213]

Hi, I use the Botpress service for my chatbots. And I have just started using integration with Huggin face. But what exactly is the matter is that the number of tokens is limited to me, that is, My chatbots simply do not add. The same thing is observed on checking the model through the API, that is, on the model page. Who can help me figure this out and increase the number of tokens for models. Ps. The translation was done by Google translate

[John6666]

I’m also DeepL. No problem.
Normally this is the way to increase max_new_tokens, but without a way to specify it on the botpress side, there is no way to do it.

How do I increase max_new_tokens

@gkrishnan I’m late to the post but you can always manually pass in the model/pipeline: from transformers import pipeline, AutoModelForCausalLM, AutoTokenizer from langchain.llms import HuggingFacePipeline model = AutoModelForCausalLM.from_pretrained(model_path) tokenizer = AutoTokenizer.from_pretrained(model_path) gen = pipeline('text-generation', model=model, tokenizer=tokenizer, max_new_tokens=200) llama_llm = HuggingFacePipeline(pipeline=gen)

Confused about max_length and max_new_tokens

I know I’m late now. But for any future preferences. what the max_length and max_new_tokens do. In max_length we get the maximum length including the input and output tokens. But in max_new_tokens we get the maximum output excluding the output. Let me show you using the code import torch from transformers import AutoModelForCausalLM, AutoTokenizer,AutoModelForSeq2SeqLM torch.set_default_device("cuda") model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", tru…

The botpress people seem to be on HF, so there’s always the option of sending them a direct mentions and asking them. (@+username)

Botpress (Botpress)