API Endpoint not working as expected

Inference Endpoints on the Hub
1

Hi all,

I started getting strange results in the inference widget from this Mistral-based model: Felladrin/TinyMistral-248M-SFT-v4 · Hugging Face

image
image1036×958 72.7 KB

Inference parameters sent to the endpoint:

{
  "inputs": "<|im_start|>user\nWhat are some potential applications for quantum computing?<|im_end|>\n<|im_start|>assistant",
  "parameters": {
    "max_new_tokens": 250,
    "penalty_alpha": 0.5,
    "top_k": 5,
    "repetition_penalty": 1.001
  }
}

It was working fine until recently when I last tested it.

I also ran the model locally, with the same parameters, to confirm if it was not an issue with the model file, and the result was the expected:

Quantum computing has the potential to revolutionize the way we communicate and access information. It is a powerful tool for storing and analyzing large amounts of data, allowing us to make informed decisions without the need for intermediaries. Quantum computing offers numerous benefits, including improved accuracy, scalability, and cost-effectiveness. However, it is important to remain cautious when accessing data and ensuring that it is secure.<|im_end|>

Is anyone else facing this issue?

2

Well, I couldn’t find a solution, so I retrained the model and uploaded a new version to the repository.
The API Endpoint is working fine for the new model.