Tool calling gets stuck in an infinite loop

John6666 · April 11, 2025, 4:12am

There is probably something wrong with the function calling in all models. It may be a bug in TGI.

github.com/huggingface/text-generation-inference

Function/tool calling never resolves

opened 06:23PM - 02 Feb 25 UTC

### Description When using the inference client with function calling, models s…eem to never resolve their calls. As we know, typically, with the OpenAI pattern, the simplest function/tool call is a series of messages of various roles (system, user, assistant, tool) organized like this: system → user ("what's the weather?") → assistant (tool_calls) → tool (result: "4ºC") → assistant (content: "it's 4ºC") The HF docs seem to indicate this is the same pattern, although the messages have some minor differences (e.g. description: null, which never happens with OpenAI). When using the Python inference client, these tool_calls never resolve even after functions are called and their return values are included and seemingly properly referenced. Instead, they look like this: system → user ("what's the weather?") → assistant (tool_calls) → tool (result: "4ºC") → assistant (tool_calls) … Instead of returning a text completion, the HF inference client returns the same "assistant" message specifying a required tool_calls. In OpenAI, they resolve to a typical "assistant" message with token content if the function calls have been satisfied and no further calls are required. Models used that exhibit this behavior: - NousResearch/Hermes-3-Llama-3.1-8B - Qwen/Qwen2.5-72B-Instruct - meta-llama/Meta-Llama-3-8B-Instruct --- It's worth noting that Mistral models also error out, specifying that a 9-character alphanumeric string is required for the `tool_call_id`. Now, the models themselves don't provide such IDs, so we need to supply them ourselves. But even when doing so, the same error occurs, that 9-char identifiers are missing. (e.g. mistralai/Mistral-7B-Instruct-v0.3) The JavaScript client also fails with the above errors, and also a third: "An error occurred while fetching the blob". ### System Info - macOS 15.2 - Python 3.13.1 - huggingface_hub 0.28.1 ### Information - [ ] Docker - [x] The CLI directly ### Tasks - [x] An officially supported command - [ ] My own modifications ### Reproduction Gist of sample error code is here: https://gist.github.com/awmartin/c64c84fbbdc3a9f0c2ce6e5ae0dab3dc 1. Provide API token 2. python inference-tool-calls.py A message results that's unexpected. I expected this to be a typical message with a string content, something like, "It's 4 degrees today." Instead, it just repeats the assistant message with the original tool_call message: [ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'unit': 'Celsius', 'location': 'Philadelphia, PA, US'}, name='get_current_temperature', description=None), id='0', type='function')]), logprobs=None)] ### Expected behavior I expected a message that resolved to something similar to "It's 4 degrees Celsius today" rather than the tool_call message repeated.

Topic		Replies	Views
Function calling not working with inference clients on (seemingly) any model Beginners	10	747	February 8, 2025
Error using function callbacks on custom inference endpoint Beginners	1	25	August 2, 2025
Inference API Issues Beginners	0	2722	September 9, 2021
Inference API model timeout (Flan-UL2) Inference Endpoints on the Hub	1	893	May 26, 2023
Inference API returns Unkown Error 🤗Hub	1	652	November 15, 2021

Tool calling gets stuck in an infinite loop

Related topics