Function calling not working with inference clients on (seemingly) any model

John6666 · February 3, 2025, 2:00pm

I tried it twice with Qwen and twice with Zephyr. I wonder what the expected behavior is…

ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino, Italy'}, name='get_current_weather', description=None), id='0', type='function')]), logprobs=None)], created=1738590766, id='', model='Qwen/Qwen2.5-72B-Instruct', system_fingerprint='3.0.2-sha-b70f29d', usage=ChatCompletionOutputUsage(completion_tokens=30, prompt_tokens=312, total_tokens=342))   
ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino, Italy'}, name='get_current_weather', description=None), id='0', type='function')]), logprobs=None)], created=1738590769, id='', model='Qwen/Qwen2.5-72B-Instruct', system_fingerprint='3.0.2-sha-b70f29d', usage=ChatCompletionOutputUsage(completion_tokens=30, prompt_tokens=339, total_tokens=369)) 

ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(fu
nction=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino (Italy)'}, name='get_current_weather', description=None), id='0', type='function')]), logprobs=None)], created=1738590905, id='', model='HuggingFaceH4/zephyr-7b-beta', system_fingerprint='3.0.1-sha-bb9095a', usage=ChatCompletionOutputUsage(completion_tokens=41, prompt_tokens=300, total_tokens=341))
ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino, Italy'}, name='get_current_weather', description=None), id='0', type='function')]), logprobs=None)], created=1738590906, id='', model='HuggingFaceH4/zephyr-7b-beta', system_fingerprint='3.0.1-sha-bb9095a', usage=ChatCompletionOutputUsage(completion_tokens=40, prompt_tokens=101, total_tokens=141))

github.com/huggingface/huggingface_hub

Passing tool results to the LM

opened 04:53PM - 15 Oct 24 UTC

closed 11:46AM - 20 Jan 25 UTC

anakin87

bug

### Describe the bug Let's start by thanking you for this great resource :blu…e_heart: The `InferenceClient` supports tool calling, as explained [here](https://huggingface.co/docs/huggingface_hub/en/package_reference/inference_client#huggingface_hub.InferenceClient.chat_completion.example-4). In many use cases, **it is useful to pass back the tool call to the Language Model and also the tool result in a message from `tool` role**. In this way, the LM can for example respond in a human-readable way. This is supported in [HF Transformers](https://huggingface.co/docs/transformers/main/chat_templating#a-complete-tool-use-example). When using the `InferenceClient` (for Serverless Inference API or TGI), I'm struggling to find a way to reproduce this desired behavior. (I mostly experimented with Mistral and Llama models supporting tool/function calling, with similar results) @Wauplin @hanouticelina Is this supported or planned? Is there any workaround you suggest? So far, I've only tried to wrap the tool result in a message from `user` and this somehow works... Probably related issue (in TGI): https://github.com/huggingface/text-generation-inference/issues/2461 ### Reproduction ```python from huggingface_hub import InferenceClient client = InferenceClient("mistralai/Mistral-Nemo-Instruct-2407") messages = [ { "role": "system", "content": "Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous.", }, { "role": "user", "content": "What's the weather like in San Giustino (Italy) in Celsius?", }, ] tools = [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "format": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location.", }, }, "required": ["location", "format"], }, }, }] client.chat_completion(messages=messages, tools=tools, max_tokens=500, temperature=0.3) # this works great and produces a similar output: # ChatCompletionOutput(choices=[ChatCompletionOutputComplete(finish_reason='stop', index=0, message=ChatCompletionOutputMessage(role='assistant', content=None, tool_calls=[ChatCompletionOutputToolCall(function=ChatCompletionOutputFunctionDefinition(arguments={'format': 'celsius', 'location': 'San Giustino, Italy'}, name='get_current_weather', description=None), id='0', type='function')]), logprobs=None)], ...) # TRYING TO PASS BACK TOOL CALLS AND TOOL RESULT new_messages = [el for el in messages] id_ = "9Ae3bDc2F" # fake ID needed to use Mistral models tool_call = {"name": "get_current_weather", "arguments": {"location": "San Giustino, Italy", "format": "celsius"}} new_messages.append({"role": "assistant", "content":"", "tool_calls": [{"type": "function", "function": tool_call, "id": id_}]}) new_messages.append({"role": "tool", "name": "get_current_temperature", "content": "22.0", "tool_call_id": id_}) client.chat_completion(messages=new_messages, tools=tools, max_tokens=500, temperature=0.3) # HfHubHTTPError: 422 Client Error: Unprocessable Entity for url: https://api-inference.huggingface.co/models/mistralai/Mistral-Nemo-Instruct-2407/v1/chat/completions (Request ID: ...) # Template error: unknown filter: filter string is unknown (in <string>:79) ``` ### System info ```shell - huggingface_hub version: 0.25.2 ```

Topic		Replies	Views
Tool calling gets stuck in an infinite loop Inference Endpoints on the Hub	2	257	April 12, 2025
Dumb Question: Seeing that my inference API links not working Beginners	1	85	July 10, 2025
Function/tool calling using Transformer models 🤗Transformers	5	1025	July 17, 2025
Unable to access model with Inference Client Beginners	2	170	May 6, 2025
Cannot get tools to work: InferenceClient + hf-inference + Qwen/Qwen3-235B-A22B -- Internal Server Error Intermediate	3	41	June 17, 2025

Function calling not working with inference clients on (seemingly) any model

Related topics