Tool calling gets stuck in an infinite loop

[hadleywickham]

I’m having problems using tool calling with the inference endpoint. I’m following the documentation (such as it is), and the examples in Tool Use, Unified. But the API doesn’t seem to recognise that I’ve returned a result, and just returns another tool call request:

{
  "messages": [
    {
      "role": "system",
      "content": "Be very terse, not even punctuation."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "What's the current date in Y-M-D format?"
        }
      ]
    },
    {
      "role": "assistant",
      "tool_calls": [
        {
          "id": "0",
          "function": {
            "name": "tool_001",
            "arguments": "{}"
          },
          "type": "function"
        }
      ]
    },
    {
      "role": "tool",
      "content": "2024-01-01",
      "name": "tool_001",
      "tool_call_id": "0"
    },
    {
      "role": "assistant",
      "tool_calls": [
        {
          "id": "0",
          "function": {
            "name": "tool_001",
            "arguments": "{}"
          },
          "type": "function"
        }
      ]
    },
    {
      "role": "tool",
      "content": "2024-01-01",
      "name": "tool_001",
      "tool_call_id": "0"
    }
  ],
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "stream": false,
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "tool_001",
        "description": "Return the current date",
        "strict": true,
        "parameters": {
          "type": "object",
          "description": "",
          "properties": {},
          "required": [],
          "additionalProperties": false
        }
      }
    }
  ]
}```

Any ideas what I'm doing wrong?

[John6666]

There is probably something wrong with the function calling in all models. It may be a bug in TGI.

Function calling not working with inference clients on (seemingly) any model

I’m attempting to use function-calling capabilities with text inference models, but none of the ways I’ve attempted to do with the inference clients work. I use OpenAI’s function calling regularly, but Hugging Face capabilities seem fragile and unpredictable, but I suspect I’m just “doing it wrong.” I’d appreciate any insights into the following cases I’m seeing: JavaScript inference client fails If I just include the first message (e.g. “What’s the weather in X place?”), the models will prope…

Function/tool calling never resolves

[hadleywickham]

Thank you!