Error using function callbacks on custom inference endpoint

Beginners
1

Hi guys, I’m having some problems related to the tool calling
testing it with my custom inference endpoint

using this example:

const testTools = [
      {
        "type": "function",
        "function": {
          "name": "get_number",
          "description": "Get the number",
          "parameters": {
            "type": "object",
            "properties": {
              "number": {
                "type": "string",
                "description": "the number to get",
              },
            },
            "required": ["number"],
          },
        },
      },
    ]

I’m not able to make it work
without the tool, it works perfectly, but adding it to the request payload makes the response something like this:

"error\\\": \\\"\\\\\\\"Tool error: Failed to parse generated text: expected value at line 1 column 1 \\\\\\\\\\\\\\\"<|python_tag

this is the call I’m doing

    const response = await inferenceClient.chatCompletion({
      endpointUrl: endpointURL,
      messages: messages.map((message) => ({
        role: message.role,
        content: message.content,
      })),
      max_tokens: xxx,
      temperature: xxx,
      tools: testTools,
      tool_choice: 'auto'
    });

Maybe I’m missing something? or is anyone having the same problem?
thanks in advance for your help!

1 Like
2

There seems to be some incompatibility between TGI and Llama’s Function Calling, so you may need to use a slightly hacky workaround.

Resources