Chapter 2 questions

Course
88

Hello.
Thanks for this fantastic course.
My question is wheteher any model can use tools. In ollama, there are certainmodels that do use tools.
And here, DeepSeek R1 is using tools in the example. I thought DeepSeek R1 cannot use tool. thanks

1 Like
89

In the case of smolagents, the model does not need to be particularly good at handling tools, because the system is designed so that when the model gives a command “in words”, smolagents executes the tool that corresponds to that command, without relying on the tool execution function inside the model. Well, there’s no disadvantage to being good at it…

90

i am taking smolagents course
the materials provided run in google colab notebook…
though, i want to try to run the materials locally as well using vscode…

i want to ask, how to run the following locally on vscode?
a) agent.from_hub(‘sergiopaniego/AlfredAgent’, trust_remote_code=True)"
b) agent.push_to_hub(‘AScythe/AlfredAgent’)

1 Like
91

Try this with Python 3.10 or later.

pip install -U huggingface_hub smolagents
1 Like
92

thanks for the response

A) for agent.push_to_hub…
i have several cases that i tried, first one is by running:

1 i run agent.push_to_hub(‘AScythe/AlfredAgent’), and got this error:
Traceback (most recent call last):
File “E:\Code\AI_Agents\smol_agents1.py”, line 164, in
agent.push_to_hub(“AScythe/display_image_tool”, token=huggingface_api_key)
^^^^^^^^^^^^^^^^^
AttributeError: ‘CodeAgent’ object has no attribute ‘push_to_hub’

2 i tried to push a single tool by running:
class SuperheroPartyThemeTool(Tool):
name = “superhero_party_theme_generator”
description = “”"
This tool suggests creative superhero-themed party ideas based on a category.
It returns a unique party theme idea.“”"

inputs = {
“category”: {
“type”: “string”,
“description”: “The type of superhero party (e.g., ‘classic heroes’, ‘villain masquerade’, ‘futuristic Gotham’).”,
}
}

output_type = “string”

def forward(self, category: str):
themes = {
“classic heroes”: “Justice League Gala: Guests come dressed as their favorite DC heroes with themed cocktails like ‘The Kryptonite Punch’.”,
“villain masquerade”: “Gotham Rogues’ Ball: A mysterious masquerade where guests dress as classic Batman villains.”,
“futuristic Gotham”: “Neo-Gotham Night: A cyberpunk-style party inspired by Batman Beyond, with neon decorations and futuristic gadgets.”
}

 return themes.get(category.lower(), "Themed party idea not found. Try 'classic heroes', 'villain masquerade', or 'futuristic Gotham'.")

party_theme_tool = SuperheroPartyThemeTool()

party_theme_tool.push_to_hub(“AScythe/display_image_tool”, token=huggingface_api_key)
and i got this error:
UnicodeEncodeError: ‘charmap’ codec can’t encode character ‘\U0001f440’ in position 39: character maps to

whatever i do, it seems like i cant push agents/tools to hub in vs code… may i kindly ask if you have sample code that you can run in your vscode?

1 Like
93

Hi

I face an issue

in the notebook unit2/smolagents/code_agents.ipynb
at step 9, I can’t connect to Langfuse
with this Error

AttributeError: module ‘smolagents’ has no attribute ‘ApiModel’

image
image1017×551 96.7 KB

Is it a problem due to smoleagent code modification ?

1 Like
94

Oh, ongoing issue…

95

In the (Behind the pipeline - Hugging Face NLP Course) “behind the pipeline” part I think there is a mistake:
it needs to be

  • Hidden size: The vector dimension of each model output.
    instead of
  • Hidden size: The vector dimension of each model input.

Right?

1 Like
96

Hmm… Maybe.

97

In chapter 2, when running ToolCallingAgent, I encountered an error, which I hope I could get some clarity on.

Background info:

  • I’m using my local machine, not Colab.
  • I’m using OpenAIServerModel instead of HfApiModel.
  • Model I’m using is “qwen25-coder-32b-instruct”.

Problem:

If I create an agent instance and then ran it with some text, I get a bad request error.

Error in generating tool call with model:
Error code: 400 - {'id': '', 'object': '', 'created': 0, 'model': '', 'choices': None, 'usage': {'prompt_tokens': 0, 'completion_tokens': 0, 'total_tokens': 0, 
'prompt_tokens_details': None, 'completion_tokens_details': None}, 'system_fingerprint': ''}
[Step 1: Duration 2.41 seconds]
Traceback (most recent call last):
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/smolagents/agents.py", line 1007, in step
    model_message: ChatMessage = self.model(
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/openinference/instrumentation/smolagents/_wrappers.py", line 287, in __call__
    output_message = wrapped(*args, **kwargs)
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/smolagents/models.py", line 1075, in __call__
    response = self.client.chat.completions.create(**completion_kwargs)
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/openai/_utils/_utils.py", line 279, in wrapper
    return func(*args, **kwargs)
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/openai/resources/chat/completions/completions.py", line 914, in create
    return self._post(
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/openai/_base_client.py", line 1242, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/openai/_base_client.py", line 919, in request
    return self._request(
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/openai/_base_client.py", line 1023, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'id': '', 'object': '', 'created': 0, 'model': '', 'choices': None, 'usage': {'prompt_tokens': 0, 'completion_tokens': 0, 'total_tokens': 0, 'prompt_tokens_details': None, 'completion_tokens_details': None}, 'system_fingerprint': ''}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/Niv/Workspace/ai-agent/main.py", line 116, in <module>
    agent.run("search for best music recommendations for a 80's themed party")
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/openinference/instrumentation/smolagents/_wrappers.py", line 128, in __call__
    agent_output = wrapped(*args, **kwargs)
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/smolagents/agents.py", line 323, in run
    return deque(self._run(task=self.task, max_steps=max_steps, images=images), maxlen=1)[0]
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/smolagents/agents.py", line 337, in _run
    raise e
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/smolagents/agents.py", line 334, in _run
    final_answer = self._execute_step(task, memory_step)
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/smolagents/agents.py", line 358, in _execute_step
    final_answer = self.step(memory_step)
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/openinference/instrumentation/smolagents/_wrappers.py", line 163, in __call__
    result = wrapped(*args, **kwargs)
  File "/opt/miniconda3/envs/ml-train/lib/python3.10/site-packages/smolagents/agents.py", line 1014, in step
    raise AgentGenerationError(f"Error in generating tool call with model:\n{e}", self.logger) from e
smolagents.utils.AgentGenerationError: Error in generating tool call with model:
Error code: 400 - {'id': '', 'object': '', 'created': 0, 'model': '', 'choices': None, 'usage': {'prompt_tokens': 0, 'completion_tokens': 0, 'total_tokens': 0, 'prompt_tokens_details': None, 'completion_tokens_details': None}, 'system_fingerprint': ''}

Following is the code:

MODEL_ID = "qwen25-coder-32b-instruct"

######## do it using tool calling agent ########
agent = ToolCallingAgent(tools=[DuckDuckGoSearchTool()], model=OpenAIServerModel(
    model_id=MODEL_ID, 
    api_base="https://api.lambdalabs.com/v1",
    api_key=os.getenv("INFERENCE_API_KEY")
))

agent.run("search for best music recommendations for a 80's themed party")
######### end  #########

However, if I do it by manually constructing the prompt and calling the tool, it works.
Following is the code that works, but not using ToolCallingAgent.

model = OpenAIServerModel(
    model_id=MODEL_ID,
    api_base="https://api.lambdalabs.com/v1",
    api_key=os.getenv("INFERENCE_API_KEY")
)

messages = [
    {
        "role": "system",
        "content": (
            "Please output a valid JSON object for a tool call with the following schema:\n"
            "{\"tool\": <tool name>, \"input\": {\"query\": <search query>}}\n\n"
            "For example, if calling DuckDuckGoSearchTool, output:\n"
            "{\"tool\": \"DuckDuckGoSearchTool\", \"input\": {\"query\": \"best music recommendations for a party\"}}\n\n"
            "Now, please output the JSON for the following query:\n"
            "\"best music recommendations for a party at Wayne Mansion.\""
        )
    }
]

response = model(messages)
json_str = response.content

try:
    tool_call = json.loads(json_str)
except json.JSONDecodeError as e:
    print("Failed to parse JSON output:", e)
    exit(1)

tool_name = tool_call.get("tool")
tool_input = tool_call.get("input", {})

print(f"Parsed tool call: tool: {tool_name}, input: {tool_input}")

if tool_name == "DuckDuckGoSearchTool":
    search_tool = DuckDuckGoSearchTool()
    
    query = tool_input.get("query")
    if query:
        search_result = search_tool(query)
        print("Search result from DuckDuckGoSearchTool:")
        print(search_result)
    else:
        print("No query provided in the tool call input.")
else:
    print(f"Unknown tool: {tool_name}")

What am I doing wrong when using ToolCallingAgent? Or is there a known issue with ToolCallingAgent?

1 Like
98

@sgugger How to use a local VLM like llava via ollama in LiteLLMModel?

I tried:

from smolagents import LiteLLMModel, CodeAgent
model = LiteLLMModel(
model_id=“ollama/llava:7b”, # Or try other Ollama-supported models
api_base=“http://127.0.0.1:11434”, # Default Ollama local server
num_ctx=8192,
)

Create a simple agent to test instrumentation

agent = CodeAgent(
tools=,
model=model
)
response = agent.run(“”"
Describe the image and generate a caption for it.
“”“,
images = [Image.open(”/data/hmaurya/hmaurya/arpah_vis_web/temp_img.jpeg")])

but I am getting this error:

python app.py
╭─────────────────────────────────────────── New run ───────────────────────────────────────────╮
│ │
│ Describe the image and generate a caption for it. │
│ │
╰─ LiteLLMModel - ollama/llava:7b ──────────────────────────────────────────────────────────────╯
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Step 1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Error in generating model output:
Cannot use images with flatten_messages_as_text=True
[Step 1: Duration 0.40 seconds]
Traceback (most recent call last):
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/smolagents/agents.py”, line 1225, in step
chat_message: ChatMessage = self.model(
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/openinference/instrumentation/smolagents/_wrappers.py”, line 287, in call
output_message = wrapped(*args, **kwargs)
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/smolagents/models.py”, line 897, in call
completion_kwargs = self._prepare_completion_kwargs(
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/smolagents/models.py”, line 279, in _prepare_completion_kwargs
messages = get_clean_message_list(
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/smolagents/models.py”, line 202, in get_clean_message_list
assert not flatten_messages_as_text, f"Cannot use images with {flatten_messages_as_text=}"
AssertionError: Cannot use images with flatten_messages_as_text=True

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/data/hmaurya/hmaurya/agentic_system/app.py”, line 60, in
response = agent.run(“”"
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/openinference/instrumentation/smolagents/_wrappers.py”, line 128, in call
agent_output = wrapped(*args, **kwargs)
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/smolagents/agents.py”, line 332, in run
return deque(self._run(task=self.task, max_steps=max_steps, images=images), maxlen=1)[0].final_answer
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/smolagents/agents.py”, line 356, in _run
raise e
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/smolagents/agents.py”, line 353, in _run
final_answer = self._execute_step(task, action_step)
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/smolagents/agents.py”, line 376, in _execute_step
final_answer = self.step(memory_step)
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/openinference/instrumentation/smolagents/_wrappers.py”, line 163, in call
result = wrapped(*args, **kwargs)
File “/data/hmaurya/hmaurya/anaconda3/envs/agentic_system/lib/python3.10/site-packages/smolagents/agents.py”, line 1241, in step
raise AgentGenerationError(f"Error in generating model output:\n{e}", self.logger) from e
smolagents.utils.AgentGenerationError: Error in generating model output:
Cannot use images with flatten_messages_as_text=True

1 Like
99

Ok, i just had to change flatten_messages_as_text = False in LiteLLMModel.

1 Like
100

Can I somehow avoid using HfApiModel? How can I use the local models? I did not find explanation in any place. It asks me to buy the paid track for HF. And I just want to try out the smolagent framework.

1 Like
101

Yea. Only on documents or on Hugging Face Discord.

102

Hi everyone,

very tiny issue at page Chat Templates - Hugging Face LLM Course

it seems the link " Here’s a guide on ChatML." is not the good one.

Maybe this is the one? openai-python/chatml.md at release-v0.28.0 · openai/openai-python · GitHub

thank you for this outsanding course.

1 Like