Avoiding the usage of HfApiModel and using local model - `smolagents`

I try to learn the basics of smolagents and I got the following big problem - please help!
I am getting the message that I have run out of the free tier for HfApiModel, and I need to buy the paid tier.
How can I use the local model to run with my CodeAgent in smolagents?

1 Like

There are several methods, but you can use TransformersModel instead of HfApiModel and it will work. When using large models, powerful GPUs are required, so please be careful. Well, I think SmolLM below will work with about 1GB of VRAM…

Also, Ollama is faster and uses less VRAM, but I think it will be a little difficult to set up (compared to TransformersModel…).

from smolagents import TransformersModel

model = TransformersModel(model_id="HuggingFaceTB/SmolLM-135M-Instruct")

I tried to run the following code:

from smolagents import TransformersModel
from smolagents import CodeAgent, DuckDuckGoSearchTool, HfApiModel

model = TransformersModel(model_id="HuggingFaceTB/SmolLM-135M-Instruct")
agent = CodeAgent(tools=[], model=model, additional_authorized_imports=['datetime'])
agent.run(
    """
    Alfred needs to prepare for the party. Here are the tasks:
    1. Prepare the drinks - 30 minutes
    2. Decorate the mansion - 60 minutes
    3. Set up the menu - 45 minutes
    4. Prepare the music and playlist - 45 minutes

    If we start right now, at what time will the party be ready?
    """
)
  1. I use my Mac M3’s CPU. And it does not produce responses. There is sertanly a bug here in code. Maybe the model is too big, and yes, I don’t have any GPU here at my place. Is the code fine?

  2. Another question, let’s say I will buy the PRO subscription. Can I use HuggingFace API also for LLamaIndex and LangGraph? I think, I saw that the API will work with LLamaIndex, so the bigger question - is the API suitable with LangGraph?
    Thank you in advance

1 Like
from smolagents import CodeAgent, TransformersModel

model = TransformersModel(model_id="HuggingFaceTB/SmolLM-135M-Instruct")
agent = CodeAgent(tools=[], model=model, additional_authorized_imports=['datetime'],  add_base_tools=True)
agent.run(
    """
    Alfred needs to prepare for the party. Here are the tasks:
    1. Prepare the drinks - 30 minutes
    2. Decorate the mansion - 60 minutes
    3. Set up the menu - 45 minutes
    4. Prepare the music and playlist - 45 minutes

    If we start right now, at what time will the party be ready?
    """
)

I think the code is correct, but it seems that the name of HfApiModel has changed (mainly the name) in the latest version of smolagents, so I made some minor corrections there.

Also, it’s a bit slow on the CPU… but it should still work. It’s just incredibly slow.
On a Mac, using MLX or Ollama might speed things up a bit…

Uff ok, CodeAgent is not good with “HuggingFaceTB/SmolLM-135M-Instruct” model. As far as I understand, the reason is that the context window is too small for the default huge prompt of the CodeAgent. So I tried to use another kind of agent - ToolCallingAgent:

from smolagents import DuckDuckGoSearchTool, ToolCallingAgent
from smolagents import MLXModel

model = MLXModel(model_id="HuggingFaceTB/SmolLM-135M-Instruct")
agent = ToolCallingAgent(tools=[DuckDuckGoSearchTool()], model=model)

agent.run("Search for the best music recommendations for a party at the Wayne's mansion.")

agent.run(
    """
    what color is the sky?
    """
)

Here, I’ve also inserted the model within the MLXModel class, which works well on my Mac, at least.

And this is the output:

There is a need to adapt, but I am not sure what exactly. The prompt does not help the model? idk. Does the code run in your environment by any chance?..

1 Like

My local environment is Python 3.9, so smolagents doesn’t work locally…:sob:
Anyway, I forgot about that bug, or rather, that feature…

from smolagents import DuckDuckGoSearchTool, ToolCallingAgent
from smolagents import MLXModel

model = MLXModel(model_id="HuggingFaceTB/SmolLM-135M-Instruct", max_new_tokens=4096)
agent = ToolCallingAgent(tools=[DuckDuckGoSearchTool()], model=model)

agent.run("Search for the best music recommendations for a party at the Wayne's mansion.")

agent.run(
    """
    what color is the sky?
    """
)