Agents Notebook: While is not supported?

🤗Transformers
1

Hello, I have a question about what tranformer agents can and can’t do. I was running the Notebook titled “Transformers can do Anything” and I modified the prompt about captioning the boat in the following way.

caption = agent.run(“Can you caption the boat_image? Use at least 35 words in the caption”, boat_image=boat)

The part that I added is the second sentence “Use at least 35 words in the caption.” The agent then generates the following.

==Code generated by the agent==
caption = image_captioner(boat_image)
while len(caption.split()) < 35:
    caption = image_captioner(boat_image)
print(f"The caption is {caption}")

==Result==
Evaluation of the code stopped at line 1 before the end because of the following error:
While is not supported.
/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1346: UserWarning: Using `max_length`'s default (20) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.  warnings.warn(
'Pokemon boats and pio'

Is this because the code is being run in a notebook? Or is it a general restriction on what code the agents can execute? Or is it just me missing something silly?

Thanks for any advice

2

hi @Awillia91,

It seems like the internal python interpreter inside Transformers Tools is not equipped to run loops, which makes sense, since you don’t want a LLM to accidentally generate an infinity loop

Ideally the Image Captioning tool implemented here could accept kargs such as max_new_tokens and the a new tool description could be

    description = (
        "This is a tool that generates a description of an image. It takes an input named `image` which should be the "
        "image to caption, and returns a text that contains the description in English,  and `max_new_tokens` to set the number of output tokens
    )

Maybe then, this prompt would work

caption = agent.run(“Can you caption the boat_image? Use at least 35 words”, boat_image=boat)

As you see this is a very experimental tool, and we’re open for PR/Issues ideas please Issues · huggingface/transformers · GitHub

You can also create your own image caption tool and solve that issue Custom Tools and Prompts

3

Yep, that makes a lot of sense, I’m quite familiar with self-writing code and the difficulty of loops with it. Thank you for your thoughtful reply. I was just trying to get to know the framework I’ll have to spend some time looking more deeply into that later! This seems like a great project where a lot of rapid development could take place, I would be excited to contribute where I can!

1 Like
4

Very cool! I’d be curious to learn about what you’re currently working on! :face_with_monocle:
Totally agree with you, Agents could be useful, given they are designed as simple and well scoped tools.

5

Hello! Thank you so much for the great open source work that you are doing at HF.

python_interpreter seems like it’s going to be a very important tool for the entire LLM field but I haven’t been able to find good open source implementation of it. So thank you again for this.

Currently it seems like the python_interpreter does not support accessing attributes. Is this because of safety reasons? I can imagine this being an issue if the state included entire libraries but not being able to access attributes also limits calling python built in method. The latter to me seems like it should be allowed. Please correct me if I’m wrong.

The issue that I ran into was for simple counting of a list. Following is a toy reproducible example.

import python_interpreter

example_dict = {'numbers': [1, 1, 2]}

python_interpreter.evaluate("""
one_count = example_dict['numbers'].count(1)
""",
  tools={},
  state={
      "example_dict": example_dict,
  })

Gives

Evaluation of the code stopped at line 0 before the end because of the following error:
It is not permitted to evaluate other functions than the provided tools (tried to execute <ast.Attribute object at 0x7fb7aa893520> of type <class ‘ast.Attribute’>.

I wrote a quick workaround that seems to work for me and I’ll share it here in case it’s useful for others.

modified evaluate_call function in python_interpreter.py

def evaluate_call(call, state, tools):
    if isinstance(call.func, ast.Attribute):
      obj = evaluate_ast(call.func.value, state, tools)
      method_name = call.func.attr
      func = getattr(obj, method_name)
    else:
      if not isinstance(call.func, ast.Name):
          raise InterpretorError(
              f"It is not permitted to evaluate other functions than the provided tools (tried to execute {call.func} of "
              f"type {type(call.func)}."
          )
      func_name = call.func.id

      if func_name not in tools:
          raise InterpretorError(
              f"It is not permitted to evaluate other functions than the provided tools (tried to execute {call.func.id})."
          )

      func = tools[func_name]

    # Todo deal with args
    args = [evaluate_ast(arg, state, tools) for arg in call.args]
    kwargs = {keyword.arg: evaluate_ast(keyword.value, state, tools) for keyword in call.keywords}
    return func(*args, **kwargs)

Which correctly outputs

2

Given that it’s related, I thought I would ask on this thread if there are any plans to add support for ast.Attribute for the python_interpreter. I can also open up another thread, this is my first time using the forum so please let me know if I shouldn’t be posting this here.

Thank you!

6

Hi, any idea about how to modify the ‘evaluate_ast’ function to support for loop in code?