Using Bloom with detailed parameters?

Models
1

Hello,

Newbie here, so my apologies if this is a stupid question or if i post in the wrong section.

I’m trying to use the bloom model through inference api and it works well, but when i try to add some parameters (from the detailed parameters list in the text generation category), i get this error:
{‘error’: ‘Parameters are not accepted for this specific model’}

import requests

API_URL = "https://api-inference.huggingface.co/models/bigscience/bloom"
#API_URL = "https://api-inference.huggingface.co/models/gpt2"

headers = {"Authorization": "****"}

def query(payload):
    response = requests.post(API_URL, headers=headers, json=payload)
    return response.json()

prompt = "Hi i am an AI and "

output = query({
    "inputs": prompt,
    "parameters": {"max_new_tokens": 50, "return_full_text": False}
})

print(output)
#print(output[0]["generated_text"])

This code works well (and the parameters are taken into account) when tried on gpt2, but fails on Bloom.

Am i doing anything wrong ?

Thanks
Cedric

2

It worked for me.

import requests

TOKEN = "Bearer <INSERT YOUR TOKEN HERE>"

API_URL = "https://api-inference.huggingface.co/models/bigscience/bloom"
headers = {"Authorization": TOKEN}

def query(payload):
	response = requests.post(API_URL, headers=headers, json=payload)
	return response.json()

while True:

    text_input = input("Insert your input: ")
            
    output = query({
            "inputs": text_input,
            "parameters": {"max_new_tokens": 10,
                           "return_full_text": False}
    })

    print(output)
1 Like
3

@roschmid , when I try this, I receive {'error': "Authorization header is invalid, use 'Bearer API_TOKEN'"}

Maybe you meant headers = {"Authorization": f"Bearer {API_TOKEN}"}?

4

@RylanSchaeffer You’re probably typing wrong your API Token.

It has to go with the “Bearer” part.

E.g. TOKEN = “Bearer 4EgJlma91939” (this is a made up Token, btw).

5

I don’t think TOKEN = “Bearer 4EgJlma91939” is a token. Rather, you’ve preappended Bearer to the actual token (in your example, the actual token is 4EgJlma91939).

6

Hi @roschmid

Thanks for your answer.
I wanted to try your code and first relaunched my script to ensure the error was still occuring with my code before trying yours, but it didn’t: now my old code works too !
I guess they must have fixed something internally.
Anyway, thanks a lot for taking the time to answer me, i marked you answer as a solution, although, for anyone bumping here, the code from the initial post works too.

Rgds
Cedric

2 Likes
7

I got your point. Fixed it.

1 Like
8

Thanks for the posts. I’m trying to add some parameters to a cURL request. Somehow it seems the parameters I’m trying to add are getting mixed up into the input string. It could be some kind of syntax error but I can’t see where I’m doing it wrong.

This works:

curl https://api-inference.huggingface.co/models/bigscience/bloom \
-X POST \
-d '{"inputs": "Two plus two equals "}' \
-H "Authorization: Bearer TOKEN"

yielding the output

[{"generated_text":"Two plus two equals four.\nTwo plus two equals four.\nTwo plus two equals four.\nTwo plus two equals"}]

However, when adding parameters, it seems that this code results in the attempted parameters being mixes up into the input text:

curl https://api-inference.huggingface.co/models/bigscience/bloom \
-X POST \
-d '{"inputs": "Two plus two equals", "parameters": {"max_new_tokens": 50, "return_full_text": False} }' \
-H "Authorization: Bearer TOKEN"

Resulting in this output:

[{"generated_text":"{\"inputs\": \"Two plus two equals\", \"parameters\": {\"max_new_tokens\": 50, \"return_full_text\": False} }\n#     }\n#     return json.dumps(json_data)\n\n# def get_tokens"}]

Maybe I just need a delimiter somewhere or the like?

If someone can help me fix this I would be really appreciative. Thanks.

9

This way the request seems to work:

model_id="bigscience/bloom" 
payload="Two plus two equals"
params='{"max_new_tokens": 50, "return_full_text":false}'
url="https://api-inference.huggingface.co/models/"$model_id
data='{"inputs":"'$payload'","parameters":'$params'}'
post=$(curl -X POST $url -H "Content-Type: application/json" --data-raw "$data")
echo $post