I ran my custom fine-tuned Mistral 7b Instruct on hf inference and want to use openAI client, so that the API won’t return whole text. Below is settings of api (With TensorFlow it fails to run)
But now code gives me this error:
Traceback (most recent call last):
File "/Users/almatkairatov/Desktop/GENERAL/CreateML training/ML/huggingFace/main.py", line 9, in <module>
chat_completion = client.chat.completions.create(
File "/opt/miniconda3/envs/huggingFace/lib/python3.9/site-packages/openai/_utils/_utils.py", line 277, in wrapper
return func(*args, **kwargs)
File "/opt/miniconda3/envs/huggingFace/lib/python3.9/site-packages/openai/resources/chat/completions.py", line 590, in create
return self._post(
File "/opt/miniconda3/envs/huggingFace/lib/python3.9/site-packages/openai/_base_client.py", line 1240, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File "/opt/miniconda3/envs/huggingFace/lib/python3.9/site-packages/openai/_base_client.py", line 921, in request
return self._request(
File "/opt/miniconda3/envs/huggingFace/lib/python3.9/site-packages/openai/_base_client.py", line 1020, in _request
raise self._make_status_error_from_response(err.response) from None
openai.UnprocessableEntityError: Error code: 422 - {'error': 'Template error: template not found', 'error_type': 'template_error'}
This code I just take for testing, but as I said it is not working
import os
from openai import OpenAI
# init the client but point it to TGI
client = OpenAI(
base_url=os.path.join("https://url.endpoints.huggingface.cloud", "v1/"),
api_key="API_KEY",
)
chat_completion = client.chat.completions.create(
model="tgi",
messages=[
{"role": "user", "content": "Why is open-source software important?"}
],
stream=True
)
# iterate and print stream
for message in chat_completion:
print(message.choices[0].delta.content, end="")