Hello all,
Trying to connect with javascript to an inference endpoint. The endpoint set up and running.
To clarify, I am refering to Inference Endpoints (dedicated), not Serverless API. It seems both in the docs and this forum people refer to Serverless API as Inference ENdpoints (or maybe it is just me).
The issue I am having is that every piece of code on Hugging Face does not work, except for the OpenAI API example (and that’s not an option for current project).
The code given on the Inference Endpoint page, which oddly doesn’t use the HF javascript library, does not connect o the endpoint correctly (404 error, no matter what variation of endpoint I try).
In the docs, this page: has this code:
const inference = new HfInference(‘hf_…’) // your user token
const gpt2 = inference.endpoint('https://xyz.eu-west-1.aws.endpoints.huggingface.cloud/gpt2-endpoint')
const { generated_text } = await gpt2.textGeneration({ inputs: 'The answer to the universe is' })
Other than being odd ( ‘gpt2’ as a variable name and the example endpoint ends in ‘gpt2-endpoint’, which makes it less clear), it doesn’t work. It looks like someone’s copied this from the Serverless API example and not tested it.
Most of the errors are 404, which normally I’d assume was the endpoint itself. But I’ve tried all variations, and the same endpoint works fine if I use the OpenAI API example.
So right now, I’m paying for an endpoint (small model, so not a lot to be fair), and the only way to access it is via the OpenAI API.
I’m from the open source world, going back years. I appreciate I can rant, but I’d be much more useful if I helped fix it.
So, if anyone can give me an example of working javascript code that connects to an Inference Endpoint (chatbot is the focus), I’d love to solve my problem and also submit a much-needed update to the docs.