Custom inference endpoint with multiple models

Inference Endpoints on the Hub
1

I would like to deploy an inference endpoint based on a combination of multiple models.
Here is the repository structure

/handler.py
/requirements.txt
/model1.pt
/model2.pt

What will be the value of path here? Adding a few prints in the handler tells me that it is set to /repository/.
However, it also looks like /repository/ does not contain model1.pt and model2.pt.
Is this on purpose? How can I update the code below to access model1.pt and model2.pt?

class EndpointHandler:
    def __init__(self, path=""):
        self.model1 = torch.load('model1.pt')
        self.model2 = torch.load('model2.pt')

    def __call__(self, data):
        pass
2

the torch load should be `os.path.join(path,“model1.pt”). The path var provided in the init points to where your files are.

3

Thanks.

The problem was that, for some reason, not all files are pulled from the model repository.

For instance, files with .ckpt extension (it was the case for me) are not available in path directory. I had to change the extension of the model files for them to be downloaded.

This behavior is probably documented somewhere but I could not find it.

The problem is now solved. os.path.join(path, "model.pt") does the trick.

4

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.