Can't deploy my custom endpoint with my Concrete ML package

Beginners
1

Hello everyone at HF!

I am trying to use Custom Inference Handler, to be able to use our Concrete ML (GitHub - zama-ai/concrete-ml: Concrete ML: Privacy Preserving ML framework built on top of Concrete, with bindings to traditional ML frameworks.) models directly in HF.

I have followed the Create custom Inference Handler tutorial. I have created zama-fhe/cml_logreg_test/ (it’s a private repo). In their, I have a requirements.txt file:

$ cat requirements.txt 

concrete-ml
numpy

There, I have also my handler.py file:

$ head handler.py 

from typing import Dict, List, Any
import numpy as np
from concrete.ml.deployment import FHEModelServer


class EndpointHandler:
    def __init__(self, path):

        # For server
        self.fhemodel_server = FHEModelServer(path)

I am able to make it work locally. Now, I try to deploy, a bit like at the end of your tuto. Now, my issue is that, when I try to create the endpoint, it seems it can’t find concrete package, while this latter is in the requirements.txt file. I am surprised because in your tuto, you use an holidays package that you also have in your requirements.txt so I would say this kind of setup is working.

The error I see in my console is like:

96w4z 2023-12-29T16:54:49.633Z   File "/app/huggingface_inference_toolkit/utils.py", line 188, in check_and_register_custom_pipeline_from_directory
96w4z 2023-12-29T16:54:49.633Z     custom_pipeline = check_and_register_custom_pipeline_from_directory(model_dir)
96w4z 2023-12-29T16:54:49.633Z   File "/repository/handler.py", line 3, in <module>
96w4z 2023-12-29T16:54:49.633Z     inference_handler = get_inference_handler_either_custom_or_default_handler(HF_MODEL_DIR, task=HF_TASK)
96w4z 2023-12-29T16:54:49.633Z   File "/app/webservice_starlette.py", line 57, in some_startup_task
96w4z 2023-12-29T16:54:49.633Z     from concrete.ml.deployment import FHEModelServer
96w4z 2023-12-29T16:54:49.633Z   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
96w4z 2023-12-29T16:54:49.633Z ModuleNotFoundError: No module named 'concrete'
96w4z 2023-12-29T16:54:49.633Z   File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
96w4z 2023-12-29T16:54:49.633Z     spec.loader.exec_module(handler)
96w4z 2023-12-29T16:54:49.633Z   File "/app/huggingface_inference_toolkit/handler.py", line 41, in get_inference_handler_either_custom_or_default_handler
96w4z 2023-12-29T16:54:49.633Z     await handler()
96w4z 2023-12-29T16:54:49.633Z   File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 682, in startup
96w4z 2023-12-29T16:54:52.165Z     from concrete.ml.deployment import FHEModelServer
96w4z 2023-12-29T16:54:52.165Z   File "<frozen importlib._bootstrap_external>", line 850, in exec_module
96w4z 2023-12-29T16:54:52.165Z 2023-12-29 16:54:52,165 | INFO | Found custom pipeline at /repository/handler.py
96w4z 2023-12-29T16:54:52.165Z 
96w4z 2023-12-29T16:54:52.165Z     await handler()
96w4z 2023-12-29T16:54:52.165Z     await self._router.startup()
96w4z 2023-12-29T16:54:52.165Z   File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 705, in lifespan
96w4z 2023-12-29T16:54:52.165Z Application startup failed. Exiting.
96w4z 2023-12-29T16:54:52.165Z ModuleNotFoundError: No module named 'concrete'

where the important part is ModuleNotFoundError: No module named ‘concrete’.

Could you help me a bit, please? I can video chat with some of you if needed, to show it to you. Thanks a lot for what you’re doing, HF is awesome, and it’s one of the reasons I want to deploy here!

2

Could we see the log of the requirements.txt installation?

3

OK, we’ve found the solution. It was about the fact that the revision for the endpoint was fixed, ie not the last revision. You have to press “get last revision”. It’s maybe not natural to beginners.

Anyway, sorry for the noise, keep the good work guys.

4

However, I would say that the “get latest revision” is not very reliable. For some reason:

  • I resume
  • I click on “get latest revision”
  • I see the new revision
  • I click on resume
  • I go to “logs”
  • I come back to “settings”
    and then, when I am back here, the latest revision has been replaced by an old one.

Is that that we have to click on “Update endpoints”? I feel the user might not see that, it’s not obvious, why not directly using the new SHA1?