Unable to deploy fine tuned model

Intermediate
1

I fine tuned a model using auto train on hugging face. However I am unable to deploy this model as get the following error in deployment. Seems it is not getting the model type during deployment. Any help would be appreciated

Mar 06, 10:49:50 ERROR
fields: {“message”:“Error when initializing model\nTraceback (most recent call last):\n File "/usr/src/.venv/bin/text-generation-server", line 10, in \n sys.exit(app())\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/main.py", line 323, in call\n return get_command(self)(*args, **kwargs)\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 1161, in call\n return self.main(*args, **kwargs)\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/core.py", line 743, in main\n return _main(\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/core.py", line 198, in _main\n rv = self.invoke(ctx)\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 1443, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 788, in invoke\n return __callback(*args, **kwargs)\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/main.py", line 698, in wrapper\n return callback(**use_params)\n File "/usr/src/server/text_generation_server/cli.py", line 119, in serve\n server.serve(\n File "/usr/src/server/text_generation_server/server.py", line 315, in serve\n asyncio.run(\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 190, in run\n return runner.run(main)\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 118, in run\n return self._loop.run_until_complete(task)\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete\n self.run_forever()\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 608, in run_forever\n self._run_once()\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 1936, in _run_once\n handle._run()\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/events.py", line 84, in _run\n self._context.run(self._callback, *self._args)\n> File "/usr/src/server/text_generation_server/server.py", line 268, in serve_inner\n model = get_model_with_lora_adapters(\n File "/usr/src/server/text_generation_server/models/init.py", line 1592, in get_model_with_lora_adapters\n model = get_model(\n File "/usr/src/server/text_generation_server/models/init.py", line 607, in get_model\n raise RuntimeError(\nRuntimeError: Could not determine model type for /repository revision None”}
target: “text_generation_launcher”
Mar 06, 10:49:50 ERROR
fields: {“message”:“Error when initializing model\nTraceback (most recent call last):\n File "/usr/src/.venv/bin/text-generation-server", line 10, in \n sys.exit(app())\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/main.py", line 323, in call\n return get_command(self)(*args, **kwargs)\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 1161, in call\n return self.main(*args, **kwargs)\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/core.py", line 743, in main\n return _main(\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/core.py", line 198, in _main\n rv = self.invoke(ctx)\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 1443, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 788, in invoke\n return __callback(*args, **kwargs)\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/main.py", line 698, in wrapper\n return callback(**use_params)\n File "/usr/src/server/text_generation_server/cli.py", line 119, in serve\n server.serve(\n File "/usr/src/server/text_generation_server/server.py", line 315, in serve\n asyncio.run(\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 190, in run\n return runner.run(main)\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 118, in run\n return self._loop.run_until_complete(task)\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete\n self.run_forever()\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 608, in run_forever\n self._run_once()\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 1936, in _run_once\n handle._run()\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/events.py", line 84, in _run\n self._context.run(self._callback, *self._args)\n> File "/usr/src/server/text_generation_server/server.py", line 268, in serve_inner\n model = get_model_with_lora_adapters(\n File "/usr/src/server/text_generation_server/models/init.py", line 1592, in get_model_with_lora_adapters\n model = get_model(\n File "/usr/src/server/text_generation_server/models/init.py", line 607, in get_model\n raise RuntimeError(\nRuntimeError: Could not determine model type for /repository revision None”}
target: “text_generation_launcher”
Mar 06, 10:49:50 ERROR
fields: {“message”:“Error when initializing model\nTraceback (most recent call last):\n File "/usr/src/.venv/bin/text-generation-server", line 10, in \n sys.exit(app())\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/main.py", line 323, in call\n return get_command(self)(*args, **kwargs)\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 1161, in call\n return self.main(*args, **kwargs)\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/core.py", line 743, in main\n return _main(\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/core.py", line 198, in _main\n rv = self.invoke(ctx)\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 1443, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n File "/usr/src/.venv/lib/python3.11/site-packages/click/core.py", line 788, in invoke\n return __callback(*args, **kwargs)\n File "/usr/src/.venv/lib/python3.11/site-packages/typer/main.py", line 698, in wrapper\n return callback(**use_params)\n File "/usr/src/server/text_generation_server/cli.py", line 119, in serve\n server.serve(\n File "/usr/src/server/text_generation_server/server.py", line 315, in serve\n asyncio.run(\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 190, in run\n return runner.run(main)\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 118, in run\n return self._loop.run_until_complete(task)\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete\n self.run_forever()\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 608, in run_forever\n self._run_once()\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 1936, in _run_once\n handle._run()\n File "/root/.local/share/uv/python/cpython-3.11.11-linux-x86_64-gnu/lib/python3.11/asyncio/events.py", line 84, in _run\n self._context.run(self._callback, *self._args)\n> File "/usr/src/server/text_generation_server/server.py", line 268, in serve_inner\n model = get_model_with_lora_adapters(\n File "/usr/src/server/text_generation_server/models/init.py", line 1592, in get_model_with_lora_adapters\n model = get_model(\n File "/usr/src/server/text_generation_server/models/init.py", line 607, in get_model\n raise RuntimeError(\nRuntimeError: Could not determine model type for /repository revision None”}
target: “text_generation_launcher”

1 Like
2

I think it’s probably related to TGI. It seems that there are many cases where an error occurs due to a different version. If it’s a different library, searching for information on that library will help you troubleshoot.

3

Could there be some problem in the model output after training? I am unable to use this model on ollama too (after gguf conversion and all steps). I get an error that manifest file is not found on ollama, while ollama list shows the model. Or is there some parameter that I should consider changing in autotrain?

1 Like
4

The error in Ollama is probably a different issue. If the cause is the same, it’s probably something like the base model settings in README.md (actually a configuration file…) being wrong, or not supporting quantized files, or something like that…

5

Hmm…

6

Hi @vinaypri ! It looks like there are two issues with your Inference Endpoint.

There is no specified lora adapter env variables. We have documentation about lora adapters and TGI here: Text-generation-launcher arguments. You can add the env variable in your Endpoint under the ‘Environment’ settings.

There’s also an issue in the base model configuration. We also recommend taking a look at our documentation related to creating models and config files at Create a custom model, as one of the missing attributes is model_type.

Hope this helps!

1 Like