Autotrain nvidia dgx cloud not working

đŸ¤—AutoTrain
1

I have been a trusty enterprise subscription client for huggingface
in the past months since I learn about using nvidia dgx cloud to train with H100 and A100 gpus I have trained multiple models for my company.
however, during the first week of july I noticed it is not working anymore

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/app/env/lib/python3.10/site-packages/autotrain/backends/nvcf.py", line 35, in _conf_nvcf
    response.raise_for_status()
  File "/app/env/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/integrations/dgx/v1/invoke/848348f8-a4e2-4242-bce9-6baa1bd70a66

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/env/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/app/env/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in __call__
    return await self.app(scope, receive, send)
  File "/app/env/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/app/env/lib/python3.10/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/env/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/app/env/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/app/env/lib/python3.10/site-packages/starlette/middleware/sessions.py", line 85, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/app/env/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/app/env/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/app/env/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/app/env/lib/python3.10/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/app/env/lib/python3.10/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/app/env/lib/python3.10/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/app/env/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/app/env/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/app/env/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/app/env/lib/python3.10/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
  File "/app/env/lib/python3.10/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
  File "/app/env/lib/python3.10/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
  File "/app/env/lib/python3.10/site-packages/autotrain/app/ui_routes.py", line 642, in handle_form
    job_id = project.create()
  File "/app/env/lib/python3.10/site-packages/autotrain/project.py", line 78, in create
    return runner.create()
  File "/app/env/lib/python3.10/site-packages/autotrain/backends/nvcf.py", line 162, in create
    nvcf_fn_req = self._conf_nvcf(
  File "/app/env/lib/python3.10/site-packages/autotrain/backends/nvcf.py", line 59, in _conf_nvcf
    raise Exception(f"HTTP Error {http_err.response.status_code}: {http_err}. Details: {error_message}")
Exception: HTTP Error 404: 404 Client Error: Not Found for url: https://huggingface.co/api/integrations/dgx/v1/invoke/848348f8-a4e2-4242-bce9-6baa1bd70a66. Details: No additional error information.

@abhishek
please help me! :hugs: