Some jwt error after `make start` the newest dataset-server

My system is Windows10 and use the newest version of Docker Desktop, which the wsl engine was go on to execute docker together with WSL2. My WSL2 system is Ubuntu-22.04, and I cloned the newest version of datasets-server in it. Then I ran make start to build the server and successfully execute and can be shown in Docker Desktop dashboard.
When i saw the log about them, I found the worker docker: api, rows and search docker occred the same error as follows, which make the proxy docker receive 502 when i requests them:

2023-10-27 15:11:50 INFO:     Stopping parent process [1]
2023-10-27 15:11:52 INFO:     Uvicorn running on http://0.0.0.0:8180 (Press CTRL+C to quit)
2023-10-27 15:11:52 INFO:     Started parent process [1]
2023-10-27 15:11:53 Process SpawnProcess-2:
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
2023-10-27 15:11:53     yield
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_backends/sync.py", line 174, in start_tls
2023-10-27 15:11:53     raise exc
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_backends/sync.py", line 169, in start_tls
2023-10-27 15:11:53     sock = ssl_context.wrap_socket(
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket
2023-10-27 15:11:53     return self.sslsocket_class._create(
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/ssl.py", line 1041, in _create
2023-10-27 15:11:53     self.do_handshake()
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/ssl.py", line 1310, in do_handshake
2023-10-27 15:11:53     self._sslobj.do_handshake()
2023-10-27 15:11:53 socket.timeout: _ssl.c:1112: The handshake operation timed out
2023-10-27 15:11:53 
2023-10-27 15:11:53 The above exception was the direct cause of the following exception:
2023-10-27 15:11:53 
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
2023-10-27 15:11:53     yield
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 228, in handle_request
2023-10-27 15:11:53     resp = self._pool.handle_request(req)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 262, in handle_request
2023-10-27 15:11:53     raise exc
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 245, in handle_request
2023-10-27 15:11:53     response = connection.handle_request(request)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
2023-10-27 15:11:53     raise exc
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
2023-10-27 15:11:53     stream = self._connect(request)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 156, in _connect
2023-10-27 15:11:53     stream = stream.start_tls(**kwargs)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_backends/sync.py", line 174, in start_tls
2023-10-27 15:11:53     raise exc
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
2023-10-27 15:11:53     self.gen.throw(typ, value, traceback)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
2023-10-27 15:11:53     raise to_exc(exc) from exc
2023-10-27 15:11:53 httpcore.ConnectTimeout: _ssl.c:1112: The handshake operation timed out
2023-10-27 15:11:53 
2023-10-27 15:11:53 The above exception was the direct cause of the following exception:
2023-10-27 15:11:53 
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/src/libs/libapi/src/libapi/jwt_token.py", line 177, in fetch_jwt_public_key_json
2023-10-27 15:11:53     response = httpx.get(url, timeout=hf_timeout_seconds)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_api.py", line 189, in get
2023-10-27 15:11:53     return request(
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_api.py", line 100, in request
2023-10-27 15:11:53     return client.request(
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 814, in request
2023-10-27 15:11:53     return self.send(request, auth=auth, follow_redirects=follow_redirects)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 901, in send
2023-10-27 15:11:53     response = self._send_handling_auth(
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 929, in _send_handling_auth
2023-10-27 15:11:53     response = self._send_handling_redirects(
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
2023-10-27 15:11:53     response = self._send_single_request(request)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 1002, in _send_single_request
2023-10-27 15:11:53     response = transport.handle_request(request)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 228, in handle_request
2023-10-27 15:11:53     resp = self._pool.handle_request(req)
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
2023-10-27 15:11:53     self.gen.throw(typ, value, traceback)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
2023-10-27 15:11:53     raise mapped_exc(message) from exc
2023-10-27 15:11:53 httpx.ConnectTimeout: _ssl.c:1112: The handshake operation timed out
2023-10-27 15:11:53 
2023-10-27 15:11:53 The above exception was the direct cause of the following exception:
2023-10-27 15:11:53 
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/src/libs/libapi/src/libapi/jwt_token.py", line 220, in get_jwt_public_keys
2023-10-27 15:11:53     payload = fetch_jwt_public_key_json(
2023-10-27 15:11:53   File "/src/libs/libapi/src/libapi/jwt_token.py", line 181, in fetch_jwt_public_key_json
2023-10-27 15:11:53     raise RuntimeError(f"Failed to fetch the JWT public key from {url}. ") from err
2023-10-27 15:11:53 RuntimeError: Failed to fetch the JWT public key from https://huggingface.co/api/keys/jwt. 
2023-10-27 15:11:53 
2023-10-27 15:11:53 The above exception was the direct cause of the following exception:
2023-10-27 15:11:53 
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-27 15:11:53     self.run()
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-27 15:11:53     self._target(*self._args, **self._kwargs)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
2023-10-27 15:11:53     target(sockets=sockets)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/uvicorn/server.py", line 60, in run
2023-10-27 15:11:53     return asyncio.run(self.serve(sockets=sockets))
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
2023-10-27 15:11:53     return loop.run_until_complete(main)
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
2023-10-27 15:11:53     return future.result()
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/uvicorn/server.py", line 67, in serve
2023-10-27 15:11:53     config.load()
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/uvicorn/config.py", line 483, in load
2023-10-27 15:11:53     self.loaded_app = self.loaded_app()
2023-10-27 15:11:53   File "/src/services/api/src/api/app.py", line 28, in create_app
2023-10-27 15:11:53     return create_app_with_config(app_config=app_config, endpoint_config=endpoint_config)
2023-10-27 15:11:53   File "/src/services/api/src/api/app.py", line 37, in create_app_with_config
2023-10-27 15:11:53     hf_jwt_public_keys = get_jwt_public_keys(
2023-10-27 15:11:53   File "/src/libs/libapi/src/libapi/jwt_token.py", line 232, in get_jwt_public_keys
2023-10-27 15:11:53     raise JWTKeysError("Failed to create the JWT public keys.") from err
2023-10-27 15:11:53 libapi.exceptions.JWTKeysError: Failed to create the JWT public keys.
2023-10-27 15:11:53 Process SpawnProcess-1:
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
2023-10-27 15:11:53     yield
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_backends/sync.py", line 174, in start_tls
2023-10-27 15:11:53     raise exc
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_backends/sync.py", line 169, in start_tls
2023-10-27 15:11:53     sock = ssl_context.wrap_socket(
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/ssl.py", line 501, in wrap_socket
2023-10-27 15:11:53     return self.sslsocket_class._create(
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/ssl.py", line 1041, in _create
2023-10-27 15:11:53     self.do_handshake()
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/ssl.py", line 1310, in do_handshake
2023-10-27 15:11:53     self._sslobj.do_handshake()
2023-10-27 15:11:53 socket.timeout: _ssl.c:1112: The handshake operation timed out
2023-10-27 15:11:53 
2023-10-27 15:11:53 The above exception was the direct cause of the following exception:
2023-10-27 15:11:53 
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
2023-10-27 15:11:53     yield
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 228, in handle_request
2023-10-27 15:11:53     resp = self._pool.handle_request(req)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 262, in handle_request
2023-10-27 15:11:53     raise exc
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 245, in handle_request
2023-10-27 15:11:53     response = connection.handle_request(request)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
2023-10-27 15:11:53     raise exc
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
2023-10-27 15:11:53     stream = self._connect(request)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 156, in _connect
2023-10-27 15:11:53     stream = stream.start_tls(**kwargs)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_backends/sync.py", line 174, in start_tls
2023-10-27 15:11:53     raise exc
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
2023-10-27 15:11:53     self.gen.throw(typ, value, traceback)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
2023-10-27 15:11:53     raise to_exc(exc) from exc
2023-10-27 15:11:53 httpcore.ConnectTimeout: _ssl.c:1112: The handshake operation timed out
2023-10-27 15:11:53 
2023-10-27 15:11:53 The above exception was the direct cause of the following exception:
2023-10-27 15:11:53 
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/src/libs/libapi/src/libapi/jwt_token.py", line 177, in fetch_jwt_public_key_json
2023-10-27 15:11:53     response = httpx.get(url, timeout=hf_timeout_seconds)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_api.py", line 189, in get
2023-10-27 15:11:53     return request(
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_api.py", line 100, in request
2023-10-27 15:11:53     return client.request(
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 814, in request
2023-10-27 15:11:53     return self.send(request, auth=auth, follow_redirects=follow_redirects)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 901, in send
2023-10-27 15:11:53     response = self._send_handling_auth(
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 929, in _send_handling_auth
2023-10-27 15:11:53     response = self._send_handling_redirects(
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
2023-10-27 15:11:53     response = self._send_single_request(request)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_client.py", line 1002, in _send_single_request
2023-10-27 15:11:53     response = transport.handle_request(request)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 228, in handle_request
2023-10-27 15:11:53     resp = self._pool.handle_request(req)
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
2023-10-27 15:11:53     self.gen.throw(typ, value, traceback)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
2023-10-27 15:11:53     raise mapped_exc(message) from exc
2023-10-27 15:11:53 httpx.ConnectTimeout: _ssl.c:1112: The handshake operation timed out
2023-10-27 15:11:53 
2023-10-27 15:11:53 The above exception was the direct cause of the following exception:
2023-10-27 15:11:53 
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/src/libs/libapi/src/libapi/jwt_token.py", line 220, in get_jwt_public_keys
2023-10-27 15:11:53     payload = fetch_jwt_public_key_json(
2023-10-27 15:11:53   File "/src/libs/libapi/src/libapi/jwt_token.py", line 181, in fetch_jwt_public_key_json
2023-10-27 15:11:53     raise RuntimeError(f"Failed to fetch the JWT public key from {url}. ") from err
2023-10-27 15:11:53 RuntimeError: Failed to fetch the JWT public key from https://huggingface.co/api/keys/jwt. 
2023-10-27 15:11:53 
2023-10-27 15:11:53 The above exception was the direct cause of the following exception:
2023-10-27 15:11:53 
2023-10-27 15:11:53 Traceback (most recent call last):
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2023-10-27 15:11:53     self.run()
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/multiprocessing/process.py", line 108, in run
2023-10-27 15:11:53     self._target(*self._args, **self._kwargs)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/uvicorn/_subprocess.py", line 76, in subprocess_started
2023-10-27 15:11:53     target(sockets=sockets)
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/uvicorn/server.py", line 60, in run
2023-10-27 15:11:53     return asyncio.run(self.serve(sockets=sockets))
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run
2023-10-27 15:11:53     return loop.run_until_complete(main)
2023-10-27 15:11:53   File "/usr/local/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
2023-10-27 15:11:53     return future.result()
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/uvicorn/server.py", line 67, in serve
2023-10-27 15:11:53     config.load()
2023-10-27 15:11:53   File "/src/services/api/.venv/lib/python3.9/site-packages/uvicorn/config.py", line 483, in load
2023-10-27 15:11:53     self.loaded_app = self.loaded_app()
2023-10-27 15:11:53   File "/src/services/api/src/api/app.py", line 28, in create_app
2023-10-27 15:11:53     return create_app_with_config(app_config=app_config, endpoint_config=endpoint_config)
2023-10-27 15:11:53   File "/src/services/api/src/api/app.py", line 37, in create_app_with_config
2023-10-27 15:11:53     hf_jwt_public_keys = get_jwt_public_keys(
2023-10-27 15:11:53   File "/src/libs/libapi/src/libapi/jwt_token.py", line 232, in get_jwt_public_keys
2023-10-27 15:11:53     raise JWTKeysError("Failed to create the JWT public keys.") from err
2023-10-27 15:11:53 libapi.exceptions.JWTKeysError: Failed to create the JWT public keys.

At first I thought the main error is network and i use ping huggingface.co and curl https://huggingface.co/api/keys/jwt in these dockers, but I had 200 request later, which means the network in dockers are not the main reason cause the prolems.
Can anyone give a solutions or mothod to solve this bug, thanks !

Interesting! Can you open an issue on Issues · huggingface/datasets-server · GitHub?

We have to find out why the app cannot get the JWT keys. And anyway, the JWT key is only needed for our prod datasets-server since it’s used to authenticate the webhooks received by the HF Hub, so we could make it optional instead of raising an exception.

OK, I just saw Some jwt error after `make start` the newest dataset-server · Issue #2019 · huggingface/datasets-server · GitHub. Thanks!

1 Like