Whenever I use a ViT it breaks my code and doesn’t let me or wandb log. There is a long discussion here: [CLI]: wandb saving to local does not work when soft links to the current project are used · Issue #4409 · wandb/wandb · GitHub but it only happens when I use hugging face’s (HF’s) ViT. Why?
Error is super cryptic but only happens if I use ViTs:
sys.stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>
os.path.realpath(sys.stdout.name)='/home/miranda9/diversity-for-predictive-success-of-meta-learning/<stdout>'
--- Logging error ---
Traceback (most recent call last):
File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/logging/__init__.py", line 1087, in emit
self.flush()
File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/logging/__init__.py", line 1067, in flush
self.stream.flush()
OSError: [Errno 116] Stale file handle
Call stack:
File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/internal_util.py", line 50, in run
self._run()
File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/internal_util.py", line 101, in _run
self._process(record)
File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/internal.py", line 263, in _process
self._hm.handle(record)
File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/handler.py", line 130, in handle
handler(record)
File "/home/miranda9/miniconda3/envs/metalearning_gpu/lib/python3.9/site-packages/wandb/sdk/internal/handler.py", line 138, in handle_request
logger.debug(f"handle_request: {request_type}")
Message: 'handle_request: stop_status'
Arguments: ()