Thanks for your response! It definitely led me in the right direction. Essentially I replaced the line
!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit1/requirements-unit1.txt
with the following lines
!pip install stable-baselines3==2.0.0a5
!pip install swig
!pip install gymnasium
!pip install box2d-py
!pip install huggingface_sb3
which does not err and appears to install the same necessary components.