After googling some more I figured out what the problem is: After getting authorized by Meta you also need to get authorization by HF which can be requested by pressing the button here: meta-llama/Llama-2-7b-chat-hf · Hugging Face
So much drama with the llama!