Iam trying to run bloom-560m, GPT-J-6B model inference on a kubernetes cluster after connecting the dl1-large resource to it as well as the the habana container image “vault.habana.ai/gaudi-docker/1.10.0/ubuntu20.04/habanalabs/pytorch-installer-2.0.1:latest”.
- pip install optimum[habana]
- cd optimum-habana/examples/text-generation
- pip install -r requirements.txt
- python …/gaudi_spawn.py --use_deepspeed --world_size 2 run_generation.py
–prompt “Tell me a poem about stone and water”
I am running into this error:-
Tried the dmesg solutions from here
but they didn’t work.
What could be the possible reason for this?