Have you tried out different learning rates? Prompt Tuning sometimes requires surprisingly high learning rates of for instance 0.3. This post is quite good in my opinion: Guiding Frozen Language Models with Learned Soft Prompts – Google AI Blog
Which dataset are you using?