Zero-Shot Classification

According to my understanding the limitation of the zero-shot classifier is, zero-shot depends on the prior knowledge of the large data. Never-the-less, zero-shot performs poorly in case of dialog or conversation data (yulongchen/DialogSum · Datasets at Hugging Face) where the topic inference accuracy of the zero-shot is around 50%. What I have understood is that, if any word or its synonym word list is not in the prior knowledge of the zero-shot, zero-shot makes wrong prediction. Thanks for your time.

Is it possible modify the zero-shot classifier?