So this is what worked for me
from transformers import pipeline
pipe = pipeline(task="your-task-name", model="./my_local_disk", tokenizer=original_tokenizer)
where original_tokenizer is the Tokenizer you used while creating the training/test data