I’m working on a KenLM model (I’m training one) and I would like to upload it into its own repository on HF. But I cannot find how to code the accompanying Python class for it.
The closest thing I could find is this repo:
But it seems to me even they couldn’t find a way to create the repo. If you look at the example code they have in their repo’s card:
from model import KenlmModel
# Load model trained on English wikipedia
model = KenlmModel.from_pretrained("wikipedia", "en")
# Get perplexity
model.get_perplexity("I am very perplexed")
# 341.3 (low perplexity, since sentence style is formal and with no grammar mistakes)
model.get_perplexity("im hella trippin")
# 46793.5 (high perplexity, since the sentence is colloquial and contains grammar mistakes)
They are loading the KenlmModel class from the model module. But this module only exists in their repo, meaning that you have to clone the repo before you can use their model (no support for pip install). Of course, this is an awful experience.
Is there a way to create a repo for a custom model like KenLM on HF?