If you just want to limit it, it seems like you can do it with just the label2id and id2label settings, but if you have any questions, I recommend asking them on HF Discord.
The ask-for-help channel and general channel are good, but there is also a channel dedicated to NLP, so you might be able to gain some knowledge there.
