Help Making a lookup table transformer

Hi, Im trying to make a Transformer model to effectivley act as a lookup table for a dataset. Would anyone here be able to point me in the direction of a guide that would allow me to make a transformer model of that kind. And furthermore, if I were to go and use hugging face for it would I be able to store the model on my computer and proceed to use it independently of hugging face.