For GPT-3, there’s the
logit_bias parameter. It allows you to control how likely or unlikely the model is to pick a particular token for the output.
Can I do something similar with transformers, particularly a T5 model?
I am trying to make the T5 translate human language into the syntax used by an API.
Human: Get all news stories from 3 days ago.
My problem is that, even after training with 9 million of examples, the model keep making up the slug part of the output.
Correct output: stories[‘D-3’]
Actual output: news stories[‘D-3’]
I think I could solve that problem if I could just effectively reduce the model’s output vocabulary to the tokens allowed by the API. In the above example, neither a space nor the word “news” would be allowed. So the model would have to pick the next best tokens, and would, I guess, come up with the correct
Now, I cannot limit the actual vocabulary, or train the model from scratch. Because I rely on the model’s vast vocabulary from its pretraining for understanding my human language inputs. I need the model to understand human language. It’s only its output that I want to restrict down to the simple syntax that my API supports.
So what envision doing is…
- Run all of my training data’s labels through the tokenizer, to build a set of tokens allowed in the output.
- Subtract this set of allowed tokens from the set of tokens in the model’s vocabulary, to get a set of tokens that should be suppressed in the output.
- Tell the model to apply a
logit_biasof, say, -10 to any of the tokens in the
Any ideas how to do this?