How to calculate the accuracy of the testing dataset when we build MLM model using scrach?
1 Like
have you got the answers?
Hi
Yes, I found a way but not sure whether it’s the best way.
What I’m doing is for each sentence I’m masking a random word in the sentence and ask the model to predict it. If the actual word is in the predicted list I’m increasing the tp(positive) value or else the tn(negative) value. finally I’m calculating the accuracy (tp/tp+tn).
and I’m calculating the perplexity value using the following code
Hope my answer is clear to you…
Hey, thank you for your response,
i am also calculating perplexity using the same code and i am using perplexity as a metric rather than accuracy.
This is because i don’t thing accuracy would be a better choice(as a metric) for mlm because there can be may words which can be used for a given mask. (feel free the correct) , instead i am calculating perplexity in the compute metric function and printing it along with training_loss and validation loss along in the logs.
Thanks again!!
1 Like
Also how much minimum data is required to fine tune roberta any idea?
sorry. I don’t have an idea about it…
do you know a way to build a n-gram model for word prediction and calculate the perplexity value of it?