Hey, I have a custom dataset. can you send a sample script to get the accuracy on such a dataset? I was going through examples and I couldn’t get a code that does that. Can someone send me a resource? my dataset is of the format- premise , hypothesis, label(0 or 1) and my model is deberta Thank…

Hey @NDugar if you’re using the Trainer my suggestion would be to run Trainer.predict(your_test_dataset) so you can get all the predictions. Then you should be able to feed those into the accuracy metric in a second step (or whatever metric you’re interested in). If you’re still having trouble, I s…

Sample evaluation script on custom dataset

Beginners

NDugar December 9, 2021, 1:56pm 4

In the sample code you wrote where do I mention the model and the dataset?

Topic		Replies	Views
Evaluation metrics 🤗Transformers	1	2011	November 16, 2020
How to evaluate models Beginners	0	2857	June 16, 2021
How to get accuracy of pre trained model in huggingface? Beginners	3	4261	September 20, 2022
Trainer.evaluate() vs trainer.predict() 🤗Transformers	6	37294	July 10, 2024
Trainer Question Answering evaluation metrics 🤗Transformers	4	3394	May 3, 2021

Sample evaluation script on custom dataset

Related topics