AutoTrain models performance (mainly F1 score)

🤗AutoTrain
1

Where can I see the performance of my models (mainly the F1 score), as shown on the homepage of AutoNLP/AutoTrain?

I have trained 5 models using my own data but am only able to see the Loss of each one under the metrics tab. Seeing as I’m researching ways to generate synthetic data for question answering models, I would like to allocate my time towards the generation of the context, question, answer triplets and not the evaluation. AutoNLP/AutoTrain seems like a good solution for this, however, not if I’m unable to get F1 or EM scores like in the screenshot below.

Screenshot from 2022-04-12 15-09-13
Screenshot from 2022-04-12 15-09-13979×595 102 KB

About my research (for context)
I have generated QA data in (1) English based on Wiki texts, (2) Dutch based on Wiki texts and (3) Dutch based on technical/domain specific documents. I have used 8 different methods of generating the data, resulting in 8 x 3 = 24 datasets that I would like to evaluate using a QA model. If I were to use AutoNLP/AutoTrain for this, I would like to be sure that the results are something I can use.

2

pinging @abhishek @sbrandeis on this!

3

Unfortunately, currently, other metrics are not available for QA tasks. I have made an internal issue to keep track of this and will update you when other metrics are available. You can, for now, calculate other metrics by generating predictions on your own. In the meantime, i would also like to know if there are any specific metrics would you want to see for question answering?

4

Please provide any update on this issue.

5

If you are asking about the QA metrics, they are available in the UI now.

6

Hi abhishek, i still see only loss metrics for models finetuned for qa tasks, not seeing f1 score or em.

7

when was the model trained? could you please provide the project name and username?

8

Hi abhishek, i just now trained a new model. Project name: testing-finetune-autotrain
username: gokul427
model used: bert-base-uncased
data: lmqg/qa_squadshifts
please help.