Hello Everyone,
I am finetuning a mistral 7b instruct model on a small custom dataset. The idea behind the task is that I am mapping three to four features together and the task is in form of question and answer task. Firstly, I divided the dataset into train, validation, and test set. Then I passed the train data, validation data, and other hyperparameter into the SFTTrainer. Then I used trainer.train() to finetune the model. The model is doing well on inferencing. However, I would like to get the accuracy of the model on the test set so that I can compare it with the accuracy of other model.
The shape of the train set before passing it into the SFTTrainer is (210, 6) (this include the input_ids and attention_mask)
The shape of the validation set before passing it into the SFFTrainer is (24, 4) this does not include input_ids and attention_mask.
Then I used trainer.predict(test_set).
I also got this shape: (26, 512, 32000) (26, 512) after doing print(result.predictions.shape, result.label_ids.shape).
How do I get the accuracy of this model on the test?