Hi,
I’m trying to use calculated sentence embeddings by average pooling of chunks of a long sentence as input to train a model based on the AutoModelForSequenceClassification class. I used the “inputs_embeds” parameter to pass the embeddings to the model, but something strange is happening. Metrics do not change over time. These are the values that practically remain in the 30 epochs:
{‘eval_loss’: 0.48057085275650024,
‘eval_f1’: 0.3008849557522124,
‘eval_roc_auc’: 0.5,
‘eval_accuracy’: 0.0,
‘eval_precision’: 0.17708333333333334,
‘eval_recall’: 1.0,
‘eval_hammingLoss’: 0.8229166666666666,
‘eval_runtime’: 0.7474,
‘eval_samples_per_second’: 149,856,
‘eval_steps_per_second’: 149,856,
‘epoch’: 30.0}
Does anyone have any tips on how to train BERT using embeddings as input?