InformationRetrievalEvaluator with training semantic search model

Intermediate
1

I am trying to train a sentence transformer model with custom data to use it for semantic search application. My data includes a chunk of text and corresponding search string queries. I am using the InformationRetrievalEvaluator.

Below is my training script:

# Enable logging
logging.basicConfig(format='%(asctime)s - %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S',
                    level=logging.INFO,
                    handlers=[LoggingHandler()])


description = [txt for txt in train_df.llm_image_description]
query = [q for q in train_df.llm_search_strs_with]

train_examples = Dataset.from_dict(
    {
    "description": description,
    "query": query
    }
)

corpus = load_from_pickle("data/IR_eval_data/corpus.pkl")
queries = load_from_pickle("data/IR_eval_data/queries.pkl")
qrels = load_from_pickle("data/IR_eval_data/qrels.pkl")

evaluator = InformationRetrievalEvaluator(queries=queries, corpus=corpus, relevant_docs=qrels,
                                          name='eval',show_progress_bar=True, write_csv=True,precision_recall_at_k=[1,2,3,4,5],mrr_at_k=[2,3,4,5],accuracy_at_k=[1,2,3,4,5],)

# Initialize a pre-trained model
# model = SentenceTransformer('distilbert-base-nli-mean-tokens') 
model = SentenceTransformer("multi-qa-mpnet-base-cos-v1") #https://www.sbert.net/docs/sentence_transformer/pretrained_models.html
                                                                                                                                                
# Define the loss function
train_loss = losses.MultipleNegativesRankingLoss(model=model)

# Define training arguments
training_args = SentenceTransformerTrainingArguments(
    eval_strategy='epoch',
    logging_strategy='epoch',
    save_strategy='epoch',
    save_total_limit=2,
    seed=10,
    load_best_model_at_end=True,
    metric_for_best_model='eval_cosine_recall@2',
    num_train_epochs=num_epochs,
    per_device_train_batch_size=train_batch_size,
    output_dir=model_save_path,
)
#This is the latest function
trainer = SentenceTransformerTrainer(
    model=model,
    args=training_args,
    loss=train_loss,
    train_dataset=train_examples,
    evaluator=evaluator,
)

train_result = trainer.train()

Below output is being printed at the end of each epoch:

Screenshot 2024-09-05 at 5.35.53 PM
Screenshot 2024-09-05 at 5.35.53 PM1679×156 17.3 KB

I have below 2 questions:

  1. Even when the write_csv=True, at the end of training I am not able to find any evaluation csv file in my output file path. I am not sure why it is not writing it out. I am also not able to retrieve the above result from the trainer or evaluator object methods. Can someone please tell me how can I : either get the csv to be written out to output path or get the dictionary/dataframe from trainer or evaluator objects?
  2. My result shows validation loss column with no information. How can I enable logging for validation loss? it will be important to track it.

I am using below versions:

transformers version: 4.44.2
sentence_transformers version: 3.0.1
PyTorch version: 2.2.0