I have stored the value of the inference stage of the pipeline in json. Can you help me to display it in the metrics of the trials and experiments? The metrics for the training is visible though. I have also tried with property_files was no success.
Hello @Sanjita,
could you please elaborate more on what you want to do and are currently doing?
To track metrics with sagemaker-experiments you have to define metric_definitions in the HuggingFace estimator, see example: notebooks/sagemaker-notebook.ipynb at main · huggingface/notebooks · GitHub
Hello @philschmid ,
I am running the Sagemaker pipeline for 4 steps [step_preprocess, step_train,step_inference,step_metrics] and I want to compare the evaluation results among different trials. I am able to see the metric_definitions added for the training stage but I am not able to add the metrics in inference stage for the step_inference,step_metrics. I would like to see the metrics for the test data in the last 2 steps.
I am not sure if this is possible through the UI already. But you can use the SDK as shown in the example to a get a pandas data frame per experiment and then compare and visualize
from sagemaker import TrainingJobAnalytics
# Captured metrics can be accessed as a Pandas dataframe
df = TrainingJobAnalytics(training_job_name=huggingface_estimator.latest_training_job.name).dataframe()
df.head(10)