I can not save the checkpoints in GCS

Hello,

I am using Seq2SeqTrainingArguments to finetune my model:

training_args = Seq2SeqTrainingArguments(output_dir='gs://my_bucket/models/') 

It is supposed to save the checkpoints in GCS (Google Cloud Storage). However, it is saving the checkpoints in my local (I am using Google Colab).

I am doing the authentication as it follows:

from google.colab import auth

auth.authenticate_user()

project_id = "my_project_id"
!gcloud config set project {project_id}

In fact, I can see my files if I type:

!gsutil ls gs://my_bucket/

But it I can not save my checkpoints.

Do you have any idea how can I fix it?

Thank you in advance.

Hi Jessica! I don’t really know how to solve that but I found a workaround.
You can create a custom callback to run at each epoch end (Therefore you will have all checkpoints) and upload the model checkpoint using this:

Hi @jessicalopez ,

Were you able to find a solution for this? How to save the checkpoints in GCS?