I am using Trainer and in the TraningArgument I am passing a s3 location, My Sagemaker notebook has access to S3 and I am able to save data directly to s3 from the notebook. but when I pass the same in the Trainer output_dir it create the same folder structure in the current director. and does not send it to s3. Is there a way I can prohibit the Trainer to create dir struct. Because when I delete the folder that it send the run details to the s3 but At the same time whenever it save a checkpoint it again checks for the folder and as it is not present it creates a new one.
training_args = Seq2SeqTrainingArguments(
output_dir=‘s3://path/to/folder/’,
evaluation_strategy=EVALUATION_STRATEGY,
save_strategy = SAVE_STRATEGY,
learning_rate=LEARNING_RATE,
per_device_train_batch_size=BATCH_SIZE,
per_device_eval_batch_size=BATCH_SIZE,
weight_decay=WEIGHT_DECAY,
save_total_limit=SAVE_TOTAL_LIMIT,
num_train_epochs=EPOCH,
predict_with_generate=PREDICT_WITH_GENERATE,
fp16=FP16,
# resume_from_checkpoint=RESUME_FROM_CHECKPOINT,
load_best_model_at_end = LOAD_BEST_MODEL_AT_END,
)
I am doing this as I do not have save the file on notebook disk