Incrementally finetuning a HF model in SageMaker

If I have an already pretrained model, and want to train it on some additional data, how do I go about that in SageMaker?

In this case I’ve trained a model loaded with AutoModelForSequenceClassification using “distilbert-base-uncased”.

You can provide your trained model as model_id instead of the base model this would further train it. But you need to make sure that you have the same labels etc.

Consider using huggingface’s Trainer and TrainingArguments. Please check this page: Trainer .
If you have any questions on how to do it, let me know and I will forward some other links or help you do it. :slight_smile:

1 Like

Thank you. Yes if you can please share any other links with additional context/backgrouynd that would be great :pray:

Can I do this without using the hub? Would I just unzip the model into a local directory and load this and train with data and use same labels?


@MaximusDecimusMeridi - if you want to do this without using the model hub then, yes, downloading the model from S3, unzipping it into a local folder, and using model = AutoModelForSequenceClassification.from_pretrained('path/to/model/folder') should work!



You can also provide the model in a similar fashion you do with your training in the estimator, e.g.{"train":s3_train_uri,"model":s3_model_uri}) which should download the s3_model_uri into /opt/ml/input/model or rather speaking stores the path in the env var SM_CHANNEL_MODEL

1 Like