Incrementally finetuning a HF model in SageMaker

Amazon SageMaker
1

If I have an already pretrained model, and want to train it on some additional data, how do I go about that in SageMaker?

In this case I’ve trained a model loaded with AutoModelForSequenceClassification using “distilbert-base-uncased”.

2

You can provide your trained model as model_id instead of the base model this would further train it. But you need to make sure that you have the same labels etc.

3

Hello!
Consider using huggingface’s Trainer and TrainingArguments. Please check this page: Trainer .
If you have any questions on how to do it, let me know and I will forward some other links or help you do it. :slight_smile:

1 Like
4

Thank you. Yes if you can please share any other links with additional context/backgrouynd that would be great :pray:

5

Can I do this without using the hub? Would I just unzip the model into a local directory and load this and train with data and use same labels?

Thanks

6

@MaximusDecimusMeridi - if you want to do this without using the model hub then, yes, downloading the model from S3, unzipping it into a local folder, and using model = AutoModelForSequenceClassification.from_pretrained('path/to/model/folder') should work!

Cheers
Heiko

2 Likes
7

You can also provide the model in a similar fashion you do with your training in the estimator, e.g. huggingface_estimator.fit({"train":s3_train_uri,"model":s3_model_uri}) which should download the s3_model_uri into /opt/ml/input/model or rather speaking stores the path in the env var SM_CHANNEL_MODEL

1 Like