Recommend an instance for MPT-7B and MPT-30B inference

Amazon SageMaker
1

I’d like to know what instance would be best for each.

I’ve already tried with Google Collab on the free and pro plans - and 25 GB isn’t enough to load the model.

I’ve tried Kaggle as well but haven’t figured out how to load the model.

I can load into ml.m5.4xlarge on Sagemaker but that’s for General Compute so I’m thinking accelerated comput is better.

2

I can train falcon-7b on ml.g5.2xlarge and falcon-40b on ml.g5.12xlarge

3

appreciate it, that’s helpful, ronan