Hello everyone,
I am trying to fine-tune cardiffnlp/twitter-xlm-roberta-base-sentiment with some additional text. I am using the Python API to setup the model.
Everything appears to work just fine, the only thing is that I encounter a memory shortage with MPS (I am working on a 2019 MacBook Pro with AMD Radeon):
MPS backend out of memory (MPS allocated: 5.69 GB, other allocations: 1.00 GB, max allowed: 6.77 GB). Tried to allocate 96.00 MB on private pool. Use PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 to disable upper limit for memory allocations (may cause system failure).
From what I understand, this is a physical limitation from my computer (perhaps I’m wrong - it’s surprising since the model is supposedly less than 2GB large)). Would it be possible to switch to CPU training rather than GPU training ? I don’t see any parameter in the model that seems to control that.
Any help or idea is welcome !
Here is the code I am using:
from autotrain.params import TextClassificationParams
from autotrain.project import AutoTrainProject
HF_USERNAME = ""
HF_TOKEN = ""
params = TextClassificationParams(
model="cardiffnlp/twitter-xlm-roberta-base-sentiment",
data_path="data/", #"Lyreck/tiktok_brat_comments", # path to the dataset on huggingface hub
text_column="text", # the column in the dataset that contains the text
target_column="label", # the column in the dataset that contains the labels
train_split="train",
valid_split="validation",
epochs=3,
batch_size=8,
max_seq_length=512,
lr=1e-5,
optimizer="adamw_torch",
scheduler="linear",
gradient_accumulation=1,
#mixed_precision="fp16", #need graphic card for this (no mps available)
project_name="finetuned-sentiment-model",
log="tensorboard",
push_to_hub=True,
username=HF_USERNAME,
token=HF_TOKEN,
)
project = AutoTrainProject(params=params, backend="local", process=True)
project.create()```
Thanks!