Hi @AlanFeder, in addition to @yusukemori’s useful advice you might find it instructive to start by working through the following tutorial on text classification: https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb
The short answer to your question is that you generally do have to fine-tune one of the pretrained language models like distilbert-base-uncased using AutoModelForSequenceClassification.
An alternative, but less performant, approach is to just use the last hidden states of the model as input features to a classifier like logistic regression. Jay Alammar has a nice example here (along with tons of great explanations about Transformers): A Visual Guide to Using BERT for the First Time – Jay Alammar – Visualizing machine learning one concept at a time.
To tackle the imbalance, you could try upsampling (downsampling) the minority (majority) class or failing that weight the classes directly in the loss function of the Trainer: Trainer — transformers 4.2.0 documentation
HTH!