I’m looking to train a RoBERTa model on protein sequences, which is in many ways similar to normal nlp training, but in others quite different. In the language of proteins, I have 20 characters instead of the normal 26 characters used in english (it is 26 right? :D), so that is rather similar. The …

Using transformers (BERT, RoBERTa) without embedding layer

agemagician December 16, 2020, 9:00pm 9

Something like that could be a good starting point for you:

Topic		Replies	Views
How could protein language models generate outputs for natural language input texts? 🤗Transformers	4	418	November 21, 2023
Transformers with protein data Beginners	0	323	July 6, 2022
PreTrain ProteinBERT from scratch Flax/JAX Projects	5	2318	July 6, 2022
How to Fine-tune Rostlab/prot_t5_xl_uniref50 Model for Sequence Generation Beginners	0	387	April 5, 2023
Unmasker probabilities for all tokens in sequence 🤗Transformers	0	223	December 23, 2022