Positional Encoding error, Protein Bert Model

Intermediate
1

Hi Guys,

This seems very obivious but I can’t seem to find an answer anywhere. I’m trying to build a very basic roberta protein model similar to ProTrans. It’s just Roberta but I need to use a very long positional encodings of 40_000, because protein seqeunces are about 40,000 amino acids long. But anytime I change the max postional embeddings to 40k I keep getting an CUDA error: device-side assert triggered error.

(additidonal question I save the linebylinetext to a variable. From what I understand linebyline splits the sentence into chunks, but how can I index in to see each chunk?)
(also other than a lambda function is there any more efficient way I can preprocess and tokenise this dataset?)

Steps

  1. Preprocess a Uniref50 sequence into a single space text document. Treating each Amino acid as a word and each protein as a sentence.
  2. Tokenise, I can load in the tokens from ProTrans model, tokenises fine.
  3. Use line by line text dataset to split each into 40_0000 blocks,
  4. Step up the config, important changes to roberta vocab size 30, max_positional_embeddings= 40_000
  5. Run through datacollator, model setup, train setup, get error.