Layoutlmv2 token classification on documents having tokens larger than 512

Hello everyone, I am trying to finetune/create a Layoutlmv2 model for documents having tokens larger than 512. I have tried following but its not working:

Initializing Tokenizer and Layoutlmv2 from scratch:

That is how I am initializing the tokenizer and model. I have am training 50 data instances but training loss /epoch is clearly showing overfitting and loss is coming down as a very steep graph

I wanted to change the num_hidden_layers=24 and num_attention_heads=16 but on google colab it shows CUDA memory error.

I want to know if I am doing it right or i am missing something…? Before I move to sagemaker to train model with num_hidden_layers=24 and num_attention_heads=16 on a bigger GPU, I want to make sure I am doing it right. Looking forward to your helpful responses.