I want to build a model that is able to read a scientific research paper and answer questions based on that knowledge. I have been surfing the web and came across such a project but it used Longformer model which might have derailed its performance.
I have also read that Scibert limits the tokens , therefore long text cannot be processed.
my questions are :
- is it advisable to use scibert as the pretrained model for this task , if not which better model is out there ?
- will scibert work on long texts ?