@Petrus you can take a look at Sagemaker Serverless Inference for LayoutLMv2 model - #15 by mansimov
to see how to install detectron2 properly.
Are you trying to fine-tune VisualBERT in a SageMaker Notebook or using the HuggingFace.fit()
estimator=