Update, we now have much better guides as well as official example scripts for object detection. This also includes nice calculations of mAP during training.
Find the guide here: Object detection
Find the scripts (both with Trainer API and Accelerate) here: transformers/examples/pytorch/object-detection at main · huggingface/transformers · GitHub