Image Comparison Models for Line Drawings

Is there a trained model best suited for comparing line drawings? The image comparison models seem to provide rough approximations of similarity, but I need a model that can compare the vertices found in diagrams to determine similarity with higher accuracy. I tried the DELF model on TensorFlow Hub, and it works reasonably well for photos, but fails miserably when I use it to identify landmarks in simple line drawings, which seems counterintuitive. Any direction or advice would be appreciated.