How to detect the relation between each objects

Models
1

Hi guys! I’m using Layoulmv3 to extract information from the invoice. There is one thing I’m confused about:
When using Layoutlmv3 for the token classification task, I got the output like this:

Screenshot 2023-12-25 102752
Screenshot 2023-12-25 1027521152×272 100 KB

(The red boxes and green boxes indicate the objects with the same class). I have a question: Are there any models to detect the relationship between 1 pair (1 green and 1 red)? For example:
These 2 boxes horizontally belong to 1 item.
Thank you so much!

2

Hi,

There’s a LayoutLMForRelationExtraction model which can be used for that purpose, got a notebook on that here: Transformers-Tutorials/LayoutXLM/Fine_tune_LayoutXLM_on_XFUND_(relation_extraction).ipynb at master · NielsRogge/Transformers-Tutorials · GitHub.

However, for relation extraction I’d recommend to instead train a generative model like Donut or Pix2Struct to simply generate key-value pairs. Refer to my demo notebooks here: Transformers-Tutorials/Donut at master · NielsRogge/Transformers-Tutorials · GitHub.

3

Thank you, @nielsr
For the Relation extraction task, the number of relations is predefined and has a fixed value. In the above image, there are 2 items in total so I understand that I can get 2 “relations” but for another case, I can get 3 “relations” (3 items), and seems like with this approach, you must detect lots of relations. Do you have any recommendations for this case?