Card forensics using VLM localized prompt approach

Beginners
1

Hi

we have passports of different countries for which we need to check card forensics on below points:

  1. check background pattern of the UAE passport is tampered

  2. Face is present on the left hand side of the UAE passport

  3. Logo is present on top right corner of the UAE passport

Ex: UAE Passport

passport.PNG
passport.PNG646Ă—450 94.6 KB

we need localized model by which we can train using our images. pls suggest

Just am thinking loud :slight_smile:

The approach we are following for point (1) using LLM model we will extract background extract feature vector and extracted prompt for the pattern
and store in json file as shown below.
Ex:
{
“pattern_vector”: [… 768 numbers …],
“prompt_respone”: “Diagonal guilloche mesh, light-gray watermark…”
}
using VIT encoder extract the pattern vector
using LLM prompt we extract the prompt response

while inference we match these 2 points and classify as original or tampered

Guys pls suggest if some better idea

@ForensicIntelligence @llm-researcher @VLM-Forgetting @transformers-ai

Thanks
vijay

1 Like
2

For now, I gathered resources.

3

that’s a detailed insight on the possible ways… thank u so much @John6666

To make things simpler instead of relying completely on the VML Models. i think we better use

step 1) card Registration: once the passport is identified we will manually update the JSON file as below

{
“passport_type”: “UAE-passport”,
“fields”: {
“Type”: {
“location”: “top left area”,
“value_type”: “text”
},
“Country Code”: {
“location”: “top center-left”,
“value_type”: “text”
},
“Passport No”: {
“location”: “top right”,
“value_type”: “alphanumeric”
},
“Names”: {
“location”: “middle right”,
“value_type”: “text”
},
“Nationality”: {
“location”: “middle right below name”,
“value_type”: “text”
},
“Date of Birth”: {
“location”: “middle right”,
“value_type”: “date”
},
“Sex”: {
“location”: “middle right”,
“value_type”: “text”
},
“Place of Birth”: {
“location”: “middle right”,
“value_type”: “text”
},
“Date of Expiry”: {
“location”: “middle right”,
“value_type”: “date”
},
“Date of Issue”: {
“location”: “middle right”,
“value_type”: “date”
},
“Issuing Authority”: {
“location”: “bottom right”,
“value_type”: “text”
},
“Holder’s Signature”: {
“location”: “bottom right”,
“value_type”: “signature”
}
},
“face_on_passport”: true,
“face_location”: “middle left side (photo region)”,
“faces_count”: 1,
“background_designs”: {
“type”: “geometric pattern”,
“description”: “interlocked grid pattern with UAE emblem watermark”,
“location”: “entire card background”
},
“MRZ_isavailable”: true,
“MRZ_location”: “bottom middle of the passport”
}

Step 2) card validation: During validation we will be using json file and extract relavant fields from json and validate accordingly

With this approach we can solve 80% of problem, but only challenge i see is

During validation how can i check card background is tampered or original based on confidence score without VLM model getting hallucinated ?

Thanks

vijay

1 Like
4

Probably. While VLM models are versatile and easy to use without much fine-tuning, if you’re planning to fine-tune them for specific tasks before use, traditional CV models are often faster and more accurate when used as the main approach.