How can I read PDFs with mPLUG/DocOwl2?

Models
1

How can I read PDFs with mPLUG/DocOwl2? If yes how should I edit the inference script?

1 Like
2

How about like this?

import torch
from transformers import AutoTokenizer, AutoModel
import fitz
# pip install PyMuPDF f icecream
# pip install flash-attn --no-build-isolation
class DocOwlInfer():
    def __init__(self, ckpt_path):
        self.tokenizer = AutoTokenizer.from_pretrained(ckpt_path, use_fast=False)
        self.model = AutoModel.from_pretrained(ckpt_path, trust_remote_code=True, low_cpu_mem_usage=True, torch_dtype=torch.float16, device_map='auto')
        self.model.init_processor(tokenizer=self.tokenizer, basic_image_size=504, crop_anchors='grid_12')
        
    def inference(self, images, query):
        messages = [{'role': 'USER', 'content': '<|image|>'*len(images)+query}]
        answer = self.model.chat(messages=messages, images=images, tokenizer=self.tokenizer)
        return answer

docowl = DocOwlInfer(ckpt_path='mPLUG/DocOwl2')

pdf = fitz.open("example.pdf")
images = [pdf[i].get_pixmap().pil_image() for i in range(len(pdf))]

answer = docowl.inference(images, query='what is the third page about? provide detailed information.')
print(answer)
1 Like
3

Thank you! Do you have any suggestions on how to extract confidence score of the answers?

1 Like
4

I don’t know!:laughing: It’s a field I know nothing about.