I think the overall direction is probably basically right:
If I were building this today, I would keep your four stages conceptually, but I would change two boundaries:
- Do not make Markdown the canonical intermediate representation. Keep the parser’s structured document/JSON with page numbers, bounding boxes, block types, tables, figures, formulas, etc., and treat Markdown as one export/view of that structure.
- Route PDFs according to what is actually inside them. A clean born-digital PDF with a usable text layer does not necessarily need the same OCR/VLM path as a scanned page.
A reasonable default route would be roughly:
PDF
│
├─ inspect: native text? bookmarks/TOC? scan? mixed?
│
├─ clean born-digital
│ └─ native-text/layout-first parsing
│
└─ scan / broken text / difficult regions
└─ OCR or document-VLM path
└─ optionally only for the difficult pages/blocks
↓
structured source document
(page / bbox / block type / text / formula / table / figure / hierarchy)
├─ Markdown export
├─ figure/image assets
├─ semantic annotations
│ Subject → Chapter → Topic → Concepts
└─ source-aware chunks
↓
embeddings + Firestore metadata
For tools, I would probably start by testing a small representative sample with these rather than trying to choose one globally from benchmark scores:
| Candidate |
Where I would try it first |
| Docling |
Good candidate when you care about a structured document model, provenance/page geometry, tables/figures, optional formula enrichment, and downstream structured chunking. |
| Marker |
Particularly interesting for a mostly born-digital corpus: its current fast route is optimized for CPU/native-text extraction with selective VLM use, while balanced is the more GPU/math-heavy route. |
| MinerU |
Useful if you want a ladder from a conventional pipeline backend to VLM/hybrid modes, plus a fairly rich intermediate representation such as middle.json. |
| PaddleOCR-VL |
Worth testing when scans, photographed/skewed pages, difficult OCR, tables, formulas, or visually messy documents are a large part of the corpus. The current PaddleOCR-VL-1.6 uses a 0.9B VLM plus layout analysis. |
I would not read that table as a ranking. The important part is that these are pipelines with different routing and structured-output behavior, not just interchangeable OCR checkpoints.
For local/cloud placement, I would also keep the boundary movable rather than deciding that whole stages must be local or whole stages must be cloud:
usually cheap/local:
- inspect PDF / detect text layer vs scan
- native text extraction
- layout / provenance capture
- canonical structured JSON
- chunking
- possibly embeddings
optional heavy local or remote:
- difficult-page OCR
- formula/table/VLM recognition
- batch processing
cloud:
- original/derived asset storage as needed
- figure assets
- Firestore documents + vector index
- optionally the heavy recognition service
Both MinerU’s HTTP client/server modes and Marker’s ability to point at a separate inference server are examples of why this boundary does not have to coincide with your logical stage boundaries.
Why I would keep structured JSON as the canonical source
The main reason is not that Markdown is bad. Markdown is very useful for inspection, prompting, and interchange.
The problem is that a textbook ingestion pipeline eventually needs information Markdown does not naturally represent very well:
{
"block_id": "p042-b017",
"type": "formula",
"page": 42,
"bbox": [112.4, 328.7, 486.1, 371.0],
"text": "...",
"parent_section": "3.2",
"parser": "example-parser",
"parser_version": "...",
"source_document_id": "..."
}
For a figure you may eventually want something more like:
{
"figure_id": "fig-p042-03",
"page": 42,
"bbox": [80, 190, 510, 430],
"image_uri": "...",
"caption": "...",
"nearby_block_ids": ["p042-b011", "p042-b012"],
"chapter_id": "...",
"topic_ids": ["..."]
}
That makes later operations much easier:
- regenerate Markdown without reparsing the PDF;
- change your chunking strategy;
- change embeddings;
- redo
Topic / Concept labeling;
- retrieve the original source region for citations;
- associate a generated question with an exact page/block;
- compare parser versions;
- fix a serializer without losing the parser’s original observations.
This is not purely theoretical. For example, there is currently an open Docling issue where OCR-created TextItems with valid geometry are present in the structured document but do not reach the Markdown serializer in a particular table-classification case: docling #3473.
That is a specific bug, not evidence that Docling generally loses content. But it illustrates a useful architectural point: if the structured representation survives, an export-layer problem can be recoverable; if Markdown was the only retained artifact, it may not be.
MinerU exposes the same general idea explicitly. Its documented middle.json contains page-level structures, bounding boxes, images, tables, equations, paragraphs, discarded blocks, backend name, and MinerU version. It also provides the simpler content_list.json for downstream use.
So I would probably think of:
structured document = source-of-truth-ish parser observation
Markdown = human/LLM-friendly serialization
semantic hierarchy = derived annotation
embeddings = replaceable retrieval index
rather than making each later layer replace the previous one.
Stage 0: use the PDF's cheap signals before OCR/VLM
Before running an expensive model, I would inspect each PDF/page for what already exists.
Useful signals can include:
- extractable native text;
- PDF bookmarks / table of contents;
- page geometry;
- embedded images;
- existing OCR text;
- heading numbering;
- possibly tagged-PDF structure.
This is especially useful for your Chapter → Topic stage.
For example, recent Docling versions have an optional heading hierarchy recovery stage that uses, in order, PDF bookmarks/TOC, heading numbering, and visual style signals. It does not require another model download.
That suggests an inexpensive ordering:
native/bookmark/numbering structure
↓
parser heading hierarchy
↓
LLM semantic annotation only where needed
rather than asking an LLM to rediscover the complete chapter hierarchy from flat Markdown.
I would still treat recovered hierarchy as parser output, not unquestionable truth: bookmarks can be absent or wrong, and visual heading inference can also be wrong.
Stage 1: text, equations, reading order, and tables
The phrase I would be careful with is “without hallucinating.”
I would treat that as an acceptance-test requirement, not as a property that can safely be attached to one parser/model.
For STEM material, the dangerous mistakes are often very small:
+ ↔ -
x² ↔ x2
d_k ↔ dk
10^-3 ↔ 10^3
O(n² d) ↔ O(n 2 d)
subscript/superscript loss
table row/column shift
missing footnote
wrong reading order
A parser can produce excellent-looking prose while one of these changes makes a generated exercise scientifically wrong.
The evaluation literature also tends to separate these failure types. OmniDocBench, for example, evaluates text, formulas, tables, layout, and reading order separately rather than reducing everything to “OCR accuracy.” It also contains book/textbook-like document categories.
So for a first comparison, I would take perhaps 10–20 representative pages from the actual textbooks and deliberately include:
1. clean born-digital page
2. scanned page
3. two-column page
4. dense display equations
5. inline math
6. table with numeric values
7. table containing formulas / superscripts
8. diagram + caption
9. small text / footnotes
10. cross-page table if the corpus has them
Then compare, separately:
| Check |
Why it matters |
| missing / invented text |
obvious grounding problem |
| reading order |
destroys paragraph/chunk meaning |
| formula exactness |
small symbol errors are semantically large |
| table cell structure |
row/column shifts can silently change facts |
| formula inside table cells |
often a separate recognition path |
| figure extraction |
needed for your Stage 2 |
| figure-caption association |
needed to understand the asset later |
| page/bbox provenance |
makes debugging and citations possible |
There is a particularly relevant current Docling limitation around formulas inside PDF table cells: issue #3828 describes PDF table cells remaining plain text while normal formula enrichment handles top-level FormulaItems. That means “formula recognition works” and “formula recognition inside tables works” should be tested separately.
This is also why I would not choose a parser from one aggregate leaderboard number.
A small sanity check I tried
I tried two deliberately small checks on four born-digital pages containing multi-column text, equations, figures, and a table.
First, I compared the original PDF pages with an image-only rasterized copy of the same pages.
On a small CPU environment, the native/digital path without OCR took roughly 41 s, while the OCR-enabled image-only path took roughly 213 s for the same four pages. Both produced structured items with page/bbox provenance.
This is only one tiny controlled test, so I would not use the ~5x figure as a general performance claim. But it supports the routing idea: paying OCR/VLM cost for every page when the native text layer is usable can be unnecessary.
More importantly, the OCR version looked generally good but still produced a small STEM-significant local error in the table (O(1) was corrupted). That is exactly the kind of error an aggregate text score can hide.
I also tried Docling with formula enrichment enabled on a T4. On those four pages it successfully converted the detected FormulaItems into LaTeX-like structured formulas while retaining page/bbox provenance, using about 5 GB peak observed GPU memory in that run.
Again, it was not perfect: there were small identifier/tokenization errors and one equation was fragmented/duplicated. The table’s mathematical complexity notation also lost superscript structure, which is consistent with the table-cell boundary mentioned above.
So my takeaway from those tests was not “use Docling.” It was:
routing matters;
structured provenance is useful;
and STEM acceptance tests should be content-type-specific.
I also tried to make a like-for-like Marker balanced test on a free T4 notebook, but the supported NVIDIA inference route requires Docker/NVIDIA container support and that sandbox could not start the Docker networking stack. I would not interpret that as a Marker quality result; it is only an operational reminder that a free notebook environment is not always a good proxy for a normal NVIDIA host.
Stage 2: figures and diagrams
I would initially not add a separate figure detector unless the document parser proves insufficient.
Docling, Marker, and MinerU already have concepts of picture/image blocks and geometry; Marker explicitly extracts/saves images, and MinerU’s intermediate format distinguishes image bodies, image captions, table bodies/captions, charts, etc.
For your first version, I would persist something like:
figure image
+ stable figure ID
+ page
+ bbox
+ caption
+ nearby text block IDs
+ chapter/topic IDs
That is enough to preserve the option of building visual questions later.
If “visual questions” eventually means:
given a natural-language query, retrieve the relevant diagram even when its caption/text is insufficient
then that becomes a separate multimodal retrieval problem.
Projects such as ColPali and VisRAG are useful references for that direction.
I would not add that complexity at the beginning, though. ColPali-style retrieval uses multi-vector late interaction; it is not simply a drop-in replacement for one normal Firestore vector field.
So I would preserve the images/provenance now and add visual retrieval only if the actual query set demonstrates a need for it.
Stage 3: Subject → Chapter → Topic → Concepts
I like this schema as an educational semantic layer.
I would just avoid making it replace the source structure.
For example:
{
"source": {
"document_id": "...",
"page": 42,
"bbox": [112, 330, 486, 371],
"block_ids": ["p042-b017"],
"heading_path": [
"Chapter 3",
"3.2 Scaled Dot-Product Attention"
]
},
"semantic": {
"subject": "Computer Science",
"chapter": "Attention",
"topic": "Scaled Dot-Product Attention",
"concepts": [
"queries",
"keys",
"values",
"attention scaling"
],
"evidence_block_ids": [
"p042-b017",
"p042-b018"
]
}
}
The important part is evidence_block_ids.
That lets you later change the semantic taxonomy without losing the connection back to the textbook.
It also lets you validate an automatically inferred concept:
Concept:
"scaled dot-product attention"
Evidence:
page 42, blocks 17–18
rather than storing a concept label whose origin is no longer recoverable.
If you use an LLM for this semantic stage, I would constrain its output to a JSON schema and require evidence/source IDs for every derived node where practical.
That makes Stage 3 an annotation process instead of an irreversible document rewrite.
Stage 4: embeddings and Firestore
For retrieval, I would probably not embed only the Topic/Concept labels.
Instead:
metadata:
subject
textbook
chapter
topic
concept
page
content_type
vector:
actual source-grounded chunk
Then a query can do something like:
metadata prefilter:
subject = Physics
chapter = Mechanics
vector search:
"examples that require applying Newton's second law"
Cloud Firestore supports vector nearest-neighbor search plus metadata pre-filtering. Its current Vector Search documentation specifies a maximum embedding dimension of 2048; pre-filter + vector search uses a composite vector index.
I would therefore decide the chunk representation before spending much time comparing embedding models.
A cheap experiment is to hold the encoder constant and compare:
A. topic label only
B. raw paragraph/chunk
C. heading path
+ caption/table-header context
+ source chunk
If C beats B, you have learned something about representation without changing the embedding model at all.
Docling’s HybridChunker follows a similar general idea: it starts from document hierarchy, keeps heading/caption metadata, then performs tokenizer-aware split/merge. It can also repeat table headers when a table must be split into multiple chunks.
For textbooks, that can matter a lot. A chunk containing:
0.72
0.81
0.93
is almost meaningless if the table header describing those values was left in another chunk.
I would initially use whatever good multilingual/general embedding model fits the language and Firestore dimension constraint, then evaluate retrieval on the actual corpus before optimizing the encoder.
A tiny retrieval set can be enough to start:
query
→ expected textbook/page/block
and then measure whether the expected source appears in top-k results.
Include a few:
- equation queries;
- table-based queries;
- cross-section queries;
- diagram-related queries;
- “no answer in this book” queries.
Local/cloud deployment choices
I would not tie the architecture to one hardware assumption yet.
Mostly born-digital textbooks, modest laptop
Something like:
local:
PDF inspection
native-text/layout parser
structured JSON
Markdown
chunking
embeddings if practical
cloud:
object storage
Firestore
optional OCR/VLM fallback service
Marker’s current fast mode is interesting in this case because it is explicitly designed around the PDF text layer and lightweight CPU detection, calling the VLM only for equations, bad blocks, or scanned/bad pages. See the current Marker README.
Mixed scans and born-digital
A hybrid parser becomes more attractive.
MinerU currently exposes local backends including pipeline, vlm-engine, and hybrid-engine, plus remote vlm-http-client and hybrid-http-client modes. Its vlm-http-client is specifically a lightweight remote-client path that does not require local torch.
That gives you a useful deployment shape:
laptop:
preprocessing / orchestration / storage contracts
GPU machine:
heavy VLM recognition
same downstream schema:
regardless of where recognition ran
Scan-heavy / distorted real-world documents
I would include PaddleOCR-VL in the first test set.
The current PaddleOCR documentation identifies PaddleOCR-VL-1.6 as the latest version in that series. It retains a compact 0.9B VLM and the PP-DocLayoutV3 layout stage, with formula/table/text and distorted-document handling being explicit targets.
Again, I would evaluate the whole pipeline, not infer expected PDF parsing quality from the model parameter count alone.
GPU and math-heavy corpus
Marker balanced, MinerU’s heavier paths, PaddleOCR-VL, or Docling with enrichment become more relevant.
Marker currently makes this distinction explicit:
fast -> CPU-oriented, text-layer-first, selective VLM
balanced -> GPU-oriented, VLM layout/OCR, inline math recognition
So the “correct” route depends strongly on whether your textbooks are mostly clean digital PDFs or scans/math-heavy documents.
What I would build first
I would resist building the entire four-stage production pipeline before checking parser behavior on your books.
A low-cost first version could be:
Step 1
Choose ~10 representative textbook pages.
Step 2
Run 2–3 parser routes.
Step 3
Keep their structured JSON + Markdown + figures.
Step 4
Manually inspect:
reading order
equations
formula-in-table cells
tables
figures/captions
missing text
invented text
page/bbox provenance
Step 5
Pick one default route and one difficult-page fallback.
Step 6
Define the canonical source schema.
Step 7
Add Subject/Chapter/Topic/Concept annotations
without deleting the original source hierarchy.
Step 8
Create chunks and a small retrieval test set.
Step 9
Only then optimize embedding choice / visual retrieval / question generation.
This should tell you much more than comparing headline benchmark scores.
It also keeps the expensive experiments small.
One more separation for the final test-generation stage
Since the eventual goal is building tests, I would keep one additional boundary in mind:
document parsing
↓
retrieval
↓
evidence selection
↓
question + answer generation
↓
question validation
Correct retrieval does not automatically mean a good educational question.
For each generated question, it may be useful to keep:
{
"question": "...",
"answer": "...",
"source_block_ids": ["..."],
"source_pages": [42],
"figure_ids": [],
"topic": "...",
"skill": "...",
"difficulty": "..."
}
The first validation criteria can remain simple:
- Is the question answerable from the cited source?
- Is the answer supported by that source?
- Did the generated question alter a number, sign, symbol, or unit?
- Is it duplicating another generated question?
- Is the intended topic/skill actually being tested?
There is existing research treating educational/document-grounded question generation as a separate problem rather than merely “RAG succeeded”; examples include FOCUS for source-grounded question generation and DiagramQG for course-question generation from diagrams.
I would not implement a sophisticated pedagogical evaluator on day one. The useful architectural point is simply to retain the source evidence so that this final stage can be checked independently.
So, if I had to choose a practical starting point without knowing the exact laptop GPU/RAM or scan ratio, I would probably do this:
1. Preserve a structured parser representation as canonical.
2. Export Markdown from it, rather than using Markdown as the only retained source.
3. Route clean digital PDFs differently from scans/difficult pages.
4. Try Docling / Marker / MinerU / PaddleOCR-VL on a small representative set.
5. Keep page/bbox/figure/table/formula provenance.
6. Add Subject → Chapter → Topic → Concepts as derived metadata with source IDs.
7. Embed source-grounded chunks; use the hierarchy as Firestore filters.
8. Validate STEM-specific failure modes before scaling.
That keeps most of your original design intact, while making it easier to swap parsers, move expensive recognition between local/cloud, change embeddings later, and trace a generated test question all the way back to the exact textbook source.