FibrumPDF

PDF extraction became the bottleneck

A PDF is an annoying format for extraction, yet almost every document system has to deal with it.

FibrumPDF started while I was building Iris. PyMuPDF4LLM produced useful structured output, but parsing a large document collection was slow enough to become part of the retrieval pipeline’s cost. I wanted to know whether I could keep the useful parts while moving much faster.

My first attempts were messy. I profiled Python, moved pieces into Cython, tried dropping into C, and eventually attempted far too much of the parser in pure C. It worked badly and the code was worse.

The version that made sense was simpler: use MuPDF for low-level PDF parsing, move the document processing into Go, and keep Python as the interface.

What it extracts

The point is not to dump text as quickly as possible. Tools such as pdftotext already do that well. FibrumPDF keeps the structure I need before a document enters search or RAG:

• headings and reading order;
• lists, emphasis and links;
• tables, including borderless tables; and
• page information that can be carried into citations.

The processing path looks like this:

PDF
 ↓
MuPDF
 ↓
text + positions + fonts + lines
 ↓
Go layout analysis
 ↓
structured document
 ↓
Python API

MuPDF extracts each page once. Go then handles grouping text, detecting headings, processing geometry, finding tables and rebuilding document structure. This keeps thousands of small values out of Python loops, reduces language-boundary crossings, and lets pages be processed concurrently.

There was no single clever optimisation. Most of the speed came from removing lots of small pieces of repeated work.

Reconstructing structure that is not there

A PDF usually does not contain an <h1>. It contains text at a coordinate, with a font size and style. FibrumPDF combines those clues with punctuation, numbering and surrounding layout to infer headings and reading order.

Borderless tables were harder. To a person, evenly spaced rows and columns obviously look like a table. To a parser, they are only text fragments at coordinates. FibrumPDF looks for repeated horizontal gaps across several rows and treats them as possible column boundaries.

That also made two-column research papers look like giant tables. I added checks for wrapped prose, newspaper-style columns, sparse grids and layouts covering most of the page. The detector is not only asking whether something looks like a table. It is also looking for evidence that it definitely is not one.

I liked this part of the project. It is geometry, whitespace and statistics trying not to get tricked, without a model looking at a screenshot.

Benchmarked on real PDFs

I benchmarked FibrumPDF against PyMuPDF4LLM and Docling across 512 PDFs using a version of Marker’s benchmark fitted to this project.

Parser	Pages / sec	Text score	Table TEDS
FibrumPDF	318.23	87.31	0.783
PyMuPDF4LLM	4.15	86.54	0.778
Docling	0.62	91.13	0.821

The surprising part was not only the speed. FibrumPDF ended up around 77 times faster than PyMuPDF4LLM on this benchmark while producing almost identical measured text and table quality.

Docling still scores better on harder documents, which makes sense because it performs more expensive document understanding. FibrumPDF occupies the middle I wanted: useful structure for retrieval, without waiting seconds for every PDF.