Iris

Why it existed

My dad had a business idea. Companies get a constant stream of cybersecurity and compliance questions, and a general chatbot is a bad way to answer them. It will invent a control number without blinking, it can’t tell you where anything came from, and it knows nothing about your own policies. When an answer ends up in front of an auditor, “probably right, source unknown” is worth nothing.

So the plan was a RAG product: ground every answer in the real standards, let a company add its own documents next to them, put a page-numbered citation on every claim, and keep each customer’s material completely separate. He asked if I could build it. I said yes before I knew what reranking or multi-tenancy meant.

I worked on it every day after school for months. It searched Qdrant and OpenSearch at the same time, fused the results, cited every source, and had row-level security, a fine-tuned model and evaluation in LangSmith. It was roughly 8,000 lines of code, and my statistics test went badly that term. I’d say it was worth it anyway, but I’d also say that to be polite to my past self.

Then Microsoft, Amazon and Google shipped managed RAG, hybrid retrieval and per-tenant indexes included, a few clicks and done, and better than one person can do it. I never asked a single customer whether they wanted it. The market I’d imagined wasn’t really there, and we drifted off it. I still got a lot out of it: the retrieval problem turned out to be genuinely interesting, it got me through the Azure AI-103 exam, and it’s the reason FibrumPDF exists.

The first version was LangChain over a local vector store. An LLM read a catalogue of documents and picked which ones to search before retrieval even started, and every document type had its own chunk size: 700 characters for NIST, 850 for ISO, 1,400 for Veeam manuals. The rewrite is what I’ll describe, because it’s the one I’d defend.

One question, end to end

question
├── embed it once, only to check the HyDE cache
├── HyDE: a small model writes a plausible fake answer
├── embed the fake answer
├── four concurrent searches
│     tenant scope:  75 dense (Qdrant)  +  25 keyword (OpenSearch)
│     global scope:  75 dense           +  25 keyword
├── fuse dense with keyword inside each scope, then fuse the two scopes
├── embed the keyword-only hits
├── maximal marginal relevance, down to 50
├── split 8 slots between the scopes by how close each one got
└── at most 2 per document, merged by rank

Two model calls leave the machine per question, usually one, because HyDE is normally a cache hit. The embedder runs locally. Retrieval takes about two seconds.

Why not just embed the question

A question and its answer aren’t written alike. “How do we handle a breach?” and “The organization implements an incident handling capability that includes preparation, detection and analysis, containment, eradication and recovery” are about the same thing, and their embeddings aren’t close. So a small model writes a short, plausible, probably wrong answer first, and that gets embedded instead. The guess doesn’t need to be right. It needs to sound like an answer and use an answer’s vocabulary. That’s HyDE, and it’s most of the project’s name.

HyDE costs a model call per question and throws the output away, so I cache it. Caching on the question string almost never hit, because nobody types the same thing twice. Caching on the question’s embedding hits constantly: “how do we handle incidents”, “what’s the incident response process” and “walk me through IR” all land in the same neighbourhood, and one rough passage serves all three. The cache is its own Qdrant collection.

Fusing in a specific order

Dense similarity lives in roughly -1 to 1 and BM25 scores are unbounded, so adding them just means whichever store emits bigger numbers wins. Reciprocal rank fusion throws the scores away and adds 1 / (k + rank) per list, with k = 60 so that one very confident store can’t bulldoze the other.

The order matters as much as the method. Every question searches two scopes, the customer’s own uploads and a shared reference corpus. Fuse all four rankings at once and the reference corpus wins on volume: it holds thousands of chunks and a new customer holds forty. So I fuse dense with keyword inside each scope first, then fuse the two scope results, and size stops being a vote.

MMR then compares candidates against each other, which means every candidate needs a vector. Dense hits arrive carrying theirs. Keyword-only hits have none, so they get embedded in one batch first. Skip that, and they look maximally different from everything, and MMR happily picks all of them.

The reranker I deleted

Everybody recommends a cross-encoder reranker, so I added one and it sat in the pipeline for months, because a stage that popular must be earning its place. It wasn’t. When I finally measured it on three real questions:

It cost 36 of the 38 seconds a question took, and it was worse than useless on the question it was meant to help. Asked whether the company’s policy met ISO 27001 clause 5.2, dense search and keyword search each ranked the policy’s own commitments paragraph first, independently, and MMR kept it there. The reranker pushed it to fourteenth and out of the answer, so the assessment said there was no evidence of board approval from a policy whose first page says it was approved by the Board.

The reason is the same one HyDE exists for. A cross-encoder scores how well a passage answers the query as written. The question says “ISO 27001” and “clause 5.2”. A published guide says those words on every page, and a company’s own policy never says them at all. So the expensive model behaved like a slow BM25 and ranked the standard above the evidence. Two retrievers agreeing was the better signal all along.

Splitting the evidence budget

I used to give each scope half the slots, and that wasted the budget in both directions. Asked for evidence that quarterly access reviews happened on time, which is a question about one company’s records, four of the eight passages were an AWS whitepaper, an ATT&CK paper and two pages of SP 800-53. None were cited and none could have been.

Now the split follows the highest dense similarity each scope reached. Those are the one pair of numbers in the pipeline that genuinely share a scale: same model, same vector space, same query. Fusion ranks don’t, and neither did the reranker’s scores, which is most of why it had to go.

Neither scope ever drops below two slots. A question the customer’s documents can’t answer still has to be able to say so, and that’s only visible if some of their documents were actually looked at.

Grounding the answer

Retrieval picks the passages. This part decides what the model’s allowed to do with them, and it was the half I got wrong for longest.

Every passage in the prompt is headed ORGANISATION DOCUMENT or REFERENCE STANDARD before its filename. Only the first kind is evidence of what the customer does; the second only shows what a framework asks for. Without the label the model can’t tell an uploaded policy from an implementation guide about the same standard, and it will cheerfully cite the guide for a claim about your company.

The evidence block also says plainly what came back, including when no customer documents did. That’s the difference between “the policy doesn’t say” and “no part of the policy was retrieved”, and a model with no way to tell them apart writes the first when it means the second.

Models also love writing [36] when the passage they’re reading is about a control numbered 36. Nothing was retrieved under 36, so the number points at nothing, and a reader can’t tell that from a real citation. So citations are rewritten on the way out: a marker naming a passage that doesn’t exist gets dropped, and [1, 3] becomes [1][3] so every number is one thing to click. It holds back any tail that could still turn out to be half a marker, so a citation split across two stream frames is still caught.

Chunking

Extraction is FibrumPDF, which I wrote for exactly this. The first version used pypdf, which returns words in roughly reading order with the layout thrown away. About half of SP 800-53 is control tables, and they came out as unpunctuated word salad. Having real blocks (headings, paragraphs, lists, tables) is what makes any of this possible:

• Sections are boundaries. Level 1 and 2 headings start a chunk and repeat at the top of every chunk inside that section. “The organization shall implement…” on its own retrieves against nothing, because it never says what it’s regulating.
• Tables are atomic. A table is its own chunk. If it’s too big, it splits on row boundaries with the header repeated on every piece, because a fragment of a control table without its header is just numbers.
• Page furniture is dropped. Headers, footers and watermarks poison a chunk twice: they dilute it, and a watermark read as a heading gets repeated into the breadcrumb of everything beneath it. The top-scoring reference passage for one question was the foreword of a preview-watermarked standard, headed iTeh STANDARD PREVIEW. I detect it instead of listing it: a short line that appears on a third of a document’s pages is furniture no matter who published it, with digits normalised so PAGE 16 through PAGE 373 collapse into one key.
• Ids are derived. A chunk’s id is a uuid5 of the file’s SHA-256 and its position, so reindexing overwrites instead of duplicating.

One quiet bug is worth confessing. The embedding model started as bge-small-en-v1.5, which has a 512 token window, while chunks run to 1,500 characters. The tail of every long chunk was being silently truncated and never embedded. Nothing errors when that happens, the results just get a bit worse, which is the most annoying kind of bug. Moving to nomic-embed-text-v1.5, which takes 8,192 tokens, got the whole chunk in.

Making it fast

MMR was quadratic in the wrong variable. Picking 50 candidates out of 400 took 46 seconds. It takes 0.8 now. Three things were wrong and only the last mattered much. Vectors were re-normalised inside the similarity function on every call, the relevance term was recomputed every round although it never changes, and each round compared every remaining candidate against every pick so far. Only the newest pick can lower a candidate’s score, so a running maximum per candidate turns the inner loop into one comparison instead of k. I checked the output against the old implementation over 800 randomised inputs and four bias values before trusting it.

A turn took three transactions. Writing the question, the answer and reading the conversation back each opened their own connection, so two requests on one conversation could interleave into question, question, answer, answer. It’s one transaction with SELECT ... FOR UPDATE on the conversation row now. The four searches also run concurrently, since none needs another’s result.

Keeping customers apart

One instance, many customers, and their documents are the whole product, so this was the part that had to be right. Four layers, on purpose.

1. The tenant id comes from the validated token. It’s read from the tid claim of a signed Entra ID token and nowhere else. No route accepts a tenant as a parameter, so there’s no request you can construct that asks for someone else’s documents. This is the layer doing the real work.
2. Rows are keyed on identity. Every Postgres table is keyed on (tenant_id, user_id, ...). An earlier version keyed conversations on the conversation id alone, so two tenants picking the same id collided and a user could read a colleague’s conversation by guessing.
3. Row-level security checks it again. It catches the query someone adds later and forgets the WHERE. It does nothing unless the app connects as a role without BYPASSRLS, and my Compose setup doesn’t, so in development the policies are decorative. I wrote that down instead of leaving it as a surprise.
4. Storage is separate. Each tenant gets its own Qdrant collection and OpenSearch index, so there’s no filter to forget.

Testing without anything running

Importing the dense module used to build a Qdrant client and load about a gigabyte of embedding weights as a side effect, and importing the database module opened a connection pool. That made the API untestable without Docker, and made --help pay for a gigabyte. Settings are one frozen value now, everything that talks to the outside world is an object with lazy clients, and the routes read them off the app, so create_app can be handed fakes. The result is 66 tests in under a second with nothing running, including one that checks a tenant can’t see another’s upload jobs. The fake database is keyed the same way as the real one, so a cross-tenant read misses in a test for the same reason it misses in Postgres.

What it doesn’t do

No OCR, so scanned PDFs come out empty. Upload jobs and OAuth state live in a dict in one process, which is wrong for two replicas. There are no migrations, so changing the embedding model drops and reindexes the collection. The hierarchical routing the repo’s name promises never got built, because I’d already made the repo by the time I picked the name. And the Compose file is a development stack, not a deployment.

Four decisions before an answer

There is no public Iris repository to inspect. This note keeps to my own account of the architecture, and the exhibit explains the system rather than connecting to any company’s documents.