Handling PDFs, Images, and Other Multimodal Inputs in an LLM Pipeline

The fastest way to add multimodal support to an LLM feature is to take whatever file a user uploads, base64-encode it, and hand it straight to a vision-capable model along with a question. It works in the demo. It gets expensive and unreliable fast, because "read the file" and "answer the question" are two different problems, and folding them into one model call means you can't debug, cache, or fix either one separately.

Separate extraction from understanding

A PDF isn't a picture of text — it's a container of structured content: paragraphs, tables, headers, footers, page numbers, sometimes scanned images with no text layer at all. Sending the raw file to a model and asking it to both parse that structure and reason about the content in one pass means every mistake looks the same from the outside: you can't tell whether the model misread a table or misunderstood a correctly-read one.

Split the pipeline into a read step and an understand step. The read step should be as deterministic as you can make it: extract the actual text layer where one exists, run OCR only on pages that need it, and preserve table and section structure explicitly rather than letting it collapse into a flat text blob. Only hand the understand step — the actual question-answering or extraction — the clean, structured output of the read step. When something goes wrong in production, you can now look at the intermediate extraction and immediately tell which stage failed.

This matters even more for images that aren't documents — screenshots, diagrams, photos of whiteboards. A vision model reasoning directly over the pixels is often the right call there, since there's no text layer to extract. But treat that as a deliberate choice for a specific input type, not a default you fall into because it's the least effort.

Budget for the real token cost, and cache the extraction

Images and scanned pages tokenize very differently from text, and it's easy to be surprised by the bill: a single high-resolution image can cost more tokens than several pages of plain text, depending on resolution and how the model tiles it internally. Downsample images to the resolution your task actually needs before sending them — a legibility check for a whiteboard photo needs far less resolution than a model default upload — and resize once, not per request.

The extraction step is also the natural cache boundary. If a user is going to ask a document multiple questions, don't re-run OCR or re-send the raw file to a vision model for every question; extract once, store the structured result, and run every subsequent question against that cached extraction. This is the same caching discipline teams already apply to embeddings, applied one layer earlier in the pipeline.

Treat file ingestion as its own pipeline stage, with its own tests and its own failure modes, rather than something that happens inside the prompt. It's the difference between a multimodal feature you can debug and one where every wrong answer is a mystery.