Table and Figure Ingestion Into Vector Stores for RAG
Enterprise RAG systems silently miss tables and figures, leaving models confident but incomplete.
A typical enterprise document corpus contains critical information in tables, figures, scanned pages, or annotated diagrams in 30 to 60 percent of documents rather than in clean text. Standard RAG systems, designed to split and embed words, miss it completely. No system failure occurs. No error is logged. The pipeline just skips anything in the document that doesn't look like prose, so the model downstream answers with missing context it never knew existed.
That quiet gap is what makes it unsafe. When a pipeline breaks loudly, someone patches it. A pipeline can stay live for months, giving fluent answers with fake confidence because it missed the table that mattered, before someone notices the pattern.
Four failure modes keep coming up, and each one needs its own answer. Scanned documents lacking text produce no retrieval instead of degraded retrieval. Without an OCR pass, a pipeline that only handles text can't embed anything from a PDF that's just a scanned page. Most parsers take Tables with nested headers, merged layouts, or uneven formats and flattened them into space-separated tokens, losing the table's row-to-column structure. A number sitting three columns away from its descriptor is reduced to a number sitting amid unrelated text.
Technical diagrams and schematics break in another way. A wiring diagram or piping schematic gets its meaning from the spatial arrangement of labeled components, and text can't faithfully convey that arrangement. And marked-up images, the kind seen in lab articles and graphic-heavy documents, usually put meaning in the image itself (different colors, nearby areas, added notes), not the text below.
Layout issues cascade, and another mechanic needs focus since it multiplies the trouble. One misclassification by a parser, seeing a sidebar as body text or with a table it merges with the paragraph above, breaks more than a line. It corrupts what gets retrieved for a whole block, and any query that touches it inherits the error.
Choosing between three architectures
Three architectures now handle visual content in a RAG pipeline well, but no single one beats the other two across the board. Each puts the cost of visual work at another point in the pipeline, shaping latency, accuracy, and storage even when people choose the trendiest route.
Caption-and-index takes on the cost at ingestion, using a vision-language model call to describe each chart or table in text before embedding. Everything else in the retrieval pipeline stays the same. Shared image-and-word vectors move the expense to creation: one system puts pictures and words in the same map, so no VLM call or image parser is needed. Retrieval that treats pages as images, a method ColPali helped popularize, defers cost furthest, putting it at retrieval through multi-vector matches with the rendered page's patch-level embeddings. It skips the parsing step.
This is not a leaderboard. The answer depends on three variables a team can actually change. Corpus matters at the start: archive scanned or digital PDF, dense with tables or with figures, multilingual? The Query matters equally: a factual lookup against a table behaves differently than one requiring visual reasoning across a diagram, while cross-modal synthesis, with a text query and picture response, stands apart. And constraints like GPU budget, latency service-level targets, and vector storage cost will eliminate some architectures before accuracy even becomes a factor.
Everyone in this space used ViDoRe V1 as their benchmark until it got saturated, and multiple models hit 90 nDCG@5. ViDoRe V1 remains a key benchmark, though newer evaluations continue to emerge.
Caption-and-index: converting visual content to text descriptions before embedding
The mechanic is straightforward. A PDF loader detects pictures and table chunks, lifting them out as individual artifacts away from the body text around them. A vision-language model gets the picture alongside a structured prompt telling it to capture axes, values, legend entries, and how the chart builds its argument. The description takes the chart's place in the index, acting in lieu of the raw pixels themselves. For a table, extraction comes before contextual enrichment: using the extracted table structure plus surrounding document text, an LLM produces a richer description that goes beyond the table entries. Both outputs are embedded as text and added to the current text index. No added retrieval setup, no second vector space, and no multimodal model in the path.
That simplicity is why some groups use it. For small document sets, caption-and-index is reasonable because a VLM call for each page barely affects the bill. It works when the queries hitting the system are the kind a good prose description can actually satisfy: "what was the trend in Q3 revenue" rather than "what color is the line that dips below the threshold in the bottom-left quadrant." And it's often the only realistic option for teams stuck with a legacy retrieval stack they can't touch for compliance or engineering-bandwidth reasons.
The failure modes are specific enough to test for directly, which is more useful than a vague warning to "be careful." Chart hallucination is the big one: VLMs misread axis scales, confuse legend colors, and, when chart resolution is poor or labels sit too close together, invent precise numbers that were never in the source. Fixing it takes discipline more than anything: have the model spell out its reasoning with the description, double-check VLM-extracted figures against structured info gathered during the initial parse, and put together a chart-QA test set before pushing the method across a corpus.
Tables face a similar issue. Converting a table to normalized text, like JSON, CSV, or markdown, can strip away context someone would see on the original page, like which subtotal goes with which region three tables back. The description can be technically accurate and still be badly suited for retrieval, because it answers the question "what does this table say" without answering "how would someone search for this table." Caption pollution compounds this risk: a hallucinated or badly suited description gets embedded into the index like any other text, so its error propagates through retrieval the same way genuine content does. A hallucinated description does more than yield a single bad result. It stays in the index as junk, degrading retrieval for any later query that touches that part of the document.
For table-heavy corpora combines table-to-markdown, the more useful variant adds contextual enrichment: working with the extracted table and surrounding page content, the LLM reviews it and crafts a description built specifically so vector retrieval can distinguish tables that look alike, like nearly-identical shareholder financial summaries across twelve quarters.
Unified vision embeddings: encoding images and text into a shared vector space
Unified vision embeddings do this another way: the same model encodes text and images into one shared vector space, letting a text query retrieve a figure or a figure query retrieve a paragraph, all within one index. Each page or chart gives a single vector, like a text chunk would, and no VLM call sits in the ingestion path adding cost on every document.
Most people start with CLIP, and it can frame the topic even though it’s not the pick here. CLIP still works for matching photos in retail listings, but it saw only everyday pictures with brief text during development, so packed grids, complex graphs, and screen grabs fall beyond what it can handle.
SigLIP 2 meaningfully improves. It keeps the SigLIP learning method and brings in image-to-text practice, self-distillation, masked prediction, work across languages, and built-in handling for different page sizes. So dense prediction and localization get better, along with retrieval of text embedded in images, which matters critically for document tasks. For documents that pair visual layout with text packed inside images, CLIP falls short of SigLIP 2.
Cohere Embed 4 is designed for document retrieval in business settings, not for broad visual AI work. It takes words and pictures in the same input and can read a PDF page as an image without any text-extraction step in between. It handles 128K tokens, roughly one 200-page document per call, with access through Amazon Bedrock, Azure AI Foundry, and the Cohere API. It also comes with Matryoshka embedding options of 256, 512, 1024, or 1536, so organizations can balance retrieval fidelity and storage cost based on what their corpus demands.
Voyage-multimodal-3 was among the first major single-vector multimodal models able to process slides, screenshots, and figures without a parser, and outperformed CLIP-large specifically for screenshot retrieval tasks and table search. Voyage-multimodal-3.5, the successor, adds moving clips plus a Matryoshka choice of sizes, useful for any corpus with screen recordings or instructional footage alongside plain documents.
On recent benchmarks, approaches such as Cohere Embed 4 or voyage-multimodal-3.5 perform competitively with ColPali-style models. On most enterprise corpora, single-vector methods now hold their own while costing far less to store than multi-vector retrieval. ColPali-style models continue to excel on challenging visual reasoning tasks.
Page-as-image retrieval with late interaction: the ColPali family
ColPali works on a defiantly plain idea: render each PDF page crisply, pass the rendered output to a model for vision-language tasks, then produce embeddings at the patch level. It takes in the rendered PDF as a human would, with spacing, grid rules, and labels intact, not the cleaned-up words extracted by parsing.
That method eliminates entire classes of failure by cutting out the parts that trigger them. OCR is skipped, so it can't stumble on messy scans. No layout detector can confuse a sidebar with multi-column body text. No table extractor sitting between steps, silently omitting a row.
Mechanically, ColPali's original model feeds the document picture to a vision-language model built on the PaliGemma-3B backbone. The model splits the page into patches, feeds each one to a vision transformer, then produces contextualized embeddings mapped to the same space as the model's representations. For each query, late interaction compares query's embeddings with all image embeddings, giving ColPali sensitivity to where on a page the answer sits, not seeing the page as an undifferentiated blob.
The lineup has grown considerably since the original version. ColQwen2 replaces the backbone with Qwen2-VL and handles multilingual content better. ColQwen2.5, built using Qwen2.5-VL, extends multilingual coverage and performs strongly on recent benchmarks. ColQwen3-4B, which combines ColPali and a Qwen3-based language model, is state-of-the-art for ViDoRe. When compute for full-size vision-language model demands is too high, ColModernVBERT brings its 250-million-parameter efficiency variant, just 0.6 nDCG@5 points off original ColPali with roughly a tenth as many parameters, for cases where GPU budget, not accuracy, is the constraint.
Parser and tooling choices that affect all three architectures
Parsers feed directly into two of the three architectures, and surprisingly shape the third as well. Both methods use a parser first to find and pull out page parts before they’re described or embedded. This architecture bypasses the parsing step when fetching results because the AI processes the visual document as-is, though numerous enterprise deployments continue employing one in parallel to create descriptive data or power a combined system merging visual lookup with keyword queries.
By 2026, most RAG setups use this group: Reducto, Unstructured, Docling, LandingAI ADE, Mistral OCR 3, AWS Textract, Google Document AI, Azure Document Intelligence, PyMuPDF4LLM, and LlamaParse. Costs vary from open-source options to paid services depending on the parser, and the best choice comes down to the corpus and budget rather than which parser tops a benchmark.
A few heuristics help guide parser selection based on common engineering needs. Some open-source parsers work locally and provide per-page confidence metrics, making it a good match for budget-tight on-premise work that has to keep documents off any third-party API. LlamaParse suits groups handling fewer than roughly a thousand pages of non-sensitive documents daily, where complex-layout accuracy matters and a no-cost plan is enough to begin. Docling, using IBM's transformer-based extraction, fits corpora with a high proportion of tables, where table structure fidelity is critical.
These programs can't handle visual content alone. The parser picks what survives extraction, and the setup controls what comes after. Mess up just one step and the model generating responses will never see a table's values or the structure of a diagram.
