Hierarchical Chunking With Document Structure Signals

Document structure signals: what they are and where they show up

Structure signals are markers already there in a document: numbered headings, H2 headers, H3 section headers, gaps between text, tables, code sections, bullets, figures. It's all already there. The structure is already in the document, ready for a parser to pick up properly, and most miss it.

How clear those signals are comes down entirely to format, and many retrieval-augmented generation setups break here before retrieval begins. Markdown makes it simple: #, ##, and ### set up a hierarchy the system can parse with no guessing. HTML buries the same structure beneath menus, CSS rules, tags, and div wrappers, so digging it out takes more than developer effort. HTML documentation can take 90% more tokens at most versus the same thing in markdown, and that eats into how much a system can fit in its context. If you feed HTML rather than markdown to an AI, the extra tokens are pure waste.

Scanned technical documents and PDFs present a unique challenge. Structure is there too, just on the page: long tables, images crossing page breaks, and bold section titles. It only becomes useful after the document goes through visual analysis and OCR, which can add errors like wrong letters and missing entries that hurt retrieval and QA results.

People often read Structure-based and hierarchical chunking as synonyms, though they aren't. Structure-based chunking relies on figures and headers to set boundary markers, full stop. Hierarchical chunking captures the parent-child relationship between sections, so a chunk carries its full heading path, something like "Product Guide > Installation > Linux," as built-in provenance. One marks the split point. The latter shows how sections connect after splitting, and that structure helps an LLM cite accurately rather than gesturing vaguely toward "the documentation."

Making a layered chunk tree using document structure signals

It builds chunks mapped to the organizational structure of a document, not one flat set of equal-sized blocks. Section-level chunks align with headings and answer wider queries that ask for an overview. Subsection chunks, often anchored to an H3 or sub-topic boundary, handle most retrieval for targeted queries. Sentence-level chunks form the base nodes, retrieved when a query targets a single specific detail instead of broader context.

The payoff comes from the parent-child retrieval setup. The first pass runs on a tiny, focused child chunk because it lines up with a query's wording so tightly. When that result appears, the system sends to the LLM the larger parent chunk, meaning its enclosing section, for generating a response. This back-and-forth gets past the accuracy-versus-context problem in a way single-level chunking can't, since a uniform setup has to pick one level of detail and accept the results for every kind of search. Short chunks win precision but lack context. Big chunks win context but sacrifice precision. A hierarchical tree just doesn't pick one size ahead of time.

Every node in the tree holds a hierarchical identifier that maps a path through the document, making provenance precise rather than approximate. Docling's HybridChunker preserves the document's hierarchical structure as it segments, contextualizing every chunk through prepending its heading hierarchy prior to embedding. Research on a multi-agent SCORM system cites this setup, and it cites Tinh plus Phuong's results to prove structure-aware chunking produces measurable differences in retrieval outcomes.

How heading-aware chunking connects structure signals to retrieval outcomes

The literature defines section-aware chunking with one hard requirement: a chunk boundary must stay entirely inside one section. No boundary spans any heading. The heading always forces a split.

Tinh Pham and Luong Ngoc Phuong presented "Optimizing Context Retrieval for RAG via Heading-Aware Chunking and Hierarchical Document Structure Integration" at the 2025 IEEE 7th Symposium on Computers & Informatics (ISCI), DOI 10.1109/ISCI65687.2025.11167620. After close review, their main point still stands: a heading carries the semantic scope of the text below it. A chunk aligned to an H2 boundary inherits its context cleanly. When a chunk straddling H2 sections blends different subjects into one embedding vector, that blended vector fits any topic less well than distinct vectors would.

Pasting the heading path in before embedding makes this worse. A retriever doesn't see "kernel version must be 5.15 or higher" floating on its own. It sees "Installation > Linux > Kernel dependencies" riding along with that sentence, so the context survives even when the chunk text itself is dense, technical, and short on clues.

What the benchmarks show about accuracy and recall gains

Peer-reviewed work on care guidance reported adaptive chunking aligned to topic breaks achieved significantly higher accuracy than fixed-size baselines, with a statistically significant gap (p = 0.001). A system right 87% of the time and a system right half the time are not variations on a theme. One is usable with patients, and the other is just guessing dressed up as software.

MultiDocFusion benchmark shows this in industrial settings. Building document structure into the process raised retrieval accuracy by 8 to 15%, and ANLS results went up 2 to 3% compared to standard approaches in real-world tests. The improvements were linked to the hierarchical tree structure in the benchmark.

Benchmarks don't all line up; the disagreement stays instructive, not a contradiction. In February 2026, Vecta compared multiple chunking methods across 50 research papers. The 512-token splitting approach led with 69% accuracy. Semantic chunking, which costs more computationally and sorts content by semantics rather than structure, hit 54%, far under an approach with no semantic understanding. That finding does not undercut the argument for structure. It proves that extra compute for chunking buys no gain when the approach ignores the document's organization.

NVIDIA's 2024 tests bring in another angle: whole-page chunking scored 0.648 accuracy and posted the smallest swing of any approach tried. That result is limited to paginated documents, not unpaginated pages or markdown docs. The variance result stands on its own, apart from the accuracy figure. When variance stays down, a system holds steady no matter the query, rather than posting strong averages while swinging wildly from one to the next. For a system fielding lots of different queries each day, that consistency matters as much as highest accuracy, arguably even more.

Adjusting chunk size and calibrating parent size within a hierarchical system

Structure signals show where to split. Chunk size determines how much content each split holds. They're two different choices, and getting the structure right doesn't let them skip sizing, an error common in systems with the tree nailed but the tokens off.

Most RAG setups work fine with 512 tokens, trading exact matches for full context. It should vary by document type: brief Q&A items often use smaller chunks; long-form technical or legal pieces may require larger chunks to keep complete thoughts intact.

Good hierarchical setups often fall short at Parent-size calibration, but people ignore it once the chunking tree is built since that part seemed hard enough. When a parent chunk that's oversized, it wastes tokens on irrelevant content while dragging precision down. Parent chunks that are too tiny just rehash what the child chunk covered, defeating the reason for two layers.

The right calibration depends on the document type, and no single figure here is worth chasing. For technical and industrial PDFs, use larger parent chunks in a tree, and keep tables or figures atomic, not split mid-table by a token cap. Markdown documentation uses H2 for the parent and H3 for the child. FAQ or Q&A pages are the one case where the extra overhead of hierarchical chunking rarely pays off. Breaking it up by line or into bite-sized pieces works better there, because a parent-child setup just gets in the way of a brief document.

Making a document hierarchy-friendly for writers and strategists

When headings guide a retrieval system that leans on them to break chunks, their caliber determines retrieval success. If a document's headings are inconsistent or absent, even the best-built chunking downstream still ends up with poor chunks. Authors who use titles just for looks end up hurting how easily their material gets found.

Each section under a heading should contain one complete thought that still works when a retriever pulls it from the document entirely and gives it to a system. That calls for resisting the urge to stretch one point over several headings, and for not burying the key fact a person needs paragraphs into dense prose, leaving a chunker with little hope to isolate it.

Giving AI programs structured markdown through negotiation, and putting llms.txt at the documentation base, drops token spend. HTML at up to 90% extra tokens over markdown sets a hard ceiling on the knowledge base an AI can reference inside a single context window. This hard ceiling caps the knowledge base volume a bot can reference inside a single context window. Additional schema types can provide extra structure for the page alongside existing headings.

Tracking AI visibility and how well chunking works

Splitting content well won't put a company in an AI-written response, and the citation gap proves it. In 2025 AirOps examined 45,000 citations and saw only 30% of companies show up from one AI response to another, while just 20% remain after the same query fires five times in a row. They're not holding onto one fixed reply and serving it again. They rebuild every answer anew, weighing freshness, topic range, and reprioritizing variety with each query.

Overlap between these tools just makes it harder. Across ChatGPT, Claude, Gemini, and Perplexity, only 25% of citations match, so a citation in one guarantees no reach elsewhere. Each one has to be tracked separately. Treating them as a single pool lets visibility gaps slip by unnoticed.

Few companies systematically monitor their AI visibility, leaving a significant gap in tracking. Structuring material clearly and seeing if that structure is earning citations are separate skills, and most teams have done one but not the other. Hierarchical chunking makes a document more likely to be retrieved and properly attributed. Without measurement alongside it, they can't know which documents get citations, which sections keep surfacing, or where citation visibility is eroding while the material remains untouched, just as it appeared at launch.