Incremental Ingestion and Index Updates for Changing Document Sets

Incremental Ingestion and Index Updates for Changing Document Sets ⟦c1⟧

Why full re-indexing breaks down as document sets grow

Keeping a retrieval system current as its underlying documents change is a solved engineering problem, not an unsolved one: change detection, targeted upserts, and soft deletes together remove the need to rebuild an index from scratch every time something shifts⟦c2⟧. Full re-indexing means every document gets re-ingested, re-chunked, re-embedded, and pushed into a vector store that replaces the one before it, cycle after cycle. That's fine at small scale. It stops being fine once the corpus grows to production scale.

At that volume, re-embedding everything on every cycle is not just slow, it's a waste of money on content that never changed. The staleness this creates is not an abstract concern either. A nightly re-index scheduled for midnight means a policy update an editor makes at 2 PM sits invisible to the retrieval system for ten hours, and every query asked in that window comes back confident and wrong⟦c4⟧. Confident and wrong is the worst failure mode a retrieval system can have, because nothing about the answer signals that it's stale.

The costs stack up past the freshness gap too. Full rebuilds burn compute time, they can knock an application offline mid-rebuild, and they spend embedding API calls on documents that never changed a single byte⟦c5⟧. None of that is inherent to retrieval architecture. It's a symptom of treating every update like a total rewrite. What follows is what an update strategy looks like when it stops doing that. The Introl.io guide states that typical production scale involves an initial backfill of 100,000 to 10 million documents and daily incremental loads of 1,000 to 50,000 new documents, and at that volume, full rebuilds are prohibitively expensive ⟦c3⟧.

The three-step logic that all incremental approaches share

Incremental loading runs on a delta discipline: move only what changed since the last run, never the whole corpus⟦c6⟧. Every incremental approach, whether it runs on a schedule or reacts to events in real time, breaks down into the same three steps ⟦c9⟧.

Step one is change detection, figuring out what actually changed. Timestamps, content hashing, CDC, and row-by-row comparison all serve this purpose, and which one fits depends on how much you trust your source system's clock and how much compute you're willing to spend confirming a hunch⟦c7⟧. Step two is selective transfer: only the documents flagged as changed move forward to re-embedding, and everything else gets skipped entirely, no exceptions⟦c8⟧.

A metadata contract underlies all of this, making it work. Documents need stable document IDs and stable chunk IDs, because deterministic updates and deletes are impossible without something fixed to key against⟦c11⟧. They need source URIs or paths so a human can audit what changed and why⟦c12⟧. And this metadata contract holds regardless of whether the pipeline underneath runs in scheduled batches or reacts instantly to events; the next two sections turn to how that timing choice changes the tradeoffs on offer⟦c13⟧. The Unstructured.io best-practices guide specifies a minimum metadata contract that makes all three steps possible ⟦c10⟧.

Batch incremental loading: simpler to operate, slower to converge

Batch incremental loading runs on a clock. Hourly, daily, weekly, whatever cadence fits, the job wakes up, pulls only the delta accumulated since the last run, and leaves the rest of the corpus untouched. It's simpler to build, easier to debug when something breaks, and for large uniform volumes of change it can genuinely be the more efficient choice.

The tradeoff is the same staleness gap that opened this piece, just narrowed rather than eliminated. New information sits unavailable to the retrieval system until the next batch fires, and depending on the cadence chosen, that could be minutes or it could be most of a working day. If changes pile up between runs and a batch triggers a massive re-embedding job all at once, that spike in compute demand can cause the exact downtime that full rebuilds were supposed to avoid. Rate limiting or priority queuing becomes necessary once batch size gets unpredictable.

A real example makes the ceiling on batch frequency concrete.

None of this makes batch the wrong choice. For small-to-medium corpora, for changes that arrive in predictable daily waves, and for teams without the appetite to stand up streaming infrastructure, batch incremental loading is the right call and often the only one that makes operational sense. A real-world illustration from the Backstage/GitHub issue shows that a 200k-entity catalog took 7 minutes per full index rebuild, the team's goal was to drop targeted updates to under 200ms, and the gap illustrates why batch frequency alone cannot close the latency problem without also changing the update unit ⟦c14⟧.

Streaming and event-driven updates: millisecond freshness through CDC

Streaming replaces the scheduled loop. Instead of waking up on a timer, the pipeline reacts to individual document change events as they happen, and only the document that actually changed gets re-embedded. Change Data Capture, CDC, is the mechanism that makes this possible: it tracks inserts, updates, and deletes in real time by tapping a database's replication log, the write-ahead log in PostgreSQL for example, rather than polling the database over and over to ask if anything's different⟦c15⟧.

A concrete pipeline works as follows: a document gets updated in PostgreSQL, CDC captures that change the moment it lands in the transaction log, an embedding function gets applied to that single document, and a materialized view holding the embeddings updates within milliseconds⟦c16⟧. That's the entire staleness gap from the batch section, collapsed to nearly nothing.

The tooling around this has matured fast. Debezium remains the dominant open-source CDC platform, with connectors covering PostgreSQL, MySQL, MongoDB, SQL Server, Oracle, Db2, and more; its recent releases add stronger exactly-once delivery guarantees, further refinement of incremental snapshots (a feature first introduced back in version 1.6) that don't require locking tables, and native support for Kafka's KRaft mode⟦c18⟧. Apache Flink's CDC connectors, available since Flink 1.12, stream directly from a database into a Flink job without needing Kafka sitting in between, which cuts both latency and the amount of infrastructure a team has to run⟦c19⟧. Kafka and Flink together remain common integration points for higher-volume event pipelines⟦c20⟧.

Storage format matters here too. Iceberg's V3 specification, ratified in 2025 and reaching full production stability with the 1.11.0 release in May 2026, replaced the older delete-file design with binary deletion vectors and native row lineage tracking⟦c21⟧. That change matters specifically for this use case, because it removes structural weaknesses that used to make Iceberg a poor fit for high-churn streaming CDC workloads⟦c21⟧.

Streaming isn't free, and it shouldn't be sold as a default. It demands event broker infrastructure, exactly-once delivery guarantees, and careful attention to ordering, none of which is trivial to run correctly. The cost is worth paying when staleness is genuinely measured in hours of business harm, not when it's a theoretical nicety. As of 2025–2026, datalakehousehub.com describes the tooling landscape as follows ⟦c17⟧.

Content hashing: the practical engine of change detection

Timestamps alone are a weak signal. System clocks skew, source systems record modification times inconsistently, and a "last updated" field can lie without anyone noticing. A hash of the actual content settles the question directly: did the bytes change, yes or no.

The mechanics are simple. Ingest the document, compute a SHA-256 hash of its content plus metadata, using sorted keys so the hash comes out the same regardless of field order⟦c23⟧. Look that hash up against what's stored in the index's metadata. If it's new or different, embed it and index it; if it matches, skip it and move on. Then update the stored hash so the next run has something current to compare against.

Think of it as version control for a document corpus: the same way a git commit only sends the files that actually changed, hashing only sends documents whose bytes actually moved. Granularity is a real design choice here too. Hashing the full document gives you a full-document skip, but hashing each page image separately gives page-level skip, so re-embedding a modified document touches only the changed pages⟦c24⟧.

There are gotchas that will bite a team that skips them. Hashes need to be persisted to disk or a database, not held in memory, because an in-memory dictionary vanishes on restart and forces a full re-embed of everything the next time the service comes up⟦c26⟧. Timestamps need normalizing before they go into the hash, since a field like 2025-01-02T14:30:45Z produces a different hash depending on whether milliseconds got appended somewhere upstream⟦c27⟧. And a version key belongs in the composite hash, something like content|metadata|v1, so that a schema change can force a full reindex simply by bumping the version to v2.

Count-based re-embedding, where a pipeline only re-embeds when the number of documents changes, leaves stale vectors sitting in the index pointing at text that's already been edited⟦c28⟧, a subtler failure mode. Hash-based validation is the actual guard against that kind of semantic drift creeping in silently, because it checks content, not just headcount⟦c28⟧. Hashing isn't universal though. Documents with non-deterministic content, embedded timestamps or random IDs baked into the text, will hash differently every time regardless of whether anything meaningful changed, and ephemeral or streamed content can pay hashing overhead for no real benefit ⟦c11⟧. For a corpus small enough that a full re-index finishes fast, hashing may just be unnecessary machinery. The Neural Base LlamaIndex Advanced course describes the mechanical steps as follows ⟦c22⟧. The Neural Base identifies production gotchas that must be called out explicitly, including the following ⟦c25⟧. Hashing should be skipped for documents with non-deterministic content, such as embedded timestamps or random IDs, for ephemeral or streamed documents where hash overhead adds latency without benefit, or for corpora small enough that full re-index is acceptably fast ⟦c29⟧.

Upserts for modifications and the harder problem of deletions

Most vector databases support upsert as a native operation: insert or update in one call, no full re-indexing required, and the new embedding simply replaces the old one at the same stable ID⟦c30⟧. This is the workhorse primitive for handling modifications, and for a large share of update traffic, it's genuinely all a team needs.

It has a real limitation though. Upsert still requires re-embedding all modified content, and it provides no version history and no support for temporal queries⟦c31⟧. That gap matters more in some domains than others, and the final section returns to it.

Deletion is the harder problem: the old chunk embeddings tied to a document have to be found and removed before the new ones go in. Updating a document is not simply an upsert with extra steps: the old chunk embeddings tied to that document have to be found and removed before the new ones go in. The system needs a reliable mapping from every chunk back to the document it came from⟦c32⟧. Get that mapping wrong and stale chunks linger in the index indefinitely.

HNSW, the graph-based index structure behind much of the vector search world, makes this worse than it needs to be. FAISS's HNSW implementation doesn't support in-place deletion at all, its remove_ids() call simply raises a runtime error, and HNSWlib only supports logical, soft deletion, marking a node as deleted without physically removing it⟦c34⟧. In both cases, true physical removal means rebuilding the index over whatever data remains⟦c34⟧. Soft deletion carries its own risk beyond the inconvenience: a soft-deleted node can remain a structural hub in the graph, still guiding the greedy walk that search performs, meaning content that's supposedly gone can still shape other users' results, and the latency difference between a soft delete and a hard rebuild can even open a timing side channel that leaks information about what was deleted.

Recent work has tried to formalize this choice rather than leave it ad hoc. Research presented at NeurIPS 2025 lays out three baseline deletion strategies, logical deletion, physical deletion, and full rebuilding, and proposes an algorithm that switches between them dynamically based on the workload⟦c35⟧. That matters because the FreshDiskANN paper found that both HNSW and Vamana indexes show a consistently deteriorating recall trend across multiple update cycles, with a fixed candidate list size retrieving fewer correct results after repeated upsert/delete cycles⟦c36⟧. Indexes built specifically for this problem exist now too: FreshDiskANN introduces a StreamingMerge procedure that folds an in-memory temp index into the long-term SSD-resident one, and SPFresh targets lower, more stable search latency specifically under continuous in-place updates⟦c37⟧. The practical implication is that recall degradation should be tracked as a metric in its own right, with a selective re-index of the affected partitions triggered before recall crosses below whatever threshold the application actually needs.

Vector index maintenance under continuous change

The index type chosen at the start of a project isn't a one-time architectural decision, it's a maintenance budget that gets paid down or racked up every day the system runs. Graph-based indexes like HNSW and DiskANN connect similar vectors with edges and search in logarithmic time, and they support incremental updates well enough, but their structure genuinely degrades under high churn⟦c38⟧.

IVF-based indexes have their own recent progress worth noting. Separate work on topology-aware local updates shows it's possible to adjust and optimize graph structure incrementally, in response to individual updates, without rebuilding the full graph at all, preserving index quality while sidestepping the rebuild cost entirely⟦c46⟧.

Text retrieval solved a version of this problem earlier. Inverted indexes maintain compressed postings lists in memory as documents ingest, use buffer doubling to keep the layout close to contiguous, and avoid the post-hoc merge step that used to be standard practice⟦c47⟧. That's decades of information retrieval engineering that vector search is still catching up to in places.

Transformer-based Differentiable Search Indexes face a stranger problem: forgetting ⟦c50⟧. Because a DSI encodes the corpus into model parameters rather than a lookup structure, naive fine-tuning on new documents causes both implicit and explicit forgetting of documents the model learned earlier⟦c50⟧. DSI++ addresses this with Sharpness-Aware Minimization, which nudges training toward flatter minima that generalize better across updates, and with generative memory replay, synthesizing pseudo-queries against older documents so the model keeps encountering them even after training has moved on⟦c51⟧. It's a genuinely different failure mode from anything HNSW or IVF faces, because "incremental update" means something different depending on what kind of index sits underneath the retrieval layer ⟦c33⟧⟦c39⟧. EmergentMind reported in 2024 ⟦c40⟧. Ada-IVF achieves 2–5× higher update throughput than leading alternatives ⟦c41⟧. It matches query-per-second at 0.85–0.9× baseline across varied update/query locality scenarios ⟦c42⟧. Zhang et al describe sliding-window graph indexing with BIC ⟦c43⟧. EmergentMind reported in 2024 that it achieves up to 14× throughput improvement and 3,900× tail-latency reduction compared to state-of-the-art dynamic indexers ⟦c44⟧. Topology-aware local updates are described in arXiv 2503.00402 by Yu et al ⟦c45⟧. Moffat et al reported in 2013 ⟦c48⟧. 2022 ⟦c49⟧.

Versioning, temporal retrieval, and the hybrid strategy that ties it together

Vector indexes are built for query speed, and they handle continuous updates as a second-class concern; data lakes are built for versioning, and they pay for it in query latency. Vector indexes are optimized for query latency but handle continuous updates poorly, while data lakes excel at versioning but introduce query latency penalties, so most systems must live in between⟦c52⟧.

LiveVectorLake's answer to that tension is content-addressable synchronization at the chunk level, which keeps version history intact without giving up retrieval speed⟦c53⟧. That kind of design matters specifically for compliance, audit, and analytics use cases that need to ask what a document said on a specific date in the past, a question that plain incremental upsert simply cannot answer, because upsert overwrites, it doesn't remember.

The practical answer most enterprise deployments land on is a hybrid, not a single strategy applied uniformly. Frequent incremental updates, whether batch or streaming, handle day-to-day freshness. A periodic full re-index, scheduled deliberately during low-traffic windows, compacts accumulated deletions, resets structural degradation in the graph, and clears out the recall decay that builds up over months of upserts. And trigger-based re-indexing of specific degraded partitions, rather than the whole corpus, handles the cases where recall drops in one region of the index without justifying a system-wide rebuild.

Which mix makes sense depends on the corpus. A small, slow-changing one does fine on a full re-index schedule, and hash-based skip logic still saves on embedding costs even there. A large, frequently updated corpus wants streaming CDC for whatever content churns fastest, batch incremental for segments that are stable but still growing, and a periodic full rebuild kept in reserve purely for index health⟦c54⟧. A compliance-sensitive corpus wants a versioned store, something like Iceberg V3, serving as the actual source of truth, with the vector index treated as a query layer that gets refreshed incrementally underneath it⟦c55⟧.

The goal was never to eliminate full rebuilds. It's to make them rare, deliberate, and scoped to only the partitions that actually need one, so that freshness and search quality stop being a tradeoff and start being two things a well-run system delivers at the same time ⟦c4⟧.