Keeping RAG Indexes Fresh: Incremental Updates Without Full Re-Embeds

Your RAG bot is only as current as its last ingest. The policy changed on Tuesday, the price changed on Thursday, and the bot keeps answering from the version it embedded 3 weeks ago, confidently and with citations. Index freshness is a pipeline property, not a model property: the fix is an incremental sync that detects what changed, re-embeds only that, and removes what no longer exists, instead of re-embedding the whole corpus on a timer.

Why do RAG answers go stale?#

RAG answers go stale because retrieval reads an index, not your source of truth. When a policy changes in Notion or a price changes in the docs, the vector index keeps serving the old chunk until something re-embeds it. The model then cites outdated text with full confidence.

Staleness is worse than absence in most products. A bot that says "I do not have that information" sends the user to a human. A bot that quotes last quarter's refund window sends the user away with a wrong answer dressed up as a sourced one, which is the same trust problem as a RAG chatbot returning confidently wrong answers for retrieval-quality reasons. The difference is that staleness is entirely self-inflicted: the pipeline knew how to ingest the document once, and nobody told it the document moved on.

The failure hides well because most corpora change slowly. Ninety-something percent of answers keep being right, the stale 5% surfaces as scattered complaints, and nothing in the retrieval metrics flags it, because similarity search works exactly as designed. It is finding the most relevant chunk among the chunks it has.

Why not just re-embed everything nightly?#

Full re-embeds are simple and they scale terribly. The cost grows with corpus size, not change size, so you pay to re-embed thousands of unchanged documents to pick up a handful of edits. Past a few thousand documents, the nightly rebuild window and the bill both stop being ignorable.

Modeled from the token arithmetic in this post. The ratio is the point; your bill depends on corpus and churn.

The modeled arithmetic makes the scaling visible:

Embedding spend is only half the waste. A full rebuild also rewrites every vector in the store, which churns the HNSW or IVFFlat index structures that make search fast, and it turns ingest into a long nightly job that fails at 3am and leaves you choosing between a stale index and a half-written one. Incremental sync keeps each run small enough to retry trivially.

The counterargument is real, though: a full rebuild is idempotent and self-healing. Whatever drift or corruption crept in, the rebuild flattens it. The mature setup uses both, incremental sync continuously and a full rebuild occasionally, the same way databases pair a write-ahead log with periodic snapshots.

How does incremental indexing actually work?#

Incremental indexing keeps the index in sync by processing only what changed: detect changed documents with a content hash, re-chunk and re-embed just those, upsert by stable chunk IDs, and tombstone chunks whose source text disappeared. Everything else in the index stays untouched.

Hashing is cheap and embedding is not, which is why detection comes first and touches everything while embedding touches almost nothing.

Two implementation details carry the whole design. The first is the content hash: store a hash of each document's normalized text alongside its chunks, and skip any document whose hash matches on the next pass. This is what makes sync cheap, because reading and hashing 10,000 documents costs practically nothing next to embedding them.

The second is stable chunk IDs. Derive each chunk's ID from the document ID plus its position or heading path, so that re-processing a document overwrites its old chunks instead of appending duplicates. Nearly every vector store exposes this as an upsert: pgvector via an insert with conflict handling, and stores like Qdrant treat points as replaceable by ID natively. Unstable IDs are the most common self-inflicted wound in ingest pipelines, and they produce a subtly degrading index where 3 near-identical versions of the same paragraph compete for the same queries. Deletes need equal care: when a document shrinks from 12 chunks to 8, the sync has to remove chunks 9 through 12 explicitly, or they linger as orphans that retrieval happily serves.

Which failure modes should you design against?#

Four failures account for most stale-index bugs: orphaned chunks that survive after a document shrinks, duplicates created by unstable chunk IDs, deleted documents that keep getting cited, and a silent mismatch after an embedding model change, where new vectors and old vectors stop being comparable.

Each of these hides behind healthy-looking retrieval metrics, because similarity search keeps working on whatever the index contains.

The model-change failure deserves the loudest warning because it is invisible at write time. Embeddings from 2 different models, or sometimes 2 versions of the same model, do not live in the same vector space. Upserting new-model vectors into an old-model index does not error; it just makes similarity scores between old and new vectors meaningless, so retrieval quietly degrades in proportion to how much of the index you touched. A model change always means a full rebuild into a fresh index version, never an in-place migration. The same logic applies if you change embedding dimensions to save storage.

Freshness metadata closes the loop on the remaining doubt. Stamp every chunk with its source's updated_at and surface it to the generation step, so the model can say "as of the March version of the policy" and you can filter retrieval by recency when 2 versions of a fact compete.

Should updates be event-driven or scheduled?#

Event-driven sync updates the index minutes after a document changes; scheduled sync batches changes on a timer. Events win when freshness is the product, like support answers about current policies. Schedules win on simplicity and cost, and a 15-minute cron covers most internal knowledge bases fine.

Webhooks are better and crons are easier, and every event pipeline needs a reconciliation sweep anyway.

The honest engineering answer is that webhooks are better and crons are easier, and the gap in outcomes is smaller than the gap in effort. An event pipeline needs webhook endpoints per source, retry handling, and a reconciliation sweep anyway, because webhooks get dropped and sources go offline. The sweep is just a scheduled sync wearing a different name. So most teams should start with the schedule, measure the actual staleness window against what their users notice, and add events only for the 1 or 2 sources where minutes genuinely beat quarter-hours.

When is a full rebuild the right answer?#

Incremental sync handles content edits. A full rebuild is for structural changes: a new embedding model, a new chunking strategy, or metadata you now wish every chunk carried. Build rebuilds as a versioned index behind an alias, so you can populate the new version and switch atomically.

Content edits flow through incremental sync; structural changes, like a new embedding model or chunking scheme, get a versioned rebuild.

The versioned-index pattern removes all the drama from rebuilds. Write the new index as docs_v7 while docs_v6 keeps serving traffic, run your retrieval evaluation against v7, and flip the alias only when the numbers hold. Rollback is flipping it back. This is also the only sane way to test a chunking strategy change, because chunking edits alter every chunk boundary and therefore every chunk ID, which makes them structural by definition.

What should you build first?#

Add 3 things to your existing pipeline this week: a content hash per document so unchanged files cost nothing, stable chunk IDs derived from the document and position, and a deletion sweep that removes chunks whose source is gone. Those 3 cover most of the value before any event plumbing.

Then make staleness measurable instead of anecdotal. Track the age of the oldest unsynced change and the count of source documents missing from the index, and alert when either crosses a line you chose on purpose. A RAG system's accuracy ceiling is set by retrieval quality, and retrieval quality is capped by whether the index still reflects reality. Freshness is not a nice-to-have on top of the pipeline. Past the first month, it is the pipeline.