When your embeddings stop understanding your documents

We set up a RAG pipeline for a client in the insurance space. Good retrieval scores at launch. Six months later, the same queries were returning noticeably weaker results, and nothing obvious had changed in the infrastructure.

What had changed was the documents. New product lines, new regulatory terms, internal jargon that had shifted since the embedding model was chosen. The model was still encoding text the way it learned to during training. The corpus had moved on.

The silent part

Embedding models learn a fixed representation of language. You pick one, you embed your documents, retrieval works. But if your organization’s vocabulary evolves, the model does not update with it. A term that now carries a specific meaning internally might be treated as a generic word by a model trained on general web data from two years ago. Retrieval does not break loudly. Scores drift down quietly.

This is easy to miss because there is no error message. Your pipeline runs fine. Users just start finding the search “a bit off” and you spend time looking at chunking strategies or prompt wording before you think to look at the model itself.

The problem is worse in specialized domains. Legal, medical, financial, technical support. Anywhere vocabulary is dense and changes with regulation, product cycles, or internal convention.

What we actually do about it

The fix is not exciting. You schedule a retrieval quality check, roughly every quarter, using a small set of test queries with known good answers. You run them against your current index and record the results. If scores drop across two consecutive checks, you treat it as a signal to re-evaluate the embedding model.

Re-evaluation means running a small benchmark with current documents. We use a sample of maybe 50 to 100 document chunks, generate questions from them manually or with a model, then compare candidate embeddings on retrieval accuracy for those questions. Tools like MTEB give you a general leaderboard, but your domain sample is more useful than any public benchmark for this decision.

If a newer model or a fine-tuned one performs better on your sample, you re-embed. That is the boring part: re-embedding the full corpus, updating the vector store, re-testing before going live. In Pinecone or Weaviate this is a namespace swap or a collection update. Time-consuming, not complicated.

The main thing is having the scheduled check at all. Most teams do not. They embed once, move on, and only investigate when someone complains. By then retrieval has been degrading for a while and nobody noticed because it happened gradually.

We do not have a perfect answer for how often to check. Quarterly works for most of our clients. If the domain is very active, monthly makes more sense. The point is to make it a calendar item, not a reaction.