Embedding models keep improving, and the upgrade looks trivial: change one config value, point your embedding calls at the new model, done. Teams that treat it that way are often surprised a week later when search quality gets worse, not because the new model is worse, but because of what the change did to the index underneath it.

Embeddings from different models can't be compared

An embedding is a vector produced by a specific model. Two different models place the same sentence at unrelated positions in their own spaces, even if both output vectors of the same length. Similarity scores, whether cosine or dot product, are only meaningful between vectors from the same model.

I want to be upfront about the evidence. I couldn't find a vendor document that states this in so many words. What I found were consistent reports from practitioners. In one OpenAI community thread (community members, not OpenAI staff), users report that vectors from text-embedding-ada-002 and the newer text-embedding-3 models can't be mixed, even when 3-small is set to the same 1,536 dimensions as ada-002, and that there is no method for converting old vectors to a new model. Their conclusion was to re-embed the whole corpus. The point also follows from how embeddings work. You can verify it on your own data in a few minutes: embed a set of queries and their known-correct passages with the old model and with the new one, then compare similarity across models. Same-model similarity will rank the right passage highly. Cross-model similarity won't.

Why it fails silently

If you start embedding new queries with the new model while the index still holds old vectors, nothing throws an error. The search returns results, just increasingly unrelated ones, and the degradation is gradual. It is often misread as a retrieval-tuning problem: blamed on the reranker, or used to justify a chunking rewrite that won't help, because the defect is a mismatch between the model that indexed the corpus and the model answering the query.

A different dimension count is the loud version of the same problem. A fixed-size vector index will reject vectors of a new length. That's easier to catch.

Dimension and normalization details

Some providers let you shorten embeddings. OpenAI's embeddings guide describes a dimensions parameter and says that if you change the dimension manually after generation, you must normalize the vector yourself. The same guide recommends cosine similarity and notes that its embeddings are normalized to length one, so cosine and Euclidean distance give identical rankings. Two practical consequences: record the exact settings used (model, dimensions, normalization) alongside the index, and don't change them casually.

A safe migration (my suggested playbook)

Treat the embedding model as part of the index's schema, not as an implementation detail.

  1. Version the index. Store the model name, dimension and settings in index metadata, and refuse queries whose settings don't match.
  2. Build a new index in parallel. Re-embed the entire corpus with the new model into a separate collection, in the background, before sending any query to it. Don't re-embed incrementally as documents happen to be touched.
  3. Test on a fixed query set. Use representative queries with known-good passages, and compare recall and top-k relevance for both indexes. The held-out test set guidance applies here too.
  4. Switch atomically. Point traffic at the new index with an alias or config flag, and keep the old index for a rollback window.
  5. Re-check the rest of the pipeline. A new embedding model can change which chunk sizes work best and how hybrid search should be weighted. Rerun your retrieval evaluation after the switch.

Plan for the cost

Re-embedding a large corpus costs money and time. That's useful information in itself: it means your embedding model choice is more load-bearing than the architecture diagram suggests. Estimate the cost before you commit to a model, and keep the original text so you can always rebuild. If you need to count tokens for the estimate, see Tokenization Surprises.

The takeaway

Never point a new embedding model at an old index. Give embeddings a version, migrate with a parallel index and a test set, and treat the change with the same rigor as a database migration, not a one-line config bump.

Sources

Developer ToolsLarge Language ModelsSoftware Engineering