Rewrite the Question Before You Retrieve: Query Rewriting for RAG That Doesn't Change What the User Meant

A retrieval system embeds whatever string you give it. Users do not give it good strings. In a conversation, "what about the enterprise plan?" contains no searchable nouns, "does it support that?" points at something three turns back, and a pasted error message buries the actual question under stack-trace noise. Tuning the index will not fix a query that never carried the information.

Query rewriting adds a cheap step before retrieval: turn the raw input (plus relevant history) into one or more standalone search queries.

What a rewrite should do

• Resolve references. Replace pronouns and ellipsis with the entities they mean, using recent turns.
• Strip noise. Drop greetings, formatting, and boilerplate so the embedding reflects the question.
• Split compound questions. "Compare pricing and data retention" is two retrievals; one blended vector often matches neither well.
• Optionally expand. Add the terms your documents actually use ("SSO" alongside "single sign-on"), if you know your corpus vocabulary.

What it must not do

The risk is a rewrite that quietly answers a different question. Constrain the prompt: output only queries, add no facts not present in the conversation, and keep the user's specific identifiers (error codes, product names, version numbers) verbatim. Rewriting a precise string into a paraphrase can destroy exact-match signal that keyword search would have caught.

Keep the original query in the loop. Retrieve with both the original and the rewrite, merge the results, and let ranking sort it out. This makes the rewrite an addition rather than a single point of failure, and it limits the damage when the rewriter misfires.

Test it as its own component

Do not judge the rewriter by final answer quality alone; too many things vary. Build a small set of real conversation snippets with a hand-written ideal standalone query for each, and check two things: does the rewrite contain the necessary entities, and does retrieval recall improve against a fixed set of known-relevant documents? Include cases where the best behavior is to change nothing, since over-eager rewriting of already-good queries is the common regression.

Finally, log both versions. When a user complains the assistant "ignored what I said", the pair of original and rewritten queries usually shows the problem in seconds. Measure recall before and after; if the rewrite doesn't move it, remove the step and save the latency.