Your retriever found the right chunk and buried it at rank 31. A cross-encoder second pass fixes ordering without touching recall, and here is how to size it.
A user asks your system a question. The chunk that answers it exactly is sitting at position 31 of your retrieved candidates. You pass the top 5 to the model, the model answers from what it was handed, and the answer comes back wrong — or worse, plausibly half-right. Nothing in the pipeline threw an error. Vector search did its job: the document was found. Ranking failed it: the document was not found first.
That gap between "in the candidate set" and "in the prompt" is the most common quality ceiling in production retrieval, and it is the one with the most direct fix. Reranking is a second scoring pass over the candidates the first stage already returned, run by a model that is allowed to look at the query and the document at the same time. It finds no new documents. It reorders the ones you have, and it reorders them well enough that the distance between recall at 50 and precision at 5 mostly disappears.
Why the first stage is structurally limited
The retriever you are using almost certainly embeds text with a bi-encoder. The name describes the architecture exactly: there are two encoding passes, and they never meet. Your documents go through the model at ingest time and come out as fixed-length vectors. Your query goes through the same model at query time and comes out as another fixed-length vector. Ranking is then whatever similarity function you picked — cosine, dot product — applied between two arrays of floats.
Everything good about first-stage retrieval follows from that independence. Because a document's vector does not depend on the query, you compute it once and store it. Because ranking is vector arithmetic, an approximate nearest-neighbour index can search millions of chunks in single-digit milliseconds. You can add documents without touching the query side. The whole thing is cheap precisely because the two halves never interact.
Everything imprecise about it follows from the same fact. When you embed a chunk at ingest, the model has to compress every question that chunk might ever answer into one point in space, without knowing what any of those questions will be. A 600-word passage about certificate rotation covers the rotation schedule, the failure mode when a chain expires mid-handshake, and the specific flag that disables OCSP stapling. One vector has to stand in for all three. It will land somewhere in the average of them, and it will be beaten by a chunk whose single topic happens to sit closer to the query's single point — even when that chunk does not contain the answer.
This is not a tuning problem. No amount of normalisation or index-parameter fiddling fixes it, because the information was discarded at encode time. The query never got to influence what the document representation emphasised. Bi-encoders are a recall instrument: very good at pulling a relevant set out of a large corpus, mediocre at ordering that set.
What a cross-encoder does differently
A cross-encoder takes the query and one document and feeds them through the model together, as a single concatenated input. Every token of the query can attend to every token of the document and the other way around. The model is not comparing two summaries written in isolation; it is reading the pair and judging the relationship directly. The output is not an embedding at all. It is a single relevance score for that specific query-document pair.
That is the whole difference, and the consequence is worth stating plainly. The score depends on both inputs, so it cannot be precomputed. There is no index to build, no vector to cache, nothing to store at ingest. Every score is a forward pass over a query-document pair, computed at query time, and the cost of reranking scales linearly with the number of candidates you score.
Linear scaling over a whole corpus is a non-starter. Scoring a million chunks per query with a transformer is not a latency problem you optimise your way out of; it is the wrong shape of computation. Linear scaling over fifty candidates is completely routine. That constraint is not an inconvenience to engineer around — it is the reason a shortlist has to exist.
Late interaction as the middle ground
Between the two sits late interaction, of which ColBERT-style architectures are the well-known family. Instead of compressing a document into one vector, these models store a vector per token. At query time, each query token is matched against the document's token vectors and the per-token similarities are aggregated into a document score. The interaction is real, but it happens late, over precomputed token embeddings, rather than inside a joint forward pass.
The position is honest and it is not free. Quality lands above a bi-encoder because the query interacts with document detail rather than with a summary. Cost lands below a full cross-encoder because there is no per-pair forward pass. The price is the index: a vector per token instead of per chunk inflates storage substantially, and multi-vector search is more complex to operate than a plain ANN lookup. You are trading storage and operational complexity for query-time compute. On a large corpus that trade can go either way, and it deserves a measurement rather than an assumption.
Other things that rerank
Cross-encoders are the default, not the only option.
An LLM as a relevance judge. Prompt a general model to score or order candidates against the query. It is flexible, needs no training, and can reason about relevance in ways a fixed scorer cannot — multi-hop questions, negations, "which of these actually contradicts the claim". It is also the slowest and most expensive option per candidate, and its scores are less stable across runs. Reserve it for cases where the judgement genuinely requires reasoning.
Reciprocal rank fusion. If you have two result lists — dense and sparse, say — RRF merges them using ranks alone: a document's fused score is the sum over lists of 1 / (k + rank), with k a smoothing constant conventionally set to 60. No model, no training, no inference cost, and no need to calibrate two incomparable score scales against each other. It is often the first thing worth trying, and on a hybrid pipeline it recovers a real share of what a reranker would give you for approximately zero latency.
Heuristic boosts. Recency, source authority, document type. Cheap, interpretable, easy to get wrong. Apply them after reranking with a bounded weight rather than as a term inside the relevance score, so a stale-but-correct document cannot be buried by a fresh-but-irrelevant one.
Retrieve wide, then rerank narrow
The architecture that falls out of all this is fixed in shape and flexible only in its numbers.
Stage one retrieves a wide candidate set cheaply. Its only job is recall: get the right document somewhere into the list. Use whatever gets you there — dense vectors, BM25, both merged with RRF. Stage two reranks that set expensively and keeps the top few. Its only job is precision: get the right document to the top.
You retrieve substantially more candidates than you intend to keep because the reranker can only reorder what it is given. If the answer is not in the candidate set, no amount of second-pass scoring will conjure it. Every document excluded from the shortlist is permanently excluded from the answer. So stage one is tuned for recall at a generous depth, not for precision at the depth you actually serve — the two stages optimise different metrics on purpose.
The tradeoff has a clean shape. A larger candidate set raises the ceiling on final quality, because it makes it likelier the right document is available to be promoted, and it costs linearly more to rerank. The gains diminish sharply — the marginal candidate at rank 200 is rarely the answer — while the cost does not. Somewhere on that curve is your operating point, and it is the main knob you have.
As starting points to measure from, not measured optima: retrieve 50 to 100 candidates, rerank them, keep 3 to 10 for the prompt. Those numbers are a place to begin an experiment. The right values depend on your corpus size, your chunk size, your context budget, and your latency ceiling, and you find them by sweeping candidate depth against retrieval metrics on your own queries.
When it is worth it, and when it is not
Reranking pays best when your corpus contains many near-duplicate or topically adjacent documents — versioned docs, per-tenant variants of one policy, advisories differing in a single field — because that is exactly the case a single vector cannot separate. It pays when queries are natural-language questions rather than keyword lookups, because the intent is carried by structure the bi-encoder flattened. And it pays when precision at a small k matters, which it does whenever only a handful of chunks fit in the prompt.
It pays least when the corpus is small enough that the right document is nearly always first anyway, when queries are effectively exact lookups, or when your latency budget has no room in it.
And the important one: if your first stage is broken, reranking will not save it. Chunks that split answers across boundaries, an embedding model mismatched to your domain, no sparse retrieval on a corpus full of identifiers and error codes — these produce candidate sets that do not contain the answer, and a reranker cannot promote what it was never given. Reranking is a precision fix layered on adequate recall. If recall at 50 is poor, fix chunking, embeddings, and hybrid search first. That work is cheaper, carries no query-time cost, and raises the ceiling reranking then works against.
The latency and cost budget
Reranking is usually the largest single latency addition in a retrieval pipeline, and unlike embedding — paid once at ingest — it sits on the critical path of every query.
cost_per_query = num_candidates * cost_per_pair
latency_per_query = ceil(num_candidates / batch_size) * time_per_batch
Four levers, in rough order of effect. Candidate set size is linear and the one you control most directly. Model size trades accuracy for throughput; smaller rerankers exist for a reason. Batching matters more than people expect, because scoring candidates one at a time wastes most of your accelerator — score the shortlist in as few batches as the hardware allows. Local versus API decides whether a per-query network round-trip is in your budget at all.
Measure all of this on your own hardware with your own chunk lengths. Figures from someone else's setup will not transfer, and your input-length distribution moves the numbers substantially.
Self-hosted versus a hosted API
The data-control argument that applies to embeddings applies here, and more sharply. Embedding sends your documents to a provider once, at ingest. Reranking sends both the query and the full text of every candidate document, on every query, for the life of the system. Queries are often the most sensitive artefact in the pipeline — they carry intent, and in an internal tool they carry who wanted to know what. For a regulated or confidential corpus that is a meaningful, continuous disclosure, and it belongs in the decision alongside latency and cost.
Measuring whether it helped
Evaluate reranking as a retrieval change, in isolation, before it touches generation.
Build a labelled set of real queries from your own logs — not synthetic ones — with the known-correct chunks marked. Then measure, before and after, at the k you actually serve: recall at k (is the right chunk in what you pass to the model), precision at k (how much of what you pass is relevant), and MRR (how high the first correct result lands). Record candidate-depth recall separately, so you can tell whether a miss was the retriever's fault or the reranker's.
Do not judge this by reading final answers. Generation confounds the signal in both directions: a strong model papers over bad retrieval by answering from parametric knowledge, and a weak model mangles perfect retrieval. You will conclude the reranker did nothing when it fixed the retrieval, or that it worked when the model simply got lucky. Isolate the stage you changed.
Implementation checklist
- Confirm first-stage recall at depth 50 or 100 on a labelled query set. If it is poor, stop and fix retrieval instead.
- Try reciprocal rank fusion over your dense and sparse lists first. It is free and it may be enough.
- Add a cross-encoder over a candidate set of 50 to 100, keeping 3 to 10. Treat those as starting points.
- Batch the shortlist into as few forward passes as your hardware allows.
- Sweep candidate depth against recall, precision, and MRR at your serving k, and plot it against added latency.
- Decide local versus hosted on latency, cost, and the fact that hosted means shipping every query and every candidate document off-site.
- Apply recency or authority boosts after reranking, with bounded weights.
- Re-measure retrieval metrics in isolation whenever you change the embedding model, the chunker, or the reranker.