A cross-encoder scores a query and a passage together to judge relevance more carefully than fast first-stage retrieval. It is useful as a second pass that improves precision before results are shown or passed to a generator.
Whilst re-ranking raises top-result quality, in practice it adds latency and can discard the only good hit if the cutoff is wrong. For example, fifty candidates are scored and the correct policy sits at rank twelve of a top-five cut. We often recommend re-ranking a bounded shortlist and validating thresholds on real query sets, not maxing model size by default.
Bi-encoders embed query and document separately for speed; cross-encoders read them together and score relevance — slower, sharper. Re-ranking turns a high-recall shortlist into a high-precision top-n before generation. Use when hybrid search still surfaces near-misses or fine phrasing discrimination matters. Budget latency; monitor score distributions and cutoffs — a bad threshold drops the only useful passage. Add light diversity rules so the top n is not five paraphrases of one weak doc.