A retrieval-augmented generation (RAG) system can fail because it retrieves the wrong document or because it fails to use the right one. Hit@1 measures the first problem: the proportion of queries whose top-ranked chunk comes from a relevant document. section 4.5.7
For each labeled query, assign 1 if the first result matches a relevant document and 0 otherwise; average those scores across queries. Hit@1 is the strictest case of Hit@k, which counts a hit if any relevant document appears among the top k results. Hit@1 matters especially without reranking, when the first chunk is seen and weighed most heavily by the model. section 4.5.7
Listing 4.5 RAG retrieval evaluation (listing_4_5_rag_eval.py)
# code/chapter04/scripts/listing_4_5_rag_eval.py (key sections -- full source in repo)
def precision_at_k(retrieved, expected_doc_id):
if not retrieved:
return 0.0
hits = sum(1 for item in retrieved if item["metadata"]["id"] == expected_doc_id)
return hits / len(retrieved)
def hit_at_k(retrieved, expected_doc_id):
return 1.0 if any(item["metadata"]["id"] == expected_doc_id
for item in retrieved) else 0.0
def hit_at_1(retrieved, expected_doc_id):
return hit_at_k(retrieved[:1], expected_doc_id)
The evaluation listing implements Hit@1 by applying Hit@k to the first retrieved result and matching its document ID against the expected ID. section 4.5.7
Read the metric alongside other quality measures:
- Precision@k measures the fraction of top-k chunks from relevant documents; a hit only requires one relevant result. section 4.5.7
- Recall@k measures retrieved relevant documents against all relevant documents. Check benchmark definitions before comparing scores. section 4.5.7
- Answer faithfulness and end-to-end accuracy test what happens after retrieval. A correct first document does not establish that the answer used it correctly. section 4.5.8
Policy-document evaluation
On eight internal policy documents and ten labeled queries, mean Hit@1 was 1.000: the expected document ranked first for every query. Mean Hit@3 was also 1.000, while mean Precision@3 was 0.433. The results distinguish finding the right document from filling the top three with relevant chunks. section 4.5.7
For a broader retrieval check, evaluate 50–100 human-labeled queries at the k used by the pipeline. A Hit@k below 0.7 signals a retrieval bottleneck; consider changing chunking, using a stronger embedder, adding reranking, or adding hybrid retrieval before tuning the prompt. This threshold applies to Hit@k, not specifically to Hit@1. section 4.5.8
