liveBook · Manning

Hit@1

As explained in LLM Customization and Fine-Tuning

The proportion of queries for which at least one relevant result appears among the top k retrieved results.

A retrieval-augmented generation (RAG) system can fail because it retrieves the wrong document or because it fails to use the right one. Hit@1 measures the first problem: the proportion of queries whose top-ranked chunk comes from a relevant document. section 4.5.7

For each labeled query, assign 1 if the first result matches a relevant document and 0 otherwise; average those scores across queries. Hit@1 is the strictest case of Hit@k, which counts a hit if any relevant document appears among the top k results. Hit@1 matters especially without reranking, when the first chunk is seen and weighed most heavily by the model. section 4.5.7

Listing 4.5 RAG retrieval evaluation (listing_4_5_rag_eval.py)
# code/chapter04/scripts/listing_4_5_rag_eval.py (key sections -- full source in repo)
 
def precision_at_k(retrieved, expected_doc_id):
    if not retrieved:
        return 0.0
    hits = sum(1 for item in retrieved if item["metadata"]["id"] == expected_doc_id)
    return hits / len(retrieved)
 
def hit_at_k(retrieved, expected_doc_id):
    return 1.0 if any(item["metadata"]["id"] == expected_doc_id
                      for item in retrieved) else 0.0
 
def hit_at_1(retrieved, expected_doc_id):
    return hit_at_k(retrieved[:1], expected_doc_id)
 

The evaluation listing implements Hit@1 by applying Hit@k to the first retrieved result and matching its document ID against the expected ID. section 4.5.7

Read the metric alongside other quality measures:

  • Precision@k measures the fraction of top-k chunks from relevant documents; a hit only requires one relevant result. section 4.5.7
  • Recall@k measures retrieved relevant documents against all relevant documents. Check benchmark definitions before comparing scores. section 4.5.7
  • Answer faithfulness and end-to-end accuracy test what happens after retrieval. A correct first document does not establish that the answer used it correctly. section 4.5.8

Policy-document evaluation

On eight internal policy documents and ten labeled queries, mean Hit@1 was 1.000: the expected document ranked first for every query. Mean Hit@3 was also 1.000, while mean Precision@3 was 0.433. The results distinguish finding the right document from filling the top three with relevant chunks. section 4.5.7

For a broader retrieval check, evaluate 50–100 human-labeled queries at the k used by the pipeline. A Hit@k below 0.7 signals a retrieval bottleneck; consider changing chunking, using a stronger embedder, adding reranking, or adding hybrid retrieval before tuning the prompt. This threshold applies to Hit@k, not specifically to Hit@1. section 4.5.8

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines the per-query score and average, implements document-ID matching, and compares Hit@1 with precision and recall.
Examples
Reports Hit@1 of 1.000 on ten labeled queries against eight internal policy documents.
What you can do after
Evaluate retrieval rankings against labeled document IDs and distinguish retrieval problems from synthesis problems.
How it connects
Connects Hit@1 to Hit@k, Precision@k, Recall@k, answer faithfulness, and end-to-end accuracy.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.