BM25 ranks documents by query-term frequency while penalizing common words. Unlike embedding-based retrieval, it matches exact terms: a search for “iPhone 14 Pro troubleshooting” can retrieve documents containing those words. Its deterministic, fast, reproducible behavior makes it a useful retrieval baseline, but exact matching can fall short when queries need semantic search. section 3.6.6
In a telecom support chatbot, BM25 retrieval can work alongside metadata filters for plan, device, topic, or year. For a question about international roaming on an iPhone 14 Pro with the Mobile Plus plan, extracted query details can narrow the retrieval search. Deterministic results avoid model inference and embedding drift, but reproducibility alone does not guarantee relevance. section 3.6.6
from haystack import Pipeline
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
retrieval_pipeline = Pipeline()
metadata_extractor = QueryMetadataExtractor()
retriever = InMemoryBM25Retriever(document_store=document_store) #1
retrieval_pipeline.add_component(instance=metadata_extractor,
name="metadata_extractor") #2
retrieval_pipeline.add_component(instance=retriever,
name="retriever") #3
retrieval_pipeline.connect("metadata_extractor.filters",
"retriever.filters") #4
query = "How do I set up international roaming on my iPhone 14 Pro?
I'm on the Mobile Plus plan."
metadata_fields = ["topic", "plan_name", "device_type", "year"]
retrieval_pipeline.run(
data={
"metadata_extractor": {"query": query, "metadata_fields":
metadata_fields},
"retriever": {"query": query}
}
) #5
See a Haystack pipeline connecting extracted metadata filters to an InMemoryBM25Retriever. section 3.6.6
