A retrieved chunk may answer a question yet lose the context that makes the answer usable: which document it came from, what section it belongs to, or whether the policy is still active. Metadata keeps that descriptive information attached to the chunk. Filenames, titles, section names, and domain keywords can restore context; policy IDs, departments, effective dates, and status can also constrain which results users see. section 3.7.2 section 4.5.4
Choose where metadata enters the retrieval workflow:
- Preserve it during chunking. Carry document fields forward and extract section titles so each chunk remains identifiable. section 4.5.4
- Use it in retrieval. Metadata fields can be incorporated into embedding creation or used to filter results after initial retrieval. In a vector database, stored fields can also support filters supplied with a query. section 3.7.2 section 4.5.4
- Budget for extraction. Automated keyword extraction with RAKE is an option, but adds implementation effort. section 3.7.2
vectors_to_upsert = [] #1
for i, chunk in enumerate(all_chunks):
vector_data = { #2
"id": chunk["chunk_id"],
"values": embeddings[i].tolist(),
"metadata": {
"content": chunk["content"],
"section_title": chunk["section_title"],
"policy_id": chunk["policy_id"],
"department": chunk["department"],
"effective_date": chunk["effective_date"],
"status": chunk["status"]
}
}
vectors_to_upsert.append(vector_data)
index.upsert(vectors=vectors_to_upsert) #3
index_stats = index.describe_index_stats() #4
print(f"Total vectors in index: {index_stats['total_vector_count']}")
The Pinecone upsert stores each chunk’s content and section title alongside its policy ID, department, effective date, and status. Those fields remain available with the vector rather than being discarded after embedding. section 4.5.4
def search_policies(query, filters=None, top_k=3):
"""
Search policy documents with semantic search and optional filters
Args:
query: User question or query string
filters: Dictionary of metadata filters to apply
top_k: Number of results to return
Returns:
List of relevant policy chunks with metadata
"""
query_embedding = model.encode([query])[0].tolist() #1
filter_dict = None #2
if filters: #2
filter_dict = {} #2
for key, value in filters.items(): #2
filter_dict[key] = value #2
query_results = index.query( #3
vector=query_embedding, #3
filter=filter_dict, #3
top_k=top_k, #3
include_metadata=True #3
)
results = [] #4
for match in query_results['matches']: #4
results.append({ #4
"score": match['score'],
"content": match['metadata']['content'],
"section_title": match['metadata']['section_title'],
"policy_id": match['metadata']['policy_id'],
"department": match['metadata']['department'],
"effective_date": match['metadata']['effective_date']
})
return results
query = "What's covered for physical therapy after knee surgery?" #5
filters = {"status": "Active", "department": "Benefits"} #5
results = search_policies(query, filters) #5
for i, result in enumerate(results): #6
print(f"\nResult {i+1} (Score: {result['score']:.4f})")
print(f"Policy: {result['policy_id']} ({result['department']})")
print(f"Section: {result['section_title']}")
print(f"Content: {result['content'][:200]}...")
The search function accepts optional metadata filters, passes them to Pinecone with the query vector, and requests metadata in the returned matches. section 4.5.4
Current policy lookup
For a health-care benefits question, a semantically similar excerpt is not enough if it belongs to the wrong policy or an outdated one. Retaining policy ID, effective date, and status lets the search narrow results and identify the excerpts used to answer. section 4.5.4
