liveBook · Manning

Retrieval layer

As explained in LLM Customization and Fine-Tuning

The supporting systems required to perform retrieval for RAG, such as indexing and vector storage.

A retrieval layer supplies the systems that let RAG find relevant material in a corpus: an embedder, vector index, chunking pipeline, and one-time corpus encoding. At runtime, retrieval adds context to the LLM call without changing the model. section 4.7.2

Treat retrieval as a cost and latency tradeoff. It has upfront infrastructure costs, then adds a lookup and context tokens to each request. Top-3 to top-5 retrieval of 400-word chunks adds roughly 1,500 to 2,500 input tokens; results vary by query, so this context cannot be cached like a fixed example bank. RAG remains useful for knowledge-bound tasks because fine-tuning does not supply missing facts. section 4.7.2

Most teams buy managed components rather than build the full stack. Compare options against where data and identity already live, the reliability of the source corpus, and the latency budget. Start with an available cloud-native vector option; retrieval infrastructure costs the same whether generation is hosted or self-hosted. section 4.7.2 section 4.8

Table 4.5 Cost framing across techniques
Technique
Fixed cost
Per-request cost
When to choose
Zero-shot prompting
None
Low (query plus output tokens only)
Low volume, experimental, quality bar is forgiving
Few-shot prompting (3 to 8 shots)
None
Low to medium (prompt plus examples plus query)
Low-to-medium-volume, stable tasks, and quick iteration matter more than per-request cost.
Many-shot prompting (50 to 200 shots)
None
High (long input on every request)
Medium volume, high input variability, edge cases matter, long context available.
RAG
Low (embedding model, vector index setup)
Medium to high (embedding plus retrieval plus 1,500 to 2,500 extra input tokens, not cacheable)
Knowledge gap, source corpus is reliable, latency budget allows the lookup
Fine-tuning (LoRA or SFT)
High (training compute, dataset prep, validation)
Low (short prompt, no example bank)
Medium to high volume, stable task, behavior gap, cost matters

Compare retrieval’s fixed and per-request costs with prompting and fine-tuning. section 4.7.2

Table 4.7 Managed platform options for prompts and RAG
Platform
What you get
Best for
Microsoft Foundry
Prompt management, evaluation pipelines, Azure-hosted endpoints, Azure OpenAI access (Prompt Flow, its earlier flow-authoring tool, is being retired; new work should use Microsoft Agent Framework)
Microsoft-aligned shops, regulated industries
AWS Bedrock + Knowledge Bases
Pay-per-call frontier models, managed RAG with OpenSearch or Pinecone backends.
AWS-aligned shops, teams already on SageMaker
Vertex AI prompt design + Vertex AI Search
Prompt iteration, evaluation, Gemini, and partner-model integration
Teams on BigQuery or other GCP data services
Databricks Vector Search
Vector index built on Unity Catalog, Mosaic AI Gateway for guardrails
Data-platform-first organizations
LangSmith
Prompt tracing, evaluation, and dataset management in the LangChain ecosystem
Teams using LangChain or LangGraph in code
PromptLayer
Prompt versioning, gradual rollout, A/B testing
Product teams that iterate on prompts frequently
Helicone
Proxy-based observability and cost tracking, framework-agnostic
Cross-framework telemetry, fast time-to-integrate
Pinecone, Weaviate, or Qdrant
Specialty vector storage with hybrid search and large-scale filtering
Retrieval needs that outgrow cloud-native search

Compare managed prompt and retrieval platforms by provider and organizational fit. section 4.8

How LLM Customization and Fine-Tuning covers it

How deep it goes
Names retrieval components and explains fixed and per-request costs, but does not detail implementation steps.
Examples
Uses top-3 to top-5 retrieval of 400-word chunks and compares cloud-managed platform options.
What you can do after
Evaluate managed retrieval options against cloud alignment, corpus reliability, cost, and latency.
How it connects
Connects RAG infrastructure to embeddings, vector databases, chunking, model hosting, and prompt management.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.