A retrieval layer supplies the systems that let RAG find relevant material in a corpus: an embedder, vector index, chunking pipeline, and one-time corpus encoding. At runtime, retrieval adds context to the LLM call without changing the model. section 4.7.2
Treat retrieval as a cost and latency tradeoff. It has upfront infrastructure costs, then adds a lookup and context tokens to each request. Top-3 to top-5 retrieval of 400-word chunks adds roughly 1,500 to 2,500 input tokens; results vary by query, so this context cannot be cached like a fixed example bank. RAG remains useful for knowledge-bound tasks because fine-tuning does not supply missing facts. section 4.7.2
Most teams buy managed components rather than build the full stack. Compare options against where data and identity already live, the reliability of the source corpus, and the latency budget. Start with an available cloud-native vector option; retrieval infrastructure costs the same whether generation is hosted or self-hosted. section 4.7.2 section 4.8
Table 4.5 Cost framing across techniques
|
Technique
|
Fixed cost
|
Per-request cost
|
When to choose
|
|
Zero-shot prompting
|
None
|
Low (query plus output tokens only)
|
Low volume, experimental, quality bar is forgiving
|
|
Few-shot prompting (3 to 8 shots)
|
None
|
Low to medium (prompt plus examples plus query)
|
Low-to-medium-volume, stable tasks, and quick
iteration matter more than per-request cost.
|
|
Many-shot prompting (50 to 200 shots)
|
None
|
High (long input on every request)
|
Medium volume, high input variability, edge cases
matter, long context available.
|
|
RAG
|
Low (embedding model, vector index setup)
|
Medium to high (embedding plus retrieval plus 1,500
to 2,500 extra input tokens, not cacheable)
|
Knowledge gap, source corpus is reliable, latency
budget allows the lookup
|
|
Fine-tuning (LoRA or SFT)
|
High (training compute, dataset prep, validation)
|
Low (short prompt, no example bank)
|
Medium to high volume, stable task, behavior gap,
cost matters
|
Compare retrieval’s fixed and per-request costs with prompting and fine-tuning. section 4.7.2
Table 4.7 Managed platform options for prompts and RAG
|
Platform
|
What you get
|
Best for
|
|
Microsoft
Foundry
|
Prompt
management, evaluation pipelines, Azure-hosted endpoints, Azure OpenAI access
(Prompt Flow, its earlier flow-authoring tool, is being retired; new work
should use Microsoft Agent Framework)
|
Microsoft-aligned
shops, regulated industries
|
|
AWS Bedrock +
Knowledge Bases
|
Pay-per-call
frontier models, managed RAG with OpenSearch or Pinecone backends.
|
AWS-aligned
shops, teams already on SageMaker
|
|
Vertex AI
prompt design + Vertex AI Search
|
Prompt
iteration, evaluation, Gemini, and partner-model integration
|
Teams on
BigQuery or other GCP data services
|
|
Databricks
Vector Search
|
Vector index
built on Unity Catalog, Mosaic AI Gateway for guardrails
|
Data-platform-first
organizations
|
|
LangSmith
|
Prompt tracing,
evaluation, and dataset management in the LangChain ecosystem
|
Teams using
LangChain or LangGraph in code
|
|
PromptLayer
|
Prompt
versioning, gradual rollout, A/B testing
|
Product teams
that iterate on prompts frequently
|
|
Helicone
|
Proxy-based
observability and cost tracking, framework-agnostic
|
Cross-framework
telemetry, fast time-to-integrate
|
|
Pinecone,
Weaviate, or Qdrant
|
Specialty
vector storage with hybrid search and large-scale filtering
|
Retrieval needs
that outgrow cloud-native search
|
Compare managed prompt and retrieval platforms by provider and organizational fit. section 4.8
