liveBook · Manning

Orchestration layer

As explained in Retrieval Augmented Generation, The Foundational Ideas

An orchestration layer coordinates modules, interfaces, routing decisions, and execution across a larger software system.

An orchestration layer coordinates modules and routes requests across a larger system. In modular RAG, it lets an application switch retrieval tools or generation models at query time without disrupting the broader pipeline. section 1.5.3

Design choices shape how easily the system can evolve:

  • Standardized interfaces let modules be added and third-party components integrated without redesigning the system. section 1.5.3
  • Version management supports gradual upgrades and testing; components can also scale independently or be iterated on without changing the whole pipeline. section 1.5.3
  • Whether implemented as microservices or a modular monolith, the main engineering investment is defining interfaces and routing logic. section 1.5.3

Prompted Self-RAG

A workflow orchestration engine such as LangGraph can model critique steps as graph nodes. Each node prompts a standard LLM to produce a check, keeping reflection outside the model’s weights. This approach is simpler to set up than fine-tuning, but its separate API calls add latency and serving cost. section 9.4

Table 9.2 Trained Self-RAG versus Prompted Self-RAG
Dimension
Trained Self-RAG (the paper)
Prompted Self-RAG (this section)
Model
Llama-2 7B/13B fine-tuned with custom tokens
Any frontier chat model, off-the-shelf, no weight changes
Setup cost
~150K critic-annotated training examples + GPU fine-tuning
A few prompts and an orchestration graph
Where the critique lives
Inside the model weights (token probabilities)
Outside the model (separate LLM calls per step)
Latency per answer
One forward pass, multiple decoded tokens
Several round-trips to the LLM (one per critique step)
Cost per answer
Local inference, no API per critique
API tokens per critique call (Retrieve, IsRel, IsSup, IsUse)
Tunability
Decoding-time weights w_T over predicted token probs
Prompt wording, temperature, and branching thresholds in the graph
Reproducing on new domains
Re-fine-tune with new critic-annotated data
Edit prompts, re-run
When to choose
High-volume traffic where API cost dominates, or strict on-device deployment
Most practical applications, prototyping, and regulated domains where you can't fine-tune

Compare prompted and trained Self-RAG to see how placing critique in an orchestration graph trades setup effort for additional inference calls. section 9.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines interfaces, routing logic, and upgrade practices, then points to adaptive-system examples for further detail.
Examples
Uses LangGraph nodes for Self-RAG critique calls and contrasts prompted orchestration with trained-model inference.
What you can do after
Build it from scratch by defining interfaces and routing logic that switch tools or models at query time.
How it connects
Connects modular RAG with retrieval, generation models, orchestration frameworks, and Self-RAG.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.