An orchestration layer coordinates modules and routes requests across a larger system. In modular RAG, it lets an application switch retrieval tools or generation models at query time without disrupting the broader pipeline. section 1.5.3
Design choices shape how easily the system can evolve:
- Standardized interfaces let modules be added and third-party components integrated without redesigning the system. section 1.5.3
- Version management supports gradual upgrades and testing; components can also scale independently or be iterated on without changing the whole pipeline. section 1.5.3
- Whether implemented as microservices or a modular monolith, the main engineering investment is defining interfaces and routing logic. section 1.5.3
Prompted Self-RAG
A workflow orchestration engine such as LangGraph can model critique steps as graph nodes. Each node prompts a standard LLM to produce a check, keeping reflection outside the model’s weights. This approach is simpler to set up than fine-tuning, but its separate API calls add latency and serving cost. section 9.4
Table 9.2 Trained Self-RAG versus Prompted Self-RAG
|
Dimension
|
Trained Self-RAG (the paper)
|
Prompted Self-RAG (this section)
|
|
Model
|
Llama-2 7B/13B fine-tuned with custom tokens
|
Any frontier chat model, off-the-shelf, no weight changes
|
|
Setup cost
|
~150K critic-annotated training examples + GPU fine-tuning
|
A few prompts and an orchestration graph
|
|
Where the critique lives
|
Inside the model weights (token probabilities)
|
Outside the model (separate LLM calls per step)
|
|
Latency per answer
|
One forward pass, multiple decoded tokens
|
Several round-trips to the LLM (one per critique step)
|
|
Cost per answer
|
Local inference, no API per critique
|
API tokens per critique call (Retrieve, IsRel, IsSup,
IsUse)
|
|
Tunability
|
Decoding-time weights w_T over predicted token probs
|
Prompt wording, temperature, and branching thresholds in
the graph
|
|
Reproducing on new domains
|
Re-fine-tune with new critic-annotated data
|
Edit prompts, re-run
|
|
When to choose
|
High-volume traffic where API cost dominates, or strict
on-device deployment
|
Most practical applications, prototyping, and regulated
domains where you can't fine-tune
|
Compare prompted and trained Self-RAG to see how placing critique in an orchestration graph trades setup effort for additional inference calls. section 9.4
