liveBook · Manning

Cross-attention sparsification

As explained in Retrieval Augmented Generation, The Foundational Ideas

Cross-attention sparsification reduces the number of decoder cross-attention layers or operations to lower memory access costs.

Cross-attention sparsification targets inference systems where repeated decoder access to a large concatenated context creates a memory-bandwidth bottleneck. FiDO reduces this cost by cutting decoder cross-attention layers, lowering repeated access to the context key-value state. section 5.6.1

FiDO combines sparsification with a complementary parameter-allocation change:

  • Reduce cross-attention operations so the decoder relies less often on the external, memory-intensive context. section 5.6.1
  • Shift parameters from encoder to decoder, retaining efficient independent encoding while expanding local decoder computation. section 5.6.1
  • Treat the result as a trade-off rather than a free optimization: reported FiDO changes produced faster inference and improved task performance. section 5.6.1
Table 5.3 Comparison of FiD and FiDO models
Model
Backbone Size
Encoder/Decoder Parameter Ratio
NQ EM (%)
Relative Inference Speed
FiD-Base
220M
~1:1
48.2
1.0x (Baseline)
FiD-Large
770M
~1:1
51.4
~0.3x
FiDO-Large-XXL
1.5B
~1:4 (Smaller Enc, Larger Dec)
52.2
~1.2x

Compare FiD and FiDO on model size, encoder-decoder parameter ratio, exact match, and relative inference speed. section 5.6.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains layer reduction, parameter reallocation, and reported speed and accuracy effects.
Examples
Uses FiDO and a FiD-versus-FiDO comparison of size, parameter ratios, exact match, and inference speed.
What it assumes
Assumes familiarity with encoder-decoder models, decoder cross-attention, and concatenated context states.
What you can do after
Evaluate decoder attention reductions against inference speed, accuracy, and memory-access trade-offs.
How it connects
Connects cross-attention with Fusion-in-Decoder, encoder-decoder design, memory bandwidth, and inference optimization.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.