Cross-attention sparsification targets inference systems where repeated decoder access to a large concatenated context creates a memory-bandwidth bottleneck. FiDO reduces this cost by cutting decoder cross-attention layers, lowering repeated access to the context key-value state. section 5.6.1
FiDO combines sparsification with a complementary parameter-allocation change:
- Reduce cross-attention operations so the decoder relies less often on the external, memory-intensive context. section 5.6.1
- Shift parameters from encoder to decoder, retaining efficient independent encoding while expanding local decoder computation. section 5.6.1
- Treat the result as a trade-off rather than a free optimization: reported FiDO changes produced faster inference and improved task performance. section 5.6.1
Table 5.3 Comparison of FiD and FiDO models
|
Model
|
Backbone Size
|
Encoder/Decoder Parameter Ratio
|
NQ EM (%)
|
Relative Inference Speed
|
|
FiD-Base
|
220M
|
~1:1
|
48.2
|
1.0x (Baseline)
|
|
FiD-Large
|
770M
|
~1:1
|
51.4
|
~0.3x
|
|
FiDO-Large-XXL
|
1.5B
|
~1:4 (Smaller Enc, Larger Dec)
|
52.2
|
~1.2x
|
Compare FiD and FiDO on model size, encoder-decoder parameter ratio, exact match, and relative inference speed. section 5.6.1
