liveBook · Manning

FiDO

As explained in Retrieval Augmented Generation, The Foundational Ideas

FiDO is a Fusion-in-Decoder architecture optimized by reducing cross-attention and reallocating parameters toward the decoder.

FiDO is a Fusion-in-Decoder optimization that targets the memory-bandwidth bottleneck created by repeated decoder cross-attention over concatenated passage representations. section 5.6.1

FiDO changes the baseline architecture in two coordinated ways:

  • Sparsifies cross-attention by reducing decoder cross-attention layers and therefore accesses to the large context key-value state. section 5.6.1
  • Reallocates parameters from the encoder to the decoder, using a smaller encoder for independent passage encoding and a larger decoder for local computation. section 5.6.1
  • Moves inference closer to compute-bound behavior, with reported gains in speed and task performance. section 5.6.1

FiD versus FiDO

The comparison lists FiD-Large at 770M parameters, 51.4% NQ exact match, and roughly 0.3× relative speed, versus FiDO-Large-XXL at 1.5B parameters, 52.2% exact match, and roughly 1.2× speed. section 5.6.1

Table 5.3 Comparison of FiD and FiDO models
Model
Backbone Size
Encoder/Decoder Parameter Ratio
NQ EM (%)
Relative Inference Speed
FiD-Base
220M
~1:1
48.2
1.0x (Baseline)
FiD-Large
770M
~1:1
51.4
~0.3x
FiDO-Large-XXL
1.5B
~1:4 (Smaller Enc, Larger Dec)
52.2
~1.2x

Use the comparison table to inspect parameter allocation, exact-match performance, and relative inference speed across FiD variants. section 5.6.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains cross-attention sparsification, parameter reallocation, and hardware effects, but omits training procedures and implementation details.
Examples
Compares FiD-Large and FiDO-Large-XXL using parameter counts, NQ exact match, and relative inference speed.
What it assumes
Builds on standard FiD and assumes familiarity with encoders, decoders, passage representations, and cross-attention.
What you can do after
Evaluate FiD variants by balancing decoder capacity, cross-attention frequency, accuracy, and inference speed.
How it connects
Connects Fusion-in-Decoder, encoder-decoder architectures, cross-attention, memory bandwidth, and compute-bound inference.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.