liveBook · Manning

Independent encoding architecture

As explained in Retrieval Augmented Generation, The Foundational Ideas

An independent encoding architecture encodes a query and each candidate document separately so their representations can be precomputed and compared efficiently.

Searching a large document collection is too slow if every query must be processed jointly with every document. Independent encoding separates that work: a bi-encoder turns the query and each document into representations that can be compared. Document embeddings can be computed before queries arrive, then searched with a vector index. This makes interactive retrieval practical, but fast retrieval does not by itself guarantee that the best evidence was found. section 2.3.4

Independent encoding also has a different role after retrieval. A question may need facts from several passages, while a generator reading one passage at a time produces an incomplete answer. Fusion-in-Decoder (FiD) pairs the same question separately with each retrieved passage, encodes those inputs in parallel, then lets one decoder attend to all their representations while generating an answer. Unlike indexed document embeddings, these passage representations depend on the current question; the method preserves separate reading before combining evidence. section 5.0 section 5.2

The two uses place different work offline and at query time:

  • For retrieval, precompute document embeddings; encode an incoming query and use similarity search to find candidates. Sentence-BERT-style independent encoding supports this latency-oriented design. section 2.3.4
  • For FiD, retrieve passages first. Build one input per passage containing the question, title, and passage text; run the encoder separately on each input. These encodings must be produced for that question. section 5.2
  • Reshape the separate encoder outputs into one sequence for decoder cross-attention. Fusion does not sum the representations: the decoder reads the concatenated states to select or combine evidence. section 5.2
Listing 5.3 Decoding for FiD.
def decode_answer(encoder_hidden_states, attention_mask, tokenizer, model):
 
    num_passages, seq_len, hidden_dim = encoder_hidden_states.shape
 
    fused_hidden_states = encoder_hidden_states.reshape(
        1, num_passages * seq_len, hidden_dim  #A
    )
    fused_attention_mask = attention_mask.reshape(1, num_passages * seq_len)
 
    encoder_outputs = BaseModelOutput(last_hidden_state=fused_hidden_states)
 
    generated_ids = model.generate(  #B
        encoder_outputs=encoder_outputs,
        attention_mask=fused_attention_mask,
        max_length=64  #C
    )
 
    return tokenizer.decode(generated_ids[0], skip_special_tokens=True)

The decoding function reshapes passage states and their attention mask before passing the fused states to generation. section 5.4

A fact among distractors

Asked why a project chose multi-GPU training, an example FiD implementation identifies the passage giving the six-week training-time reason despite plausible distractors. That is easier than a question requiring genuine synthesis across passages: an unfine-tuned small model instead copied a dominant passage in a revenue-growth example. Fine-tuning is needed to teach the decoder to combine evidence rather than merely favor the strongest signal. section 5.4 section 5.5

Choose the architecture for the bottleneck at hand. Precomputed independent embeddings address retrieval speed; FiD-style independent encoding addresses reading many retrieved passages without concatenating their raw text into one large input. Neither removes the need to retrieve relevant evidence. FiD also pays for fusion during generation: decoder cross-attention repeatedly reads the concatenated states, creating a memory-bandwidth and latency limit. For deep joint reasoning, a shared representation may be preferable when its cost is acceptable. section 2.3.4 section 5.6

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains both embedding precomputation and FiD's per-passage inputs, tensor reshape, decoder fusion, and Python implementation.
Examples
Uses a multi-GPU training question with distractor passages and a revenue-growth question that exposes weak cross-passage synthesis.
What it assumes
Builds on retrieved passages and encoder-decoder models while introducing FiD's input structure and forward-pass steps.
What you can do after
Apply it via Sentence-BERT-style embeddings for retrieval or a Hugging Face T5-based FiD implementation for multi-passage answering.
How it connects
Connects bi-encoders and vector search to Fusion-in-Decoder, encoder-decoder models, cross-attention, and multi-document synthesis.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.