liveBook · Manning

Multilingual support

As explained in Retrieval Augmented Generation, The Foundational Ideas

A multilingual language model understands and generates natural-language text across multiple languages.

Multilingual support enables a language model and retrieval system to understand and generate text across languages. It addresses a production failure in which English-heavy embedding models miss relevant non-English documents. section 7.4 section 2.3.4

A multilingual pipeline combines a generative LLM that writes a coherent hypothetical document in the query language with a multilingual encoder that maps queries and documents into a shared vector space. HyDE uses the generated document to reduce query-document mismatch without an explicit translation step. section 7.4 section 2.3.4

Korean question retrieval

For “인간은 언제 불을 사용했는가?” (“When did humans first use fire?”), a multilingual LLM generates a Korean hypothetical answer. paraphrase-multilingual-mpnet-base-v2 embeds that answer, compares it with Korean corpus vectors, and returns the closest document. The extra generation call improves retrieval potential but adds latency and computational cost. section 7.4

Listing 7.4 HyDE’s multilingual potential.
from sentence_transformers import SentenceTransformer
import numpy as np
 
query_kr = "인간은 언제 불을 사용했는가?"  # "When did humans first use fire?"  #A
korean_hypo = "고고학적 증거에 따르면, 인류는 약 백만 년 전에 불을 통제하여 사용하기 시작하였다."  #B
 
encoder = SentenceTransformer(  #C
    'paraphrase-multilingual-mpnet-base-v2')
v_query = encoder.encode(korean_hypo)
corpus = ["불의 발견은 인간 진화의 전환점이었다.",
          "도구 사용은 초기 인류의 특징이다.",
          "고대 인류는 약 백만 년 전에 불을 사용했다."]
corpus_vecs = encoder.encode(corpus)
 
scores = np.dot(corpus_vecs, v_query)  #D
best_idx = np.argmax(scores)
print("Retrieved:", corpus[best_idx])
 

Shows Korean hypothetical-document generation, multilingual encoding, vector scoring, and retrieval in Python. section 7.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the generation-and-encoding pipeline with a Korean example, but stops short of training multilingual models.
Examples
Uses Korean fire-history retrieval and German-to-English cross-lingual retrieval as concrete examples.
What it assumes
Assumes familiarity with embeddings and retrieval, while defining multilingual generation and shared vector spaces.
What you can do after
Apply multilingual HyDE with a generative LLM and encoder such as paraphrase-multilingual-mpnet-base-v2.
How it connects
Connects multilingual support to HyDE, RAG, Sentence-BERT, multilingual embeddings, and semantic similarity.

Also explained in

Compare how 2 books explain multilingual support →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.