A data modality is a distinct type or channel of information that can contribute to search, such as text, images, audio, video, metadata, or behavioral signals. Foundation models allow searchable content to be represented as dense vectors rather than limiting retrieval to text. section 15.4 section 15.6
A search system can encode different modalities into compatible representations and combine them as context for matching, ranking, and reasoning. Query timing, metadata, sensory input, and observed behavior can add signals beyond the document itself, although these technologies are often still integrated piecemeal. section 15.6 section 15.3.2
Common modalities in emerging search systems include:
- Text, including queries, documents, and language context. section 15.4
- Images, audio, and video, which can be searched directly or combined with text. section 15.4 section 15.6
- Metadata, sensory input, temporal context, and behavioral signals, which provide additional context for relevance. section 15.6 section 15.3.2
Combining an image and a text query
An image of a DeLorean retrieves visually similar cars. Adding the text query “superhero” shifts results toward images containing superheroes, sporty cars, and related lighting, combining intent from both modalities. section 15.3.2
Listing 15.15 Multimodal text/image vector search using CLIP
device = "cuda" if torch.cuda.is_available() else "cpu" #1
model, preprocess = clip.load("ViT-B/32", device=device) #1
def movie_search(query_embedding, limit=8): #2
collection = engine.get_collection("tmdb_with_embeddings")
request = {"query_vector": query_embedding, #3
"query_field": "image_embedding", #3
"return_fields": ["movie_id", "title", #3
"image_id", "score"], #3
"limit": limit, #3
"quantization_size": "FLOAT32"} #4
return collection.search(**request)
def encode_text(text, normalize=True): #5
text = clip.tokenize([text]).to(device) #5
text_features = model.encode_text(text) #5
embedding = text_features.tolist()[0] #5
if normalize: #5
embedding = normalize_embedding(embedding) #5
return embedding #5
def encode_image(image_file, normalize=True): #6
image = load_image(image_file) #6
inputs = preprocess(image).unsqueeze(0).to(device) #6
embedding = model.encode_image(inputs).tolist()[0] #6
if normalize: #6
embedding = normalize_embedding(embedding) #6
return embedding #6
def encode_text_and_image(text_query, image_file): #7
text_embedding = encode_text(text_query, False)
image_embedding = encode_image(image_file, False)
return numpy.average((normalize_embedding( #8
[text_embedding, image_embedding])), axis=0).tolist() #8
Inspect an implementation that normalizes cached image embeddings, stores them with movie metadata, and searches the resulting vector collection. section 15.3.2
Listing 15.16 Multimodal vector search on text and image embeddings
def search_and_display(text_query="", image_query=None):
if image_query:
if text_query:
query_embedding = encode_text_and_image(text_query, image_query)
else:
query_embedding = encode_image(image_query)
else:
query_embedding = encode_text(text_query)
display_results(movie_search(query_embedding), show_fields=False)
Inspect the query path that selects text, image, or combined text-and-image embeddings before displaying search results. section 15.3.2
