Suppose someone describes a scene but wants to find an image, not a document containing those words. Cross-modal search retrieves one content type using a query in another: a text query can retrieve images, and an uploaded image can be used to search text. It is a form of multimodal search; searching images with another image uses the same machinery but is not itself cross-modal. section 15.1 section 15.3
The mechanism is a shared embedding space. A model trained on paired images and captions maps related text and images to nearby vectors. Index image embeddings, encode an incoming text query with the same model, then search the image-vector field. Once the content is represented in overlapping vector spaces, retrieval uses vector search rather than a separate cross-modal index. section 15.3.2 section 15.3
Find movie images from a description
A movie-image collection stores normalized image embeddings alongside titles and image IDs. With CLIP, a query such as “singing in the rain” becomes a normalized text embedding; a vector search against the image_embedding field returns ranked movie images. The query need not match stored movie text, because the search is against image vectors. section 15.3.2
Listing 15.15 Multimodal text/image vector search using CLIP
device = "cuda" if torch.cuda.is_available() else "cpu" #1
model, preprocess = clip.load("ViT-B/32", device=device) #1
def movie_search(query_embedding, limit=8): #2
collection = engine.get_collection("tmdb_with_embeddings")
request = {"query_vector": query_embedding, #3
"query_field": "image_embedding", #3
"return_fields": ["movie_id", "title", #3
"image_id", "score"], #3
"limit": limit, #3
"quantization_size": "FLOAT32"} #4
return collection.search(**request)
def encode_text(text, normalize=True): #5
text = clip.tokenize([text]).to(device) #5
text_features = model.encode_text(text) #5
embedding = text_features.tolist()[0] #5
if normalize: #5
embedding = normalize_embedding(embedding) #5
return embedding #5
def encode_image(image_file, normalize=True): #6
image = load_image(image_file) #6
inputs = preprocess(image).unsqueeze(0).to(device) #6
embedding = model.encode_image(inputs).tolist()[0] #6
if normalize: #6
embedding = normalize_embedding(embedding) #6
return embedding #6
def encode_text_and_image(text_query, image_file): #7
text_embedding = encode_text(text_query, False)
image_embedding = encode_image(image_file, False)
return numpy.average((normalize_embedding( #8
[text_embedding, image_embedding])), axis=0).tolist() #8
The Python listing shows the key handoff: CLIP encodes the text, and movie_search sends its vector to the image_embedding field with a result limit. section 15.3.2
The same collection also accepts image queries. A combined text-and-image query can seek results matching both signals—for example, “superhero” paired with an image of a car. Query wording can matter: the illustrated results distinguish “superhero flying” from “superheroes flying.” Inspect returned images rather than assuming the model will interpret every phrasing as intended. section 15.3.2
Listing 15.16 Multimodal vector search on text and image embeddings
def search_and_display(text_query="", image_query=None):
if image_query:
if text_query:
query_embedding = encode_text_and_image(text_query, image_query)
else:
query_embedding = encode_image(image_query)
else:
query_embedding = encode_text(text_query)
display_results(movie_search(query_embedding), show_fields=False)
This small dispatcher shows how one search interface routes text-only, image-only, and combined inputs to the appropriate query encoder before searching. section 15.3.2
