A prompt tells a model what to act on. In image segmentation, it identifies the object whose mask should be produced. The prompt accompanies the image; it is not the mask itself. section 11.4.2
For Segment Anything, a prompt can be:
- A point inside the object to mask. section 11.4.2
- A box around the object to mask. section 11.4.2
Segment Anything is trained on (image, prompt, mask) triples. An image encoder represents the image, a prompt encoder represents the point or box, and a mask decoder uses both representations to predict possible masks. Training compares predictions with a ground-truth mask. The prompt-encoder and mask-decoder internals are not needed to use this high-level picture. section 11.4.2
Figure 11.9 The SAM high-level architecture overview
The architecture diagram shows where the image and prompt enter and how their representations feed mask prediction. section 11.4.2
A prompt can take a different form for a different job. In text-to-image generation, it is a short description of the desired image. A text encoder turns that description into embeddings that guide a diffusion model as it denoises an image. A negative prompt can steer generation away from an unwanted feature. section 17.3 section 17.3.1
Generate an image from a text prompt
In R, load KerasHub’s pretrained Stable Diffusion 3 model, then pass a description to generate(): library(keras3) py_require("keras-hub==0.21.*") keras_hub <- import("keras_hub") task <- keras_hub$models$TextToImage$from_preset( "stable_diffusion_3_medium", image_shape = shape(512, 512, 3), dtype = "float16" ) prompt <- "A NASA astronaut riding an origami elephant in New York City" image <- task$generate(prompt) plot(as.raster(image, max = max(image))) Here the prompt describes an image to create, rather than selecting an object to mask. section 17.3
In character-by-character language generation, a prompt instead supplies starting text. Feeding its characters into a GRU primes the model’s state before it predicts new characters. Across these uses, the prompt supplies context or direction, while the model produces the output. section 15.1.2
