A prompt is a short text description used to guide image generation. In a text-to-image diffusion model, a text encoder turns the prompt into embeddings. The denoising model uses those embeddings while turning noise into an image, so the description helps determine what appears in the result. section 17.3
Generate an image from a description
With KerasHub’s Stable Diffusion preset, a prompt such as “A NASA astronaut riding an origami elephant in New York City” can be passed to `task.generate(prompt)`. The pretrained model handles tokenization and diffusion; training a capable text-to-image model from scratch is expensive and time-consuming. section 17.3
Two prompt roles are useful in practice:
- A positive prompt describes the image to generate. section 17.3
- A negative prompt describes something to steer away from; passing `negative_prompts: "blue color"` illustrates this control. section 17.3
For finer control, prompts can be tokenized and encoded separately from generation. The illustrated Stable Diffusion workflow produces a sequence representation and a final vector for each positive and negative prompt—four tensors in all. Interpolating positive-prompt embeddings before denoising can produce images between two descriptions, such as a friendly dog and a tentacled creature in a field of flowers. section 17.3.1
“Prompt” is not limited to text-to-image generation. In image segmentation it can be a point or box indicating the object to mask; in character-level language generation it can be text fed in first to establish the model’s starting state. section 11.3.2 section 15.1.2
