A data modality is a type of input or output data, such as text, images, audio, or structured data. For sequence models, the practical challenge is to represent each modality as a sequence that a Transformer can process. section 16.4.2
Turning an image into sequence data
Gemma handles an image with a separate 420-million-parameter image encoder. The encoder divides the image into 256 patches and converts them into a (256, 2560) sequence, matching the Transformer's hidden dimension. That sequence can then be spliced into the text sequence after token embedding, functioning like 256 image-specific soft tokens. section 16.4.2
A useful implementation pattern is:
- Choose an encoder suited to the modality, such as an image encoder for visual input. section 16.4.2
- Convert the encoder output into vectors with the same dimensionality as the Transformer hidden states. section 16.4.2
- Insert those vectors into the model's sequence alongside text tokens, distinguishing flexible soft tokens from fixed-vocabulary hard tokens. section 16.4.2
- Use a special image marker to reserve positions that are replaced by the encoded modality. section 16.4.2
>>> gemma_lm.preprocessor.max_images_per_prompt = 1 #1 >>> gemma_lm.preprocessor.sequence_length = 512 #1 >>> prompt = "What is going on in this image? Be concise!<start_of_image>" >>> gemma_lm.generate({ ... "prompts": PROMPT_TEMPLATE.format(prompt), ... "images": [image], ... }) <start_of_turn>user What is going on in this image? Be concise! <start_of_image> <end_of_turn> <start_of_turn>model A snake wearing glasses is sitting in a leather armchair, surrounded by a large bookshelf, and reading a book. It's a whimsical, slightly surreal image. <end_of_turn> >>> prompt = "What is the snake wearing?<start_of_image>" >>> gemma_lm.generate({ ... "prompts": PROMPT_TEMPLATE.format(prompt), ... "images": [image], ... }) <start_of_turn>user What is the snake wearing? <start_of_image> <end_of_turn> <start_of_turn>model The snake is wearing a pair of glasses! They are red-framed and perched on its head.<end_of_turn>
A working Gemma example supplies an image, places <start_of_image> in the prompt, and generates descriptions and answers about the visual input. section 16.4.2
