A multimodal AI system processes or combines information from multiple modalities, such as text, images, audio, and video. LLaVA demonstrates the practical shift from natural-language processing toward asking questions about an input image. section 11.4.3
Many multimodal systems connect non-text data to language through labels, embeddings, or projection layers. This allows one model to search or reason across modalities without treating each data type as an entirely separate problem. section 11.4.3 section 12.2.3
Production choices involve clear trade-offs:
- Use a model projector to connect a converted language-model component with the vision component required by LLaVA. section 11.4.3
- Expect fragile dependencies: llama.cpp multimodality support was removed and later expected to return through another implementation. section 11.4.3
- Consider shared embedding spaces for fast cross-modal search, including finding images related to an uploaded audio clip. section 12.2.3
Deploying LLaVA
A local server can combine a quantized LLaVA language-model file with a separate vision projector, then expose the result through a GUI or API. An OpenAI-compatible client can resize an image, encode it as base64, and send it with a question. section 11.4.3
$ wget https://huggingface.co/cjpais/llava-1.6-mistral-7b- gguf/resolve/main/mmproj-model-f16.gguf $ mv mmproj-model-f16.gguf ./models/llava/mmproj.gguf
Inspect the model-projector download used to restore the vision component alongside the converted LLaVA model. section 11.4.3
Current research also explores unified representations: ImageBind maps several modalities into a familiar embedding space, while OneLLM uses one multimodal encoder for eight modalities and aligns encoding through language. section 12.2.3
