The State of Multimodal LLMs

What Is Multimodal

Multimodal models understand text, images, audio, and other input types simultaneously. GPT-4V’s release marked the maturation of multimodal LLMs.

Mainstream Architectures

Vision Encoder + LLM. Use ViT to extract image features, project into LLM text space via an adapter layer. LLaVA exemplifies this approach.

Native Multimodal. Gemini and GPT-4o fuse multimodal data during pre-training rather than post-hoc stitching.

The text capabilities of multimodal models are still built on the architecture introduced in Transformer Attention Mechanism Explained.