A vision encoder is a necessary component for allowing many leading LLMs to be able to work with images uploaded by users.

LLaVA 1.5 improves upon the original by connecting the language model and vision encoder through a multi-layer perceptron (MLP), a simple deep learning model where all neurons are fully connected.

Google claims that Gemma 3 is the "world's best single-accelerator model," outperforming competitors ... Gemma 3 packs an upgraded vision encoder that handles high-res and non-square images ...

It employs a vision transformer encoder alongside a large language model (LLM). The vision encoder converts images into tokens, which an attention-based extractor then aligns with the LLM.