A vision encoder is a necessary component for allowing many leading LLMs to be able to work with images uploaded by users.

LLaVA 1.5 improves upon the original by connecting the language model and vision encoder through a multi-layer perceptron (MLP), a simple deep learning model where all neurons are fully connected.

It employs a vision transformer encoder alongside a large language model (LLM). The vision encoder converts images into tokens, which an attention-based extractor then aligns with the LLM.

According to Hugging Face, the 256M model, with just 256 million parameters ... The models feature a reduced-size vision encoder with 93 million parameters, replacing the previously used SigLIP ...