A vision encoder is a necessary component for allowing many leading LLMs to be able to work with images uploaded by users.

It employs a vision transformer encoder alongside a large language model (LLM). The vision encoder converts images into tokens, which an attention-based extractor then aligns with the LLM.