A vision encoder is a necessary component for allowing many leading LLMs to be able to work with images uploaded by users.

The core innovation lies in replacing the traditional DETR backbone with ConvNeXt, a convolutional neural network inspired by ...

OpenAI describes Whisper as an encoder-decoder transformer, a type of neural network ... an open source computer vision model that arguably ignited the recent era of rapidly progressing image ...

It employs a vision transformer encoder alongside a large language model (LLM). The vision encoder converts images into tokens, which an attention-based extractor then aligns with the LLM.