A vision encoder is a necessary component for allowing many leading LLMs to be able to work with images uploaded by users.

The core innovation lies in replacing the traditional DETR backbone with ConvNeXt, a convolutional neural network inspired by ...