A team of researchers from Hugging Face and Sorbonne Université has conducted in-depth studies on vision-language models (VLMs), aiming to better understand the critical factors that impact their performance. These models, capable of processing both images and text, have become popular in a variety of areas, such as information retrieval in scanned documents to code generation from screenshots. However, their advancement has been slowed due to a lack of clarity on the design choices that most significantly affect their performance.
The researchers investigated different model architectures, including cross-attention and fully autoregressive architectures. They also studied the impact of using pre-trained backbones for the vision and language parts of the models. Their findings show that the quality of the language model backbone has a larger impact on VLM performance than the vision backbone. Specifically, if a lower-quality language model was replaced with a superior one, the performance improvement was more substantial than upgrading the vision encoder.
Moreover, the team explored multimodal training procedures, such as the learned pooling methodology, which reduces the number of visual tokens needed for each image, thus reducing computational costs. The researchers also analyzed strategies to preserve the original image’s aspect ratio and resolution, which resulted in flexible computation during training and inference without negatively affecting performance.
In order to evaluate the effectiveness of their findings, the researchers developed and trained Idefics2, an open-source VLM with 8 billion parameters. The model was trained on diverse data sources to improve its ability to process various multimodal inputs. Rigorous evaluations using benchmark datasets revealed that Idefics2 outperformed current state-of-the-art VLMs, exhibiting performance comparable to models four times larger.
The researchers recongnized that their research is just one step forward in understanding VLM development, and that there are likely additional areas for improvement. They provided the training dataset, The Cauldron, a collection of 50 vision-language datasets, and open-sourced their model and research findings to support further investigation and advancements in vision-language modeling.
