Vision-Language Models (VLMs) provide state-of-the-art performance across a spectrum of vision-language tasks, including captioning, object localization, commonsense reasoning, and vision-based coding, amongst others. Recent studies, such as one undertaken by Apple, showed that these models excel in extracting text from images and interpreting visual data, including tables and charts. However, when tested on complex tasks demanding advanced vision-based deductive reasoning, these VLMs were found to have limitations.
To assess their capabilities, the Apple team used Raven’s Progressive Matrices (RPMs), which are renowned for evaluating an individual’s multi-hop relational and deductive reasoning skills using visual cues only. The team used techniques like in-context learning, self-consistency, and Chain-of-thoughts (CoT) to methodically evaluate these models on three different datasets: Mensa IQ exam, IntelligenceTest, and RAVEN.
While the models did well on text-based reasoning tasks, they were not as successful at visual deductive reasoning. Some tactics that worked to improve Large Language Models’ (LLMs) performance did not translate well to visual reasoning problems. According to the research, the primary issue lies in the models’ difficulty to understand and identify abstract patterns contained in RPM samples.
Despite this, the Apple team made several key contributions in this field. They developed a systematic approach to evaluate the Vision-Language Models using RPMs. This approach used the Mensa IQ exam, IntelligenceTest, and RAVEN datasets, providing comprehensive insight into the performance of VLMs in image-based reasoning tasks.
The team also applied common LLM inference-time techniques, such as self-consistency and in-context learning, to VLMs. They found that some tactics that were effective for LLMs were not as successful for VLMs. Detailed performance analysis revealed that perception was primarily where the current VLMs struggled, with specific problems identified in a case study using GPT-4V.
The researchers discovered several issues in the existing operation of VLMs, including overconfidence, sensitivity to prompt design, and an inability to effectively use in-context examples. The influence of prompts on model performance was scrutinized, and structured prompts surfaced as a potential tactic for enhancing effectiveness.
In conclusion, while the Vision-Language Models have made significant progress in many areas, there are still critical areas where improvement is required, particularly when it comes to complex visual reasoning tasks. This research sheds light on the potential and limitations of these models, providing a valuable resource for future improvements to AI capabilities in visual reasoning.
