Large Vision-Language Models (VLMs) have shown remarkable abilities to perform a wide range of tasks by utilizing language thinking. One way to improve these models’ performance is by fine-tuning them with specified visual instruction data, enabling them to follow precise visual directions. However, this approach relies heavily on supervised learning from pre-collected data and isn’t optimal for training agents in interactive environments requiring both language comprehension and visual recognition. The pre-gathered datasets may lack the necessary diversity needed for complex decision-making scenarios encountered by these agents.
An alternative solution is offered by Reinforcement Learning (RL), which holds potential for VLMs’ decision-making ability in complex situations. Although RL has been successful in training agents for various text-based tasks, it has not been extensively used for enhancing VLMs for tasks imbued with language and visual processing.
Addressing this gap, a group of researchers have developed an algorithmic framework, leveraging RL to optimize VLMs. In this framework, the task description given to the VLM initiates “Chain Of Thought” (CoT) reasoning in the model, allowing it to understand the reasoning steps leading to the required text-based action for task completion. The text output generated by the VLM is translated into executable actions through which the agent interacts with its environment. Depending on the effectiveness of their actions in accomplishing the task objectives, the agents are then rewarded. These rewards inform the RL fine-tuning process, thus enhancing the VLMs’ decision-making competency.
Empirical findings from this approach’s tests demonstrate significant improvements in VLM agents’ performance in decision-making tasks. Notably, a 7-billion parameter model bested widely-used commercial models like GPT-4V and Gemini using this method. The researchers noted that these performance gains were only achievable with the CoT reasoning element. When this element was omitted, the model’s performance experienced a notable decline, underscoring the crucial function of CoT reasoning in the RL training framework and its significant role in boosting VLMs’ decision-making abilities.
All the research credit is attributed to the researchers involved in this project. For more in-depth information, the research paper and project can be accessed via their platforms.
