Large Vision-Language Models (VLMs) have proven to have impressive capacities as adaptable agents who are able to solve many tasks. They can be optimized through fine-tuning with specific visual instruction-following data, thus enhancing their performance. However, this strategy can be limited as it mostly relies on supervised learning from pre-collected data. Consequently, it may not be the best approach for training agents in multi-step interactive environments that require both language understanding and visual recognition.
Reinforcement Learning (RL) offers an alternative solution, and has been successful in training agents for various text-based tasks. However, it has rarely been used to improve vector language models (VLMs) for tasks requiring end-to-end language and visual processing.
Recognising this, a group of researchers have designed an algorithmic framework that employs RL to optimize VLMs. The framework first provides the task description to the VLM, provoking the model to produce Chain-Of-Thought (CoT) reasoning. This stage is significant as it allows the VLM to inspect intermediate stages in reasoning that logically direct to the final text-based action necessary to complete the task.
The text output generated by the VLM is then converted into executable actions, enabling the agent to interact with its environment. The agent is subsequently rewarded based on how successfully their actions attain the task objectives. These rewards are used to further fine-tune the VLM through RL, thereby enhancing their decision-making abilities.
Empirical results from tests demonstrate that this paradigm significantly improves the performance of VLM agents in decision-making tasks. For instance, through this approach, a 7-billion parameter model outperformed popular commercial models such as GPT-4V and Gemini. The research team discovered that these performance advantages were only achievable with the CoT reasoning component. Without this component, the model’s overall performance significantly declined. It shows the importance of CoT reasoning within the RL training framework and its pivotal role in advancing VLMs’ decision-making capacities.
All credit goes to the researchers from UC Berkeley, UIUC, and NYU who developed this algorithmic framework that uses reinforcement learning to optimize vision-language models. They have shared their paper and project for further exploration and understanding. Their work underscores the importance of improvement in artificial intelligence, and the potential of reinforcement learning to make more sophisticated models. They encourage those interested in their work to keep up to date with their latest developments via social media and their newsletter.
