Large Vision Language Models (LVLMs) have been successful in text and image comprehension tasks, including Referring Expression Comprehension (REC). Notably, models like Griffon have made significant progress in areas such as object detection, denoting a key improvement in perception within LVLMs. Unfortunately, known challenges with LVLMs include their inability to match task-specific experts in intricate situations due to an image resolution constraint. This constraint has limited LVLMs’ efficacy in tasks requiring both textual and visual referential cues.
Against this backdrop, a team of researchers unveiled Griffon v2, a high-resolution model that can flexibly refer to objects through both visual and textual cues. An important innovation in Griffon v2 is a lightweight downsampling projector designed to surmount the constraints imposed on the number of input tokens Large Language Models can handle. In doing so, Griffon v2 surpasses limitations of other LVLMs by retaining fine features and entire contexts, improving multimodal perception, especially for elements that lower-resolution models might overlook.
Further, the researchers augmented Griffon v2 with visual-language co-referring capabilities through a plug-and-play visual tokenizer. This feature makes Griffon v2 user-friendly, allowing it to interact with a variety of inputs such as coordinates, free-form text, and flexible target pictures.
The efficacy of Griffon v2 is evident in its performance in tasks like Referring Expression Generation (REG), phrase grounding, and REC, outperforming expert models in object detection and object counting. Its two primary features are its high-resolution multimodal perception model and a visual-language co-referring structure.
The multimodal perception model eliminates the need to partition images, thus enhancing localized understanding. In addition, it can handle resolutions up to 1K, improving the model’s ability to capture finer details. The co-referring structure, on the other hand, broadens the model’s use by allowing for multiple modes of interaction combining language and visual inputs, greatly enhancing user-friendly communication.
The research team performed extensive experiments to verify Griffon v2’s efficacy in numerous localization tasks. It has reportedly reached state-of-the-art performance in phrase grounding, REC, and REG. In both quantitative and qualitative object counting, the model outperformed expert models, suggesting its superior perception and comprehension abilities. Credit for the research goes to the researchers involved in the project. The relevant paper and Github can be accessed for further details.
