Multimodal machine learning combines various data types such as text, images, and audio to create more accurate and comprehensive models. However, large multimodal models (LMMs), like LLaVA, have been facing problems dealing with high-resolution graphics due to their inflexible and inefficient nature. Many have recognized the necessity for methods that may adjust the number of tokens dynamically based on the complexity of the input received visually.
Past responses to this issue, like token pruning and merging, have aimed to reduce the number of visual tokens fed into the language model, though these have also fallen short, leaving a dire need for better methods. Researchers from the University of Wisconsin-Madison and Microsoft Research have answered this call by introducing Matryoshka Multimodal Models (M3). Drawing inspiration from Matryoshka dolls, the M3 model displays visual content in the form of nested sets of visual tokens that record information across several granularities. This approach offers control over the visual granularity during inference, making adjustments of the number of tokens based on complexity, or simplicity, of the content possible.
The M3 model encodes images into multiple sets of visual tokens with increased levels of granularity, learning to derive coarser tokens from finer ones during the training. It uses scales such as 1, 9, 36, 144, and 576 tokens enabling the model to hold spatial information while adjusting the level of detail tailored to specific requirements.
On benchmarks in the COCO style, the M3 model achieved notable precision similar to using all 576 tokens with only about 9 per image. This represented a substantial leap in efficiency without sacrificing accuracy. The model also displayed high performance on other benchmarks, even with a significantly reduced number of tokens.
M3’s model can adjust to different computational and memory difficulties during deployment by allowing versatile control over the quantity of visual tokens. This flexibility has wide applicability in real-world scenarios where resources may be scarce. The M3 model also provides a framework for evaluating the visual complexity of datasets, aiding researchers to assess the optimum granularity needed for various tasks.
To sum it all up, the Matryoshka Multimodal Models (M3) addresses the LMMs’ inefficiencies and offers an adjustable, adaptive method for visual content representation. The ability to dynamically fine-tune granularities based on content complexity strikes a balance between performance and computational cost. This innovative approach opens new prospects for multimodal models’ application in diverse environments, emerging as an encouraging step towards efficient and robust multimodal systems.
