Recent advancements in Multi-Modal (MM) pre-training have led to significant improvements in Machine Learning (ML) models’ capacity to understand diverse data types such as text, pictures, audio, and video. This has resulted in the development of advanced MultiModal Large Language Models (MM-LLMs) by integrating Large Language Models (LLMs) with multimodal data processing.
Instead of creating multimodal models from scratch, the integration of pre-trained LLMs with added modalities is utilized in MM-LLMs. This approach reduces computation costs and improves the model’s ability to manage different data types.
Models such as GPT-4(Vision) and Gemini are examples of recent achievements in this field, exhibiting notable capabilities in understanding and generating multimodal content. Research has also been carried out on models like Flamingo, BLIP-2, and Kosmos-1, capable of processing images, sounds, text, and even video.
An obstacle in MM-LLMs is effectively incorporating the LLM with other modal models. The different modalities need to be accurately synchronized and adjusted to function according to human comprehension and intentions. Current research aims to amplify the abilities of traditional LLMs while preserving their inherent capacity to rationalize and make decisions across wide-ranging multimodal tasks.
A detailed study on MM-LLMs was undertaken by researchers from Tencent AI Lab, Kyoto University, and Shenyang Institute of Automation. The research covers several aspects, from defining basic design formulations for model architecture to the training pipeline.
After elaborating on design formulations, the study explores the present state of MM-LLMs. A succinct introduction is given to each of the 26 recognized MM-LLMs, highlighting their unique compositions and characteristics. The study aims to give readers an understanding of the diversity and subtleties of presently applied models in the MM-LLMs domain.
An evaluation of MM-LLMs was conducted using industry standards. The assessment helped understand these models’ performance against industry standards and in real-world situations. The study summarizes key training methods successful in enhancing the overall effectiveness of MM-LLMs.
The researchers explored the five main components of MM-LLMs: modality encoder, LLM Backbone, Modality Generator, Input projector, and Output Projector. Modality Encoder transforms input data from various modalities into a comprehendible format for the LLM. LLM Backbone is often a pre-trained model, providing language processing and generation abilities. The Modality Generator turns the LLM outputs into multiple modalities. The input projector integrates and aligns the encoded multimodal inputs with the LLM, while the output projector changes the LLM’s output into a suitable format for multimodal expression.
The insightful research offers a comprehensive overview of MM-LLMs and their current effectiveness.
