The field of multimodal learning, which involves training models to understand and generate content in multiple formats such as text and images, is evolving rapidly. Current models have inefficiencies in dealing with text-only and text-image tasks, often excelling in one domain but underperforming in the other. This necessitates distinct systems to retrieve different forms of information, labelling the need for a more unified approach.
Contrastive Language-Image Pre-training (CLIP) models strive to align images and text by pairing each image with its respective caption. However, these models struggle with tasks that involve long text inputs, leading to suboptimal performance when retrieving textual information.
Jina AI Researchers propose overcoming these challenges with an open-source model called jina-clip-v1. The model uses a novel multi-task contrastive training approach to optimise the alignment of text-image and text-text representations. The aim is to facilitate the model’s ability to handle both kinds of tasks efficiently, reducing the need for separate models.
The training method involves a three-stage process. The first stage aligns text and image using short, human-made captions. The second stage introduces longer, synthetic image captions, and the third stage fine-tunes the text encoder using hard negatives to enhance its ability to distinguish relevant texts from irrelevant ones.
Performance evaluations show that the jina-clip-v1 model achieves superior results in text-image and retrieval tasks. In the Massive Text Embedding Benchmark (MTEB), which involves 58 datasets across eight tasks, jina-clip-v1 performed competitively with top-tier text-only embedding models. It achieved an average score of 60.12%, which is approximately 15% better than other CLIP models overall and 22% better in retrieval tasks.
The evaluation included various training stages, involving datasets like the LAION-400M. While the initial stages showed significant improvements in multimodal performance, discrepancies in text lengths in the data types used led to shortfalls in text-text performance. However, synthetic data with longer captions and the use of hard negatives in subsequent stages improved performances in text-text and text-image retrieval tasks.
In summary, the research confirms that jina-clip-v1, a unified multimodal model, can streamline information retrieval systems. This is achieved by integrating text and image understanding capabilities into a single framework, which boosts efficiency in numerous applications and leads to the reduced need for separate models. The model addresses the inefficiencies of current multimodal models and outperforms them in areas like text-image and retrieval tasks. The development of the jina-clip-v1 model signals a step forward in multimodal learning, promising enhanced efficiency and potential savings in computational resources and complexity.
