Researchers have developed a text-to-image diffusion transformer called Hunyuan-DiT. Its intention is to understand both English and Chinese text prompts in a nuanced way. Its creation involves important elements and steps to ensure optimal image production and finer language understanding.
The fundamental components of Hunyuan-DiT include its Transformer structure, a Bilingual and Multilingual encoding, and Enhanced Positional Encoding. The transformer structure is built to enhance the model’s capability to create visuals from textual descriptions. This includes enhancing the model’s ability to process complex language inputs and ensuring accurate data recording.
The model’s ability to comprehend prompts is largely reliant on the text encoder. This uses both a bilingual CLIP, capable of handling English and Chinese, and a multilingual T5 encoder to improve comprehension and handling of context.
The Enhanced Positional Encoding algorithm has been tailored to better manage sequence in text and spatial characteristics in images. Assisting the model to correctly assign tokens to appropriate image attributes and conserving the token sequence.
Specific structures have been put in place to augment and support Hunyuan-DiT’s capabilities. The team curated and collected a large and diverse dataset, augmented and filtered the dataset, and iteratively optimized the model. This involved constantly upgrading the model’s performance based on new data and user feedback, using a ‘data convoy’ technique.
To increase language understanding accuracy, a specially trained MLLM has been utilized. This generator uses contextual knowledge to produce accurate and detailed captions, improving the quality of its generated images.
Hunyuan-DiT also enables multi-turn dialogues for interactive image generation. Users providing input over multiple iterations can refine the generated images, leading to more accurate and satisfactory results.
To assay Hunyuan-DiT, an evaluation methodology was created, involving over 50 evaluators. The method checks the clarity of the subject, visual quality, absence of AI artifacts, and text-image consistency in the created images. Compared to other models, Hunyuan-DiT showed superior performance in Chinese-to-image creation, creating clear and semantically correct visuals based on Chinese cues.
In conclusion, Hunyuan-DiT stands out in text-to-image creation, especially for Chinese prompts. Its superior performance emerges from expertly constructed transformer architecture, text encoders, and positional encoding, coupled with a robust data pipeline. Its ability for interactive, multi-round dialogues increases its versatility and application in a range of uses.
