Researchers from Peking University, Pika, and Stanford University have devised a novel text-to-image generation framework called RPG (Recaption, Plan, and Generate). RPG efficiently converts text prompts into images, with a specific focus on complex prompts that involve rendering multiple objects with various attributes and relationships. RPG is an evolution over previous models as it outperforms them mainly in terms of handling complex prompts.
Prior models relied on additional layouts, image understanding feedback, or prompt-aware attention guidance, and were hindered by limitations such as inability to handle overlapped objects and heightened training costs for complex prompts. RPG sidesteps these drawbacks by leveraging multimodal Large Language Models (MLLMs) to enhance compositionality within text-to-image diffusion models, a feature entirely absent in traditional conversion techniques.
Three core strategies constitute the RPG: Multimodal Recaptioning, Chain-of-Thought Planning, and Complementary Regional Diffusion. The first converts text prompts into more descriptive sub-prompts. The second partitions the image into complementary subregions and assigns individual sub-prompts to each one. The last strategy generates image content guided by sub-prompts within assigned regions and spatially combines them.
The RPG framework employs GPT-4 for the recaptioning and planning stages, while SDXL functions as the base diffusion backbone. Experimental results highlight RPG’s superiority in tasks such as multi-category object composition and text-image semantic alignment. RPG also effectively adapts to various MLLM architectures and diffusion backbones.
Quantitative and qualitative evaluations reveal that RPG outperforms existing models in metrics such as attribute binding, object relationship recognition, and prompt complexity. The resultant images generated by RPG are significantly detailed, capturing all the elements present in the text prompts and surpassing other diffusion models’ precision, flexibility, and generative power.
The researchers have made the paper available, as well as its code on Github for those interested in the project. The research suggests RPG has the potential to significantly advance the text-to-image synthesis field.
