The recent advancement in text-to-image generation using diffusion models has produced impressive results with high-quality, realistic images. Yet, despite these successes, diffusion models, including Stable Diffusion, often struggle to follow prompts correctly, particularly when spatial or common-sense reasoning is needed. These shortcomings become evident in four key scenarios: negation, numeracy, attribute assignment, and spatial relationships. A new method called LLM-grounded Diffusion (LMD) is proposed to be a viable alternative that outperforms in these problem areas.
Collecting vast, multi-modal datasets and training large diffusion models could address these issues. However, this approach presents significant challenges, including the time, resources, and cost associated with training both large language models (LLMs) and diffusion models.
The solution offered involves a new two-stage generation process that adjusts existing diffusion models to increase their efficiency in spatial and common-sense reasoning without additional training costs. The first stage involves using an LLM to generate a text-based layout. When given an image prompt, the LLM produces a scene layout with bounding boxes and individual descriptions. The second stage involves guiding a diffusion model to create images based on that layout. This process is existing-pretrained-model-based and does not require any additional training of LLMs or optimizing the parameters of diffusion models.
Moreover, the LMD approach also has additional capabilities. It naturally allows for multi-round, dialog-based scene specification and can handle prompts in languages not supported by the underlying diffusion model. With an LLM that facilitates multi-round dialog, users can add further information or clarifications, enabling modifications to each prompt along the way.
An example of this is adding an object to the scene or changing the existing objects’ location or descriptions. Additionally, LMD can accept non-English prompts and produce layouts, even if the underlying model doesn’t support the language. These layouts made will then have descriptions of boxes and the background in English for subsequent imagery creation.
The effectiveness of LMD has been validated through comparisons with the base diffusion model that LMD uses, namely SD 2.1. Results show that LMD not only outperforms the base model in generating accurate images corresponding to prompts, but it also allows for counterfactual text-to-image generation—a capability the base model lacks.
To understand more about LLM-grounded Diffusion (LMD), readers are encouraged to visit the dedicated website and refer to the research paper on arXiv.
For researchers seeking to advance their work inspired by LLM-grounded Diffusion, the authors have provided a BibTex citation for proper referencing.
