Autoregressive image generation models have traditionally been built using vector-quantized representations. However, these models have exhibited drawbacks, particularly related to their limited flexibility and computational intensity that often result in suboptimal image reconstruction. The vector quantization process involves the conversion of continuous image data into discrete tokens, which can also give rise to loss of image quality due to reconstruction errors.
To address these challenges, a team of researchers from MIT CSAIL, Google’s DeepMind, and Tsinghua University has come up with a new technique that dispenses the need for vector quantization. The innovation rests on the diffusion process that models the probability distribution per token within a continuous-valued space. The Diffusion Loss function used in the new technique enhances the quality of image generation and efficiency of autoregressive models by keeping the data’s continuous nature integrity intact.
The novel method begins with a noisy version of the target token and refines it iteratively through a small denoising network conditional on preceding tokens. The diffusion process predicts continuous-valued vectors for each token, with the network being trained via backpropagation using the Diffusion Loss function. Key success metrics such as the Fréchet Inception Distance (FID) and Inception Score (IS) suggest significant improvements in image generation quality with this technique. Models employing Diffusion Loss consistently outscored those using traditional cross-entropy loss. Specifically, masked autoregressive models with Diffusion Loss achieved an FID score of 1.55 and an IS of 303.7, indicating considerable progress over previous methods.
The new technique attains impressive generation rates of less than 0.3 seconds per image and produces considerable improvements across various model variants, further cementing the effectiveness of the method.
In a nutshell, the diffusion-based technique has come as a groundbreaking solution to address the ongoing dependency on vector quantization in autoregressive image generation. By facilitating the modeling of continuous-valued tokens, the novel technique has substantially enhanced the efficiency and quality of autoregressive models. The innovation has the potential for wide applications in the realm of AI research, especially concerning image generation and other continuous-valued domains.
