A new method for manipulating and improving control levels in image generative models has been introduced by researchers from MIT, Tsinghua University, and NVIDIA. The technique, known as Condition-Aware Neural Network (CAN), enhances the image generation process by variably adjusting the neural network’s weight. This is achieved via a condition-aware weight generation module which generates conditional weight for convolution or linear layers according to the input condition.
The condition-aware aspect of the model is applied to a subset of modules, an approach which enhances both efficiency and performance. The CAN technique also aligns with the notion that generating the conditional weight directly is more effective.
During evaluations, CAN was measured against two diffusion transformer models, DiT and UViT, and exhibited significant performance improvements for these models while adding minimal computational cost. It was also shown to outperform earlier conditional control methods by a substantial margin, while using only a fraction of the Multiply–accumulate operation (MAC) used by other methods per sampling step.
An alternate approach, Adaptive Kernel Selection (AKS), generates scaling parameters to merge base convolution kernels, as opposed to generating the conditional weight directly. While this method has a smaller overhead than CAN, it cannot compete with CAN’s performance, demonstrating that dynamic parameterization alone does not guarantee optimal performance.
When evaluating CAN, the team performed tests on class-conditional image generation using ImageNet and text-to-image generation on COCO, demonstrating consistent and significant improvements over prior conditional control methods.
Ultimately, the CAN method has significant potential for improving efficiency and boosting performance of image generative models. Notably, a new family of diffusion transformer models was created, marrying CAN and EfficientViT. The method also holds promise for application in more complex tasks, such as large-scale text-to-image generation and video generation.
In sum, the Condition-Aware Neural Network is a novel method which enhances control of image-generative models, demonstrating the potential of weight manipulation for conditional control. This could pave the way for improved deployment of image generative models in the future, yielding improved performance and efficiency. The paper detailing the researchers’ findings is available for further reading.
