The challenge of image restoration is complex and has gained considerable attention from researchers. The primary goal of this is to create visually appealing and natural images while still preserving the perceptual quality of the degraded input. When there is no information available about the subject or degradation (known as blind restoration), it is essential to have an understanding of the range of natural images. For facial images, it is essential to include identity before ensuring that the output still has the individual’s unique facial features. In the past, research has looked into using reference-based face image restoration to meet this requirement, yet integrating personalization into diffusion-based blind restoration systems has remained a persistent challenge.
A team of researchers from the University of California, Los Angeles, and Snap Inc. have developed a method for personalized image restoration known as Dual-Pivot Tuning. Dual-Pivot Tuning is an approach used to customize a text-to-image prior in the context of blind image restoration. This process uses a limited set of high-quality images of an individual to improve the restoration of their other degraded images. It is designed to ensure that the restored images showcase the person’s identity and the degraded input image, while still having a natural look.
The study discusses how diffusion-based blind restoration techniques can struggle to keep the unique identity of an individual when applied to degraded facial images. It also looks at previous attempts at reference-based face image restoration, such as GFRNet, GWAINet, ASFFNet, Wang et al., DMDNet, and MyStyle. These approaches leverage single or multiple reference images to achieve personalized restoration, guaranteeing better fidelity to the specific features of the person in the degraded images. The proposed technique is different from these methods, as it uses a diffusion-based personalized generative prior, instead of feedforward architectures or GAN-based priors.
The study outlines the method for personalizing guided diffusion models for image restoration. Dual-Pivot Tuning consists of two steps: text-based fine-tuning to embed identity-specific information within diffusion priors and model-centric pivoting to harmonize the guiding image encoder with the personalized priors. The personalization operator of text-to-image diffusion models is defined so that a model is fine-tuned with a pivot to create a customized version. This technique involves in-context textual pivoting, where identity information is injected, followed by model-based pivoting, which uses general restoration to achieve high-fidelity restored images.
The Dual-Pivot Tuning technique for personalized restoration has been proven to achieve high identity fidelity and natural appearance in restored images. Qualitative and quantitative evaluations using metrics such as PSNR, SSIM, and ArcFace similarity have demonstrated the effectiveness of this method in restoring images with high fidelity to the person’s identity. The technique has been found to outperform generic priors regarding general image quality and is agnostic to different types of degradation, providing consistent restoration while still retaining identity.
In conclusion, the researchers have developed a groundbreaking AI approach for personalized facial image restoration. The Dual-Pivot Tuning technique has been found to achieve high identity fidelity and natural appearance in restored images. It outperforms various state-of-the-art alternatives for blind and few-shot personalized face image restoration and provides consistent results. This method has the potential to revolutionize image restoration and have a major impact on the field of computer vision.
