The capabilities of computer vision studies have been vastly expanded due to deep features, which can unlock image semantics and facilitate diverse tasks, even using minimal data. Techniques to extract features from a range of data types – for example, images, text, and audio – have been developed and underpin a number of applications in fields such as semantic segmentation, neural rendering, and image generation. Their evolution continues to push the boundaries of what is possible in computer vision.
Deep feature applications do have limitations, however, as they often require increased spatial resolution to carry out tasks involving dense prediction tasks like segmentation and depth prediction. Models such as ResNet-50 and Vision Transformers (ViTs) condense and significantly reduce resolution, respectively, creating a challenge to apply deep features to tasks requiring precise spatial information.
Addressing this issue, researchers from MIT, Google, Microsoft and Adobe introduced FeatUp; a task-agnostic and model-agnostic framework designed to restore lost spatial information in deep features. Their development consisted of two variants. The first guides high-resolution features through a single forward pass, whilst the second tailors an implicit model to a single image to reconstruct features at any given resolution. Importantly, the restored features retain their original semantics and can replace existing features in numerous applications, thereby greatly enhancing resolution and performance.
Two key steps were followed by developers of FeatUp. The first created low-resolution feature views to hone into a single high-resolution output, with the team extracting a collection of low-resolution feature maps and producing information that would train the upsampler. The second step involved building a high-resolution feature map capable of reproducing lower-resolution, ‘jittered’ features when downscaled. This process, described as ‘ray-marching,’ reduces high-resolution features to low-resolution versions.
Further research revealed that a successful high-resolution feature map should completely rebuild the features observed across all varying views. The compression of spatially varying features (to their top 128 principal components) reduced the memory footprint and sped up training whilst retaining almost all pertinent information.
In conclusion, the novel framework FeatUp successfully restores lost spatial information in deep features and learns high-quality feature at any resolution, successfully overcoming a common issue in computer vision. Impressively, the two versions of FeatUp surpassed a range of baselines across linear probe transfer learning, model interpretability, and end-to-end semantic segmentation – helping significantly push the boundaries of what is achievable in computer vision. This ground-breaking research is credited to the collaborative research team at MIT, Google, Microsoft and Adobe.
