The computer vision sector is currently dominated by large-scale models that offer remarkable performance but demand high computational resources, making them impractical for real-world applications. To address this, the Google Research Team has opted to reduce these models into smaller, more efficient architectures via model pruning and knowledge distillation. The team’s focus is on knowledge distillation, which shrinks a large and inefficient model into a smaller and more efficient version, aligning the “student” model’s predictions with the “teacher” model’s to facilitate compression. Using aggressive data augmentation techniques and support points, the student model can improve its generalizability.
The team’s findings indicate that aggressive augmentation, long training periods, and consistent image views are important for successful model compression via knowledge distillation. However, there are various challenges, such as the temptation to precompute image operations to save computation, the discrepancy in design choices across different knowledge distillation usage scenarios, and the necessity of long-term training for optimal performance.
In their empirical study, the team focused on compressing the large BiT-ResNet-152×2, trained on the ImageNet-21k dataset and fine-tuned to suit applicable datasets. They managed to reduce it to a ResNet-50 architecture without losing accuracy, and upon testing on smaller datasets, achieved an impressive 82.8% on ImageNet with 9600 distillation epochs.
The team’s efforts demonstrate the robustness and efficacy of their distillation formula, showcasing the potential for successful model compression and transition across model families, like from BiT-ResNet to MobileNet architecture. This gives hope for the future of model compression in computer vision, making it a more practical alternative for real-world applications.
