Traditionally, machine learning models have been trained and tested on data from the same distribution. However, researchers have found that models perform more effectively when dealing with data from multiple distributions. This flexibility is often achieved through “rich representations,” surpassing the capabilities of models trained on traditional sparsity-inducing regularization or common stochastic gradient methods.
However, optimizing these models to function well across various distributions remains challenging. While researchers have utilized various methods, such as adversarially reweighting the training dataset and combining representations from multiple networks, these have limitations especially with data distributions that diverge substantially from the training set.
Researchers from New York University and Facebook AI Research have proposed a pioneering tactic to ensure out-of-distribution (OOD) performance – the use of high dropout rates. Conventionally, training a deep network from scratch with high dropout rates has proved nearly impossible due to the network’s complexity and depth. But fine-tuning a pre-trained model with such conditions has not only been feasible, but it has also topped the performance of ensembles and weight-averaging methods.
The method employs a complex fine-tuning process on a deep learning network that has primarily been trained on large datasets, applying very high dropout rates to the penultimate layer during fine-tuning. This practice blocks any contributions from all residual blocks without introducing new representations, instead utilizing existing ones. Impressively, this technique performs comparably to, or even better than conventional methods like ensembles and weight averaging.
Furthermore, the results showed a considerable improvement in OOD performance across several benchmarks. For instance, models fine-tuned with this method demonstrated significant advancements in the VLCS dataset, a domain adaptation benchmark that presents substantial generalization challenges.
In summary, this research gives a persuasive argument for reconsidering fine-tuning practices in machine learning. By testing and validating large dropout rates, more adaptable and resistant models can be developed, capable of handling diverse data distributions. This approach aids our comprehension of rich representations and sets a new standard for OOD performance, marking a noteworthy progression towards more generalized machine-learning solutions. The researchers hope that this development could enhance the robustness and reliability of machine-learning models across diverse datasets.
