Google’s research team is working on developing an optimized machine learning (ML) method known as “boosting.” Boosting involves creating high performing models using a “weak learner oracle” which gives classifiers a performance slightly better than random guessing. Over the years, boosting has evolved into a first-order optimization setting. However, some in the industry erroneously define it like this, even though its original model did not require first-order loss information.
The concept of “zeroth order optimization” refers to optimization methods that function without using gradient information to identify a function’s minimum and maximum values. They excel in situations where a function is either noisy and non-differentiable, or when calculating the gradient would be highly costly or unfeasible. In contrast to gradient-based optimization, zeroth order optimization’s search for the best solution is solely guided by function evaluations.
This shift toward zeroth-order optimization in machine learning has prompted debate on whether differentiability is necessary for boosting and which loss functions can be enhanced with a “weak learner.” Another question is how boosting measures up to the recent progress made on incorporating gradient descent into zeroth-order optimization.
Google’s researchers aim to formally use boosting for handling loss functions with sets of discontinuities with zero Lebesgue measure. Most stored loss functions would satisfy this condition with conventional floating-point encoding. Unlike classical zeroth-order optimization solutions, the researchers include losses that are not necessarily convex, differentiable, Lipschitz, or continuous. To avoid derivate use in boosting, they use or build on techniques from quantum calculusℎ, some of which are common in zeroth-order optimization research.
They have also proposed a method known as SECBOOST, which reveals two extra areas where consciously formulated decisions can maintain assumptions across a greater number of rounds. This aids not only in addressing local minima problems but also in controlling losses that exhibit stable values over portions of their area. According to the findings, boosting is better than the recent advancements in zeroth-order optimization. SECBOOST’s capacity is significant, giving hope for the future of boosting research and application.
Though the team hasn’t completely resolved the losing minima issue, they have revealed some experimental results in the appendix hinting that SECBOOST can optimize rare types of losses. The experiments suggest that although there’s still room for improvement, the advancements made in zeroth order optimization can facilitate the innovation of such protocols.
All credit for this research goes to Google’s research team. The findings were shared publicly to invite further research and collaboration from the wider machine learning community. People interested in staying updated about this research are encouraged to follow the team on Twitter and join their Telegram Channel and LinkedIn group.
