Boosting, a highly effective machine learning (ML) optimization setting, has evolved from a model that did not require first-order loss information into a method that necessitates this knowledge. Despite this transformation, few investigations have been made into boosting, even as machine learning witnesses a surge in zeroth-order optimization – methods that bypass the use of gradient information to find a function’s minimum and maximum values. Early boosting models simply required a weak learner that could provide classifiers slightly better than random guessing.
As zeroth-order optimization gains popularity, the question arises whether differentiability is necessary for boosting, which loss functions can be boosted, and how boosting stacks up against the recent advances in bringing gradient descent to zeroth-order optimization.
Google’s research team is actively working to develop a formal boosting technique that can handle loss functions with sets of discontinuities that have zero Lebesgue measure. Practically speaking, any stored loss function would likely meet this criterion based on standard floating-point encoding. In theory, the researchers are considering loss functions that are not necessarily convex, differentiable, Lipschitz, or continuous.
The proposed SECBOOST technique offers a promising solution by managing local minima and stable value losses over parts of their domain. As the technique is applied to a wider range of contexts, the potential of SECBOOST becomes increasingly significant and could prove impactful in boosting research and application.
In comparison with recent developments in zeroth-order optimization, researchers have found that boosting performs better. Although the team has yet to fully resolve the issue of optimizing the offset oracle, they have made strides through the use of design tricks used in the implementation of such algorithms. In fact, preliminary experiments suggest that the SECBOOST technique can optimize unconventional types of losses.
All credits for this research go towards the research team behind the project from Google and this research paper. More insights and discussions about this work can be found on social media platforms like Twitter, Telegram Channel, LinkedIn Group and the 46k+ ML SubReddit.
