6533b860fe1ef96bd12c2e44
RESEARCH PRODUCT
Online Hyperparameter Search Interleaved with Proximal Parameter Updates
Baltasar Beferull-lozanoLuis M. Lopez-ramossubject
HyperparameterComputer scienceStability (learning theory)Approximation algorithm020206 networking & telecommunications02 engineering and technologyStationary pointLasso (statistics)Hyperparameter optimization0202 electrical engineering electronic engineering information engineering020201 artificial intelligence & image processingProximal Gradient MethodsOnline algorithmAlgorithmdescription
There is a clear need for efficient hyperparameter optimization (HO) algorithms for statistical learning, since commonly applied search methods (such as grid search with N-fold cross-validation) are inefficient and/or approximate. Previously existing gradient-based HO algorithms that rely on the smoothness of the cost function cannot be applied in problems such as Lasso regression. In this contribution, we develop a HO method that relies on the structure of proximal gradient methods and does not require a smooth cost function. Such a method is applied to Leave-one-out (LOO)-validated Lasso and Group Lasso, and an online variant is proposed. Numerical experiments corroborate the convergence of the proposed methods to stationary points of the LOO validation error curve, and the improved efficiency and stability of the online algorithm.
| year | journal | country | edition | language |
|---|---|---|---|---|
| 2021-01-24 | 2020 28th European Signal Processing Conference (EUSIPCO) |