Multi-Fedlity Optimization for Hyperparameter Tuning in Multi-Layer Perceptrons (MLPs)
Utilizing Coarse-to-Fine Search (CFS) for an informed search of best neural network hyperparameters
Too many technical terms in the title and subtitle? Let me explain one by one!
Hyperparameters are a type of variables that define how the model is trained. We need to manually specify the values of hyperparameters as they do not learn their values from the data. But, we cannot specify the optimal values in the first chance.
Hyperparameter tuning is the process of finding the optimal values for the hyperparameters. There are several methods to do that.
Grid search and random search are two popular hyperparameter tuning techniques.
Coarse-to-Fine Search (CFS) is also a hyperparameter tuning technique that falls under Multi-Fedlity Optimization (MFO) category which is suitable for large AI models such as neural networks as MFO category essentially includes hyperparameter tuning techniques that require very low computational resources compared to other methods.
CFS combines grid search and random search for hyperparameter tuning. It falls under the category of informed search which uses knowledge from previous…
