Convex vs Non-Convex Loss Functions in Machine Learning and Optimization
Plus concave functions!
In machine learning and optimization, we often focus on loss minimization to compare the predicted values of an AI model with its corresponding ground truth values.
The loss function calculates the loss score, the discrepancy between the predicted and ground truth values.
Training a machine learning or deep learning model generally involves an iterative process until the loss function is minimized. The loss function captures how well the model performs in each iteration.
The loss function is minimized using an optimization algorithm (optimizer) such as gradient descent. The objective of the optimizer is to find the global minimum for the loss function. The optimizer can only sometimes find a global minimum by avoiding all the local minima (if any)! But, there are situations in which the optimizer will never reach the global minimum of the loss function. Today, we’ll find out the reasons for this.
Minimum and maximum points of a function
Just like any other function, we can analyze the loss function by taking the derivative (gradient) of the function. The gradient of a loss function identifies the direction in which we can adjust…
