What happens if you do not use any activation function in a neural network’s hidden layer(s)?

Let me explain this using an example — Neural Networks and Deep Learning Course: Part 6

Image by author, made with draw.io

In Part 5, we’ve discussed different types of activation functions and their uses in neural networks.

So, “What happens if you do not use any activation function in a neural network’s hidden layer(s)?”. We’ll explain this using the following example.

Consider the following neural network model with two hidden layers.

(Image by author, made with draw.io)

Here, we represent the input data and parameter values in matrices. In the following mathematical expressions, the X represents the input data. The W1, W2 and W3 are weight matrices and the b1, b2 and b3 are bias vectors.

Let’s perform the calculations inside the above neural network model without using any activation function.

Hidden layer 1: Here, the input is X.

X*W1 + b1

Hidden layer 2: Here, the input is X*W1 + b1.

(X*W1 + b1)*W2 + b2