Sigmoid Function

What is the Sigmoid Function?

The sigmoid function is a mathematical function that produces an S-shaped curve. It maps any real-valued number into a value between 0 and 1.

Why is the Sigmoid Function Used in Neural Networks?

The sigmoid function is used as an activation function in neural networks because it introduces non-linearity into the model.

Why Sigmoid Activation function is squeezing function?

As the activation function squeezes the input values fed to the hidden layers, the function returns the output between 0 and 1 only. So no matter how positive or negative numbers are fed to the layer, this function squeezes it between 0 and 1.

What is the main issue with the sigmoid function while backpropagation?

The main issue related to the activation function is when the new weights and biases are calculated by the gradient descent algorithm, if these values are very small, then the updates of the weights and biases will also be very low and hence, which results in vanishing gradient problem, where the model will not learn anything.

Graphically, the sigmoid function looks as shown below which is similar to S but rotated 90 degrees anti-clockwise.

Sigmoid function is one of the most commonly used activation functions in Machine learning and Deep learning. The sigmoid function can be used in the hidden layers, which take the output from the previous layer and brings the input values between 0 and 1. Now while working with neural networks it is necessary to calculate the derivate of the activation function.

Sigmoid function is also known as the squashing function, as it takes the input from the previously hidden layer and squeezes it between 0 and 1. So a value fed to the sigmoid function will always return a value between 0 and 1, no matter how big or small the value is fed.