Mathematics Behind Gradient Descent

In the Machine Learning Regression problem, our model targets to get the best-fit regression line to predict the value y based on the given input value (x). While training the model, the model calculates the cost function like Root Mean Squared error between the predicted value (pred) and true value (y). Our model targets to minimize this cost function. 
To minimize this cost function, the model needs to have the best value of θ₁ and θ₂(for Univariate linear regression problem). Initially model selects θ₁ and θ₂ values randomly and then iteratively update these value in order to minimize the cost function until it reaches the minimum. By the time model achieves the minimum cost function, it will have the best θ₁ and θ₂ values. Using these updated values of θ₁ and θ₂ in the hypothesis equation of linear equation, our model will predict the output value y.  

How do θ₁ and θ₂ values get updated?  

Linear Regression Cost Function: 

so our model aim is to Minimize  \frac{1}{2m} \sum_{i=1}^{m} (h_\theta(x^{(i)}) – y^{(i)})^2  and store the parameters which makes it minimum. 

Gradient Descent Algorithm For Linear Regression 

-> θj     : Weights of the hypothesis.
-> hθ(xi) : predicted y value for ith input.
-> i     : Feature index number (can be 0, 1, 2, ......, n).
-> α     : Learning Rate of Gradient Descent.

How Does Gradient Descent Work  

Gradient descent works by moving downward toward the pits or valleys in the graph to find the minimum value. This is achieved by taking the derivative of the cost function, as illustrated in the figure below. During each iteration, gradient descent step-downs the cost function in the direction of the steepest descent. By adjusting the parameters in this direction, it seeks to reach the minimum of the cost function and find the best-fit values for the parameters. The size of each step is determined by parameter α known as Learning Rate. 
In the Gradient Descent algorithm, one can infer two points : 

• If slope is +ve : θ_j = θ_j – (+ve value). Hence the value of θ_j decreases.

• If slope is -ve : θ_j = θ_j – (-ve value). Hence the value of θ_j increases.

How To Choose Learning Rate 

The choice of correct learning rate is very important as it ensures that Gradient Descent converges in a reasonable time. : 

• If we choose α to be very large, Gradient Descent can overshoot the minimum. It may fail to converge or even diverge. 
   

• If we choose α to be very small, Gradient Descent will take small steps to reach local minima and will take a longer time to reach minima. 
   

We know that in any machine learning project our main aim relies on how good our project accuracy is or how much our model prediction differs from the actual data point. Based on the difference between model prediction and actual data points we try to find the parameters of the model which give better accuracy on our dataset\, In order to find these parameters we apply gradient descent on the cost function of the machine learning model.