Python Implementation of Gradient Descent
At first, we will import all the necessary Python libraries that we will need for mathematical computation and plotting like numpy for mathematical operations and matplotlib for plotting. Then we will define a class Linear_Regression that represents the linear regression model.
We will make a update_coeffs method inside the class to update the coefficients (parameters) of the linear regression model using gradient descent. To calculate the error between the predicted output and the actual output we will make a predict method that will make predictions using the current model coefficients.
For updating and calculating the gradient of the error we will make compute_cost which will apply gradient descent on (mean squared error) between the predicted values and the actual values.
Python3
# Implementation of gradient descent in linear regressionimport numpy as npimport matplotlib.pyplot as pltclass Linear_Regression: def __init__(self, X, Y): self.X = X self.Y = Y self.b = [0, 0] def update_coeffs(self, learning_rate): Y_pred = self.predict() Y = self.Y m = len(Y) self.b[0] = self.b[0] - (learning_rate * ((1/m) * np.sum(Y_pred - Y))) self.b[1] = self.b[1] - (learning_rate * ((1/m) * np.sum((Y_pred - Y) * self.X))) def predict(self, X=[]): Y_pred = np.array([]) if not X: X = self.X b = self.b for x in X: Y_pred = np.append(Y_pred, b[0] + (b[1] * x)) return Y_pred def get_current_accuracy(self, Y_pred): p, e = Y_pred, self.Y n = len(Y_pred) return 1-sum( [ abs(p[i]-e[i])/e[i] for i in range(n) if e[i] != 0] )/n # def predict(self, b, yi): def compute_cost(self, Y_pred): m = len(self.Y) J = (1 / 2*m) * (np.sum(Y_pred - self.Y)**2) return J def plot_best_fit(self, Y_pred, fig): f = plt.figure(fig) plt.scatter(self.X, self.Y, color='b') plt.plot(self.X, Y_pred, color='g') f.show()def main(): X = np.array([i for i in range(11)]) Y = np.array([2*i for i in range(11)]) regressor = Linear_Regression(X, Y) iterations = 0 steps = 100 learning_rate = 0.01 costs = [] # original best-fit line Y_pred = regressor.predict() regressor.plot_best_fit(Y_pred, 'Initial Best Fit Line') while 1: Y_pred = regressor.predict() cost = regressor.compute_cost(Y_pred) costs.append(cost) regressor.update_coeffs(learning_rate) iterations += 1 if iterations % steps == 0: print(iterations, "epochs elapsed") print("Current accuracy is :", regressor.get_current_accuracy(Y_pred)) stop = input("Do you want to stop (y/*)??") if stop == "y": break # final best-fit line regressor.plot_best_fit(Y_pred, 'Final Best Fit Line') # plot to verify cost function decreases h = plt.figure('Verification') plt.plot(range(iterations), costs, color='b') h.show() # if user wants to predict using the regressor: regressor.predict([i for i in range(10)])if __name__ == '__main__': main() |
Output:
100 epochs elapsed Current accuracy is : 0.9836456109008862
Regression line before gradient descent iteration
Regression line after gradient descent iteration
Accuracy graph for gradient descent on model
Note: Gradient descent sometimes is also implemented using Regularization.
Advantages Of Gradient Descent
- Flexibility: Gradient Descent can be used with various cost functions and can handle non-linear regression problems.
- Scalability: Gradient Descent is scalable to large datasets since it updates the parameters for each training example one at a time.
- Convergence: Gradient Descent can converge to the global minimum of the cost function, provided that the learning rate is set appropriately.
Disadvantages Of Gradient Descent
- Sensitivity to Learning Rate: The choice of learning rate can be critical in Gradient Descent since using a high learning rate can cause the algorithm to overshoot the minimum, while a low learning rate can make the algorithm converge slowly.
- Slow Convergence: Gradient Descent may require more iterations to converge to the minimum since it updates the parameters for each training example one at a time.
- Local Minima: Gradient Descent can get stuck in local minima if the cost function has multiple local minima.
- Noisy updates: The updates in Gradient Descent are noisy and have a high variance, which can make the optimization process less stable and lead to oscillations around the minimum.
Overall, Gradient Descent is a useful optimization algorithm for linear regression, but it has some limitations and requires careful tuning of the learning rate to ensure convergence.
Gradient Descent in Linear Regression
We know that in any machine learning project our main aim relies on how good our project accuracy is or how much our model prediction differs from the actual data point. Based on the difference between model prediction and actual data points we try to find the parameters of the model which give better accuracy on our dataset\, In order to find these parameters we apply gradient descent on the cost function of the machine learning model.