Visualizing the convergence of Gradient Descent using Linear Regression
Linear regression is a method for modeling the linear relationship between a dependent variable (also known as the response or output variable) and one or more independent variables (also known as the predictor or input variables). The goal of linear regression is to find the values of the model parameters (coefficients) that minimize the difference between the predicted values and the true values of the dependent variable.
The linear regression model can be expressed as follows:
where:
- is the predicted value of the dependent variable
- are the independent variables are the coefficients (model parameters) associated with the independent variables.
- b is the intercept (a constant term).
To train the linear regression model, you need a dataset with input features (independent variables) and labels (dependent variables). You can then use an optimization algorithm, such as gradient descent, to find the values of the model parameters that minimize the loss function.
The loss function measures the difference between the predicted values and the true values of the dependent variable. There are various loss functions that can be used for linear regression, such as mean squared error (MSE) and mean absolute error (MAE). The MSE loss function is defined as follows:
where:
- is the predicted value for the th sample
- is the true value for the th sample
- N is the total number of samples
- The MSE loss function measures the average squared difference between the predicted values and the true values. A lower MSE value indicates that the model is performing better.
Python3
import tensorflow as tf import matplotlib.pyplot as plt # Set up the data and model X = tf.constant([[1.], [2.], [3.], [4.]]) y = tf.constant([[2.], [4.], [6.], [8.]]) w = tf.Variable(0.) b = tf.Variable(0.) # Define the model and loss function def model(x): return w * x + b def loss(predicted_y, true_y): return tf.reduce_mean(tf.square(predicted_y - true_y)) # Set the learning rate learning_rate = 0.001 # Training loop losses = [] for i in range(250): with tf.GradientTape() as tape: predicted_y = model(X) current_loss = loss(predicted_y, y) gradients = tape.gradient(current_loss, [w, b]) w.assign_sub(learning_rate * gradients[0]) b.assign_sub(learning_rate * gradients[1]) losses.append(current_loss.numpy()) # Plot the loss plt.plot(losses) plt.xlabel("Iteration") plt.ylabel("Loss") plt.show() |
Output:
Loss vs Iteration
The loss function calculates the mean squared error (MSE) loss between the predicted values and the true labels. The model function defines the linear regression model, which is a linear function of the form .
The training loop performs 250 iterations of gradient descent. At each iteration, the with tf.GradientTape() as tape: block activates the gradient tape, which records the operations for computing the gradients of the loss with respect to the model parameters.
Inside the block, the predicted values are calculated using the model function and the current values of the model parameters. The loss is then calculated using the loss function and the predicted values and true labels.
After the loss has been calculated, the gradients of the loss with respect to the model parameters are computed using the gradient method of the gradient tape. The model parameters are then updated by subtracting the learning rate multiplied by the gradients from the current values of the parameters. This process is repeated until the training loop is completed.
Finally, the model parameters will contain the optimized values that minimize the loss function, and the model will be trained to predict the dependent variable given the independent variables.
A list called losses store the loss at each iteration. After the training loop is completed, the losses list contains the loss values at each iteration.
The plt.plot function plots the losses list as a function of the iteration number, which is simply the index of the loss in the list. The plt.xlabel and plt.ylabel functions add labels to the x-axis and y-axis of the plot, respectively. Finally, the plt.show function displays the plot.
The resulting plot shows how the loss changes over the course of the training process. As the model is trained, the loss should decrease, indicating that the model is learning and the model parameters are being optimized to minimize the loss. Eventually, the loss should converge to a minimum value, indicating that the model has reached a good solution. The rate at which the loss decreases and the final value of the loss will depend on various factors, such as the learning rate, the initial values of the model parameters, and the complexity of the model.
Visualizing the Gradient Descent
Gradient descent is an optimization algorithm that is used to find the values of the model parameters that minimize the loss function. The algorithm works by starting with initial values for the parameters and then iteratively updating the values to minimize the loss.
The equation for the gradient descent algorithm for linear regression can be written as follows:
where:
- w_i is the ith model parameter.
- is the learning rate (a hyperparameter that determines the step size of the update).
- is the partial derivative of the MSE loss function with respect to the ith model parameter.
This equation updates the value of each parameter in the direction that reduces the loss. The learning rate determines the size of the update, with a smaller learning rate resulting in smaller steps and a larger learning rate resulting in larger steps.
The process of performing gradient descent can be visualized as taking small steps downhill on a loss surface, with the goal of reaching the global minimum of the loss function. The global minimum is the point on the loss surface where the loss is the lowest.
Here is an example of how to plot the loss surface and the trajectory of the gradient descent algorithm:
Python3
import numpy as np import matplotlib.pyplot as plt # Generate synthetic data np.random.seed(42) X = 2 * np.random.rand(100, 1) - 1y = 4 + 3 * X + np.random.randn(100, 1) # Initialize model parameters w = np.random.randn(2, 1) b = np.random.randn(1)[0] # Set the learning rate alpha = 0.1 # Set the number of iterations num_iterations = 20 # Create a mesh to plot the loss surface w1, w2 = np.meshgrid(np.linspace(-5, 5, 100), np.linspace(-5, 5, 100)) # Compute the loss for each point on the grid loss = np.zeros_like(w1) for i in range(w1.shape[0]): for j in range(w1.shape[1]): loss[i, j] = np.mean((y - w1[i, j] \ * X - w2[i, j] * X**2)**2) # Perform gradient descent for i in range(num_iterations): # Compute the gradient of the loss # with respect to the model parameters grad_w1 = -2 * np.mean(X * (y - w[0] \ * X - w[1] * X**2)) grad_w2 = -2 * np.mean(X**2 * (y - w[0] \ * X - w[1] * X**2)) # Update the model parameters w[0] -= alpha * grad_w1 w[1] -= alpha * grad_w2 # Plot the loss surface fig = plt.figure(figsize=(10, 6)) ax = fig.add_subplot(projection='3d') ax.plot_surface(w1, w2, loss, cmap='coolwarm') ax.set_xlabel('w1') ax.set_ylabel('w2') ax.set_zlabel('Loss') # Plot the trajectory of the gradient descent algorithm ax.plot(w[0], w[1], np.mean((y - w[0]\ * X - w[1] * X**2)**2), 'o', c='red', markersize=10) plt.show() |
Output:
Gradient Descent finding global minima
This code generates synthetic data for a quadratic regression problem, initializes the model parameters, and performs gradient descent to find the values of the model parameters that minimize the mean squared error loss. The code also plots the loss surface and the trajectory of the gradient descent algorithm on the loss surface.
The resulting plot shows how the gradient descent algorithm takes small steps downhill on the loss surface and eventually reaches the global minimum of the loss function. The global minimum is the point on the loss surface where the loss is the lowest.
It is important to choose an appropriate learning rate for the gradient descent algorithm. If the learning rate is too small, the algorithm will take a long time to converge to the global minimum. On the other hand, if the learning rate is too large, the algorithm may overshoot the global minimum and may not converge to a good solution.
Another important consideration is the initialization of the model parameters. If the initialization is too far from the global minimum, the gradient descent algorithm may take a long time to converge. It is often helpful to initialize the model parameters to small random values.
It is also important to choose an appropriate stopping criterion for the gradient descent algorithm. One common stopping criterion is to stop the algorithm when the loss function stops improving or when the improvement is below a certain threshold. Another option is to stop the algorithm after a fixed number of iterations.
Overall, gradient descent is a powerful optimization algorithm that can be used to find the values of the model parameters that minimize the loss function for a wide range of machine learning problems.
Gradient Descent Optimization in Tensorflow
Gradient descent is an optimization algorithm used to find the values of parameters (coefficients) of a function (f) that minimizes a cost function. In other words, gradient descent is an iterative algorithm that helps to find the optimal solution to a given problem.
In this blog, we will discuss gradient descent optimization in TensorFlow, a popular deep-learning framework. TensorFlow provides several optimizers that implement different variations of gradient descent, such as stochastic gradient descent and mini-batch gradient descent.
Before diving into the details of gradient descent in TensorFlow, let’s first understand the basics of gradient descent and how it works.