Initiating Hyperparameters and DataLoader for Training Sentence Autocomplete Model
In the model, we will define hyperparameters basically hyperparameters are the configurable parameters that determine the behavior of the model during training. In this code, the hyperparameters are defined as follows:
- sequence_length: The length of the input sequence.
- batch_size: The number of samples processed in each iteration.
- learning_rate: The step size at which the optimizer adjusts the model’s parameters.
- num_epochs: The number of times the entire dataset is passed through the model during training.
We divided our dataset into two parts for training and validation using Pytorch. We used PyTorch’s DataLoader to create iterators for efficient loading of data during training and validation. The training and validation datasets are passed to the respective data loaders, specifying the batch size and whether shuffling is required for the training data.
For training the model we have used a training loop that iterates over the specified number of epochs. For each epoch, the model is put in training mode using(model. train()), and the total loss is initialized. The loop iterates over batches of data from the training data loader. In every loop a sequence of task happen these are as:
- For each batch, the optimizer’s gradients are reset using the Pytorch function (optimizer.zero_grad()),
- The initial hidden state for the LSTM model is obtained using the equation (hidden = model.init_state(sequence_length))
- The model is called with the inputs and hidden state using equation (outputs, _ = model(inputs, hidden)).
- The loss is then computed using the predicted outputs and the targets and the gradients are calculated using the Pytorch (loss.backwards ()) function.
The optimizer then performs a parameter update using the optimizer.step(), and the loss is added to the total loss for the epoch. After the loop, the average loss for the epoch is calculated and printed.
Python3
from torch.utils.data import DataLoader, random_split # Hyperparameters sequence_length = 10batch_size = 64learning_rate = 0.001num_epochs = 10 # Create the dataset dataset = TextDataset(sequence_length) # Split the dataset into training and validation sets train_size = int(0.8 * len(dataset)) val_size = len(dataset) - train_size train_dataset, val_dataset = random_split(dataset, [train_size, val_size]) # Create data loaders train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True) val_loader = DataLoader(val_dataset, batch_size=batch_size) # Create the model model = LSTMModel(dataset) # Define the loss function and optimizer criterion = nn.CrossEntropyLoss() optimizer = torch.optim.Adam(model.parameters(),\ lr=learning_rate) # Training loop for epoch in range(num_epochs): model.train() total_loss = 0.0 for batch in train_loader: inputs, targets = batch optimizer.zero_grad() hidden = model.init_state(sequence_length) outputs, _ = model(inputs, hidden) loss = criterion(outputs.view(-1, len(dataset.unique_words)), \ targets.view(-1)) loss.backward() optimizer.step() total_loss += loss.item() # Calculate average loss for the epoch average_loss = total_loss / len(train_loader) # Print the epoch and average loss print(f"Epoch [{epoch+1}/{num_epochs}],\ Average Loss: {average_loss:.4f}") # Validation loop model.eval() val_loss = 0.0 with torch.no_grad(): for batch in val_loader: inputs, targets = batch hidden = model.init_state(sequence_length) outputs, _ = model(inputs, hidden) loss = criterion(outputs.view(-1, len(dataset.unique_words)), \ targets.view(-1)) val_loss += loss.item() # Calculate average validation loss for the epoch average_val_loss = val_loss / len(val_loader) # Print the epoch and average validation loss print(f"Epoch[{epoch+1}/{num_epochs}], Validation Loss: {average_val_loss: .4f}") |
Output:
Epoch [1/10], Average Loss: 6.8103
Epoch [1/10], Validation Loss: 6.5937
Epoch [2/10], Average Loss: 6.4668
Epoch [2/10], Validation Loss: 6.3104
Epoch [3/10], Average Loss: 6.2176
Epoch [3/10], Validation Loss: 6.1290
Epoch [4/10], Average Loss: 6.0840
Epoch [4/10], Validation Loss: 6.0208
Epoch [5/10], Average Loss: 5.9850
Epoch [5/10], Validation Loss: 5.9312
Epoch [6/10], Average Loss: 5.8937
Epoch [6/10], Validation Loss: 5.8397
Epoch [7/10], Average Loss: 5.8056
Epoch [7/10], Validation Loss: 5.7547
Epoch [8/10], Average Loss: 5.7232
Epoch [8/10], Validation Loss: 5.6692
Epoch [9/10], Average Loss: 5.6379
Epoch [9/10], Validation Loss: 5.5762
Epoch [10/10], Average Loss: 5.5473
Epoch [10/10], Validation Loss: 5.4821
Sentence Autocomplete Using Pytorch
Natural Language Processing(NLP) is one of the most flourishing parts of deep learning. Several applications of NLP are being used continuously in daily life. In this article, we are going to see how we can use NLP to autocomplete half-written sentences using deep learning methods. We will also see how we can generate clean data for training our NLP model. We will cover the following steps in this article
- Cleaning the text data for training the NLP model
- Loading the dataset using PyTorch
- Creating the LSTM model
- Training an NLP model
- Making inferences from the trained model
We have seen applications like google keyboard where Google recommends what to type next based on the words which we have already written in the chatbox draft. However, to recommend the next term application like Google has been trained on billions of written sentences. In our model, we will use Wikipedia sentences that are freely available on the internet to download and that we can use for training our model.