What are Stacked RNNs

A single-layered RNN model has only a hidden layer which is liable to process sequential data. But Stacked RNN is a special kind of model that has multiple RNN layers one on each layer. This creates a ‘Stack’. Each layer of this stack processes the input sequence.

• When an Input is passed to Layer 1:
  • The input ( ) passes through the RNN layer 1. There, the hidden state gets updated as:
    
    where
    • = present Hidden state
    • = previous hidden state
    • = input to the RNN layer
    • = weights associated with the input
    • = weights associated with the hidden layer
    • = bias associated with RNN layer
    • = activation function

• What happens in the hidden state is that, using the information or knowledge it retained in the previous time step, the hidden state updates itself.
• The present hidden state is used in getting the output of the hidden layer, using an appropriate activation function.
  • where,
    • W = Weights assigned to the layer
    • = hidden state
    • = bias associated with output layer

• For the second layer, the output of first RNN layer is fed into it, which goes through the same process again.

This feature of Stacked RNNs enables to capture of both short-term and long-term patterns. For that reason, Stacked RNNs can learn and remember information patterns in longer sequences and at the same it can analyze their current state’s information with just learned previous state’s information. The more layers you add to your model, the stacked network will able to capture more complex patterns present in the sequential data. If your data is nested and has different types of complex patterns then Stacked RNNs will be a better model as its each layer can learn different abstractions present in your data.

Understanding Stacked RNN with code implementation

First we will implement the Stacked RNN as code implementation where we will output the predicted value using the model’s method.

Output:

1/1 [==============================] - 0s 344ms/step
Prediction from the neural network: 
 [[0.8465775]]

Now we will explore the mathematical concept behind the working of the stacked RNN

After the initial model is trained, we will get the weights associated with each layer in the architecture.

As per the equations used above, the present hidden state can be explained as a function of the activation function applied on the product of weights for input and respective inputs added to the product of weights for hidden states and hidden states with bias terms.

Output:

Prediction from manual computation: 
 [0.84657752]

Stacked RNNs refer to a special kind of RNNs that have multiple recurrent layers on top of one layer. Stacked RNNs are also called Deep RNNs for that reason. In this article, we will load the IMDB dataset and make multiple layers of SimpleRNN (stacked SimpleRNN) as an example of Stacked RNN.