Training Agent: Actor-Critic Algorithm
Let’s understand how the Actor-Critic algorithm works in practice. Below is an implementation of a simple Actor-Critic algorithm using TensorFlow and OpenAI Gym to train an agent in the CartPole environment.
Import Libraries
import numpy as np
import tensorflow as tf
import gym
2. Creating CartPole Environment
Create the CartPole environment using the gym.make() function from the Gym library because it provides a standardized and convenient way to interact with various reinforcement learning tasks.
# Create the CartPole Environment
env = gym.make('CartPole-v1')
3. Defining Actor and Critic Networks
- Actor and the Critic are implemented as neural networks using TensorFlow’s Keras API.
- Actor network maps the state to a probability distribution over actions.
- Critic network estimates the state’s value.
# Define the actor and critic networks
actor = tf.keras.Sequential([
tf.keras.layers.Dense(32, activation='relu'),
tf.keras.layers.Dense(env.action_space.n, activation='softmax')
])
critic = tf.keras.Sequential([
tf.keras.layers.Dense(32, activation='relu'),
tf.keras.layers.Dense(1)
])
4. Defining Optimizers and Loss Functions
Adam optimizer is used for both the Actor and the Critic networks.
# Define optimizer and loss functions
actor_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
critic_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
5. Training Loop
- Main training loop runs for a specified number of episodes (1000).
- Agent interacts with the environment, and for each episode, it resets the environment and initializes the episode reward to 0.
- The with tf.GradientTape block is used to compute gradients for the actor and critic networks.
- Agent chooses an action based on the actor’s output probabilities and takes that action in the environment.
- It observes the next state, reward, and whether the episode is done.
- Advantage function is computed, which is the difference between the expected return and the estimated value at the current state.
- Actor and Critic losses are calculated based on the advantage function.
- Gradients are computed using tape.gradient and then applied to update the actor and critic networks using the respective optimisers.
- Episode’s total reward is updated, and the loop continues until the episode ends.
- Every 10 episodes, the current episode number and reward are printed.
# Main training loop
num_episodes = 1000
gamma = 0.99
for episode in range(num_episodes):
state = env.reset()
episode_reward = 0
with tf.GradientTape(persistent=True) as tape:
for t in range(1, 10000): # Limit the number of time steps
# Choose an action using the actor
action_probs = actor(np.array([state]))
action = np.random.choice(env.action_space.n, p=action_probs.numpy()[0])
# Take the chosen action and observe the next state and reward
next_state, reward, done, _ = env.step(action)
# Compute the advantage
state_value = critic(np.array([state]))[0, 0]
next_state_value = critic(np.array([next_state]))[0, 0]
advantage = reward + gamma * next_state_value - state_value
# Compute actor and critic losses
actor_loss = -tf.math.log(action_probs[0, action]) * advantage
critic_loss = tf.square(advantage)
episode_reward += reward
# Update actor and critic
actor_gradients = tape.gradient(actor_loss, actor.trainable_variables)
critic_gradients = tape.gradient(critic_loss, critic.trainable_variables)
actor_optimizer.apply_gradients(zip(actor_gradients, actor.trainable_variables))
critic_optimizer.apply_gradients(zip(critic_gradients, critic.trainable_variables))
if done:
break
if episode % 10 == 0:
print(f"Episode {episode}, Reward: {episode_reward}")
env.close()
Output:
Episode 0, Reward: 29.0 Episode 10, Reward: 14.0 Episode 20, Reward: 15.0 Episode 30, Reward: 15.0 Episode 40, Reward: 31.0 Episode 50, Reward: 20.0 Episode 60, Reward: 22.0 Episode 70, Reward: 8.0 Episode 80, Reward: 51.0 Episode 90, Reward: 14.0 Episode 100, Reward: 11.0 Episode 110, Reward: 25.0 Episode 120, Reward: 16.0 ....
Actor-Critic Algorithm in Reinforcement Learning
Reinforcement learning (RL) stands as a pivotal component in the realm of artificial intelligence, enabling agents to learn optimal decision-making strategies through interaction with their environments.
Let’s Dive into the actor-critic algorithm, a key concept in reinforcement learning, and learn how it can improve your machine learning models.
Table of Content
- What is the Actor-Critic Algorithm?
- How Actor-Critic Algorithm works?
- A2C (Advantage Actor-Critic)
- Training Agent: Actor-Critic Algorithm
- Advantages of Actor Critic Algorithm
- Advantage Actor Critic (A2C) vs. Asynchronous Advantage Actor Critic (A3C)
- Conclusion