Prioritized Experience Replay(PER)
Prioritized Experience Replay (PER) is a technique for improving Deep Q-Networks (DQN) described in “Overview of Deep Q-Network (DQN) and Examples of Algorithms and Implementations“, a type of reinforcement learning. ) and is usually learned by reusing them, usually by randomly sampling from the experience replay buffer, but PER improves on this and becomes a way to prioritize learning important experiences.
The main idea of PER is that when an agent learns from past experiences, it samples based on the importance of the past experiences, and specifically follows these steps
1. calculate the importance of the experience: For each experience, a priority is calculated that assesses how important the experience is to the agent’s learning. The general method is to calculate the importance using the TD error (Temporal Difference error) described in “Overview of Temporal Difference Error (TD error) and related algorithms and implementation examples” of the experience and the prediction error of the reward.
2. Sampling based on priority: Experiences with high importance have a higher probability of being sampled preferentially. This will provide more learning opportunities for important experiences.
3. use of sampled experiences: Sampled experiences are used to update the agent’s value function. This allows the agent to acquire more knowledge from key experiences and learning proceeds more efficiently.
PER has typically been shown to improve DQN performance and is especially useful in environments where the replay buffer size is large or learning is difficult to converge; while PER improves the efficiency of experience replay, increasing convergence speed and learning stability, proper hyperparameter settings are important and incorrect settings can negatively impact learning stability.
Algorithm used for Prioritized Experience Replay
The algorithm used for Prioritized Experience Replay (PER) refers to the specific method used to calculate the priority of experience replay and adjust sampling, and in general, the following algorithms are used for PER
1. Priority Calculation: An algorithm is needed to calculate the priority of experience, and common priority calculation methods include priority based on TD error (Temporal Difference error) and priority based on reward prediction error. error, the higher the prediction error of the state value after updating the experience, the higher the priority of the experience.
2. Priority Sampling: A method for sampling experience based on priority is needed. Usually, probabilistic sampling methods are used, and experiences with higher priority are more likely to be selected. A common method is to sample using a probability distribution based on priority.
3. Importance Sampling Correction: Priority-based sampling experiences may introduce bias compared to normal random sampling, so a method to correct for bias is needed. Importance Sampling Correction is used to adjust the weights of the sampled experience to reduce bias.
The PER algorithm typically calculates priorities when updating experiences in the experience replay buffer, and sampling is based on priorities when learning. In this way, important experiences are used for training more frequently, which is expected to increase learning efficiency and improve performance.
Common PER algorithms include Proportional Prioritization, Rank-Based Prioritization, and others, and there are many variations on specific implementations. The algorithm and hyperparameter settings chosen depend on the specific task and environment.
Example implementation of Prioritized Experience Replay
Prioritized Experience Replay (PER) implementations will generally be based on reinforcement learning libraries and frameworks using Python. Below are the general steps for implementing PER and an example implementation in Python. This example shows a PER implementation of DQN using OpenAI Gym and TensorFlow.
See also “New Developments in Reinforcement Learning (2) – Approaches Using Deep Learning” for details.
import numpy as np
import tensorflow as tf
import gym
# definition
REPLAY_BUFFER_SIZE = 10000
BATCH_SIZE = 32
ALPHA = 0.6 # Hyperparameters for adjusting priority weights
BETA = 0.4 # Hyperparameters for importance sampling correction
# Experience Replay buffer class
class PrioritizedReplayBuffer:
def __init__(self, capacity):
self.buffer = []
self.priorities = np.zeros(capacity)
self.capacity = capacity
self.pos = 0
def add(self, experience, priority):
if len(self.buffer) < self.capacity:
self.buffer.append(experience)
else:
self.buffer[self.pos] = experience
self.priorities[self.pos] = priority
self.pos = (self.pos + 1) % self.capacity
def sample(self, batch_size):
priorities = self.priorities[:len(self.buffer)]
probs = priorities ** ALPHA
probs /= probs.sum()
indices = np.random.choice(len(self.buffer), batch_size, p=probs)
samples = [self.buffer[i] for i in indices]
weights = (len(self.buffer) * probs[indices]) ** (-BETA)
weights /= weights.max()
return samples, indices, weights
def update_priorities(self, indices, priorities):
for i, priority in zip(indices, priorities):
self.priorities[i] = priority
# Class of DQN agents
class DQNAgent:
def __init__(self, state_dim, action_dim):
# Definition of Neural Network
# ...
def train(self, states, actions, rewards, next_states, dones):
# Calculate priority
td_errors = self.compute_td_errors(states, actions, rewards, next_states, dones)
priorities = np.abs(td_errors) + 1e-6
# Experience added to replay buffer
for i in range(len(states)):
self.replay_buffer.add((states[i], actions[i], rewards[i], next_states[i], dones[i]), priorities[i])
# Sampling batches
batch, indices, weights = self.replay_buffer.sample(BATCH_SIZE)
# Update Neural Networks
# ...
# Update priority
new_td_errors = self.compute_td_errors(states, actions, rewards, next_states, dones)
new_priorities = np.abs(new_td_errors) + 1e-6
self.replay_buffer.update_priorities(indices, new_priorities)
# Setting up the environment
env = gym.make('CartPole-v1')
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n
# Initialization of DQN agents
agent = DQNAgent(state_dim, action_dim)
agent.replay_buffer = PrioritizedReplayBuffer(REPLAY_BUFFER_SIZE)
# learning loop
for episode in range(EPISODES):
state = env.reset()
done = False
while not done:
# Action Selection
action = agent.select_action(state)
next_state, reward, done, _ = env.step(action)
# learning
agent.train(state, action, reward, next_state, done)
state = next_state
This example implements a DQN agent running in OpenAI Gym’s CartPole environment and shows how PER can be used to replay experiences. key elements of PER include priority calculation, sampling, and importance sampling correction. hyperparameters of PER (ALPHA, BETA) adjustment is also a key component.
Challenge for Prioritized Experience Replay
Prioritized Experience Replay (PER) is an improved technique for experience replay in reinforcement learning that is expected to improve performance, but also presents some challenges. The following describes some of the challenges of PER. 1.
1. Tuning of hyperparameters: Effective use of PER requires tuning of hyperparameters. For example, it can be difficult to properly set the priority weight adjustment parameter (ALPHA) or the importance sampling correction parameter (BETA), and incorrect hyperparameter settings can adversely affect study stability.
2. over-prioritization: In PER, some experiences are given high priority and these experiences tend to be selected frequently. This can cause some experiences to over-learn, leading to learning instability.
3. memory usage: Because of the need to retain priorities for priority sampling, PER typically increases memory usage compared to regular random sampling. Memory requirements may be higher when using large experience buffers.
4. introduction of delay: Priority sampling may introduce a delay in sampling experience. Prioritized experiences are selected more frequently, increasing the time it takes for new experiences to be reflected in learning.
5. implementation complexity: Correct implementation of PER requires dealing with complex algorithms for priority calculation, sampling, and importance sampling correction. Incorrect implementation will negatively impact learning stability.
Prioritized Experience Replay’s Response to Challenges
Several approaches and improvements have been proposed to address the challenges of Prioritized Experience Replay (PER). The following describes the main challenges of PER and ways to address them.
1. Hyperparameter Tuning:
Automatic Tuning: Methods to automate the tuning of hyperparameters may be employed. For example, hyperparameter optimization algorithms can be used to find appropriate ALPHA and BETA values.
2. overprioritization:
Priority clipping: excessive priority assignment can be controlled by setting a priority limit. This can mitigate excessive attention to some experiences.
3. memory usage:
Subsampling: Memory usage can be reduced by using a method of subsampling some experiences from the experience buffer. Sometimes it is not necessary to store all experiences while focusing on the important ones.
4. introducing delays:
PRIORITIZED EXPERIENCE REPLAY WITH DELAYED UPDATES (PERDUE): PERDUE introduces delayed updates to reduce excessive latency and improve learning stability while maintaining the effectiveness of PER.
5. implementation complexity:
Use of libraries and frameworks: Reinforcement learning libraries and frameworks are sometimes used to simplify the implementation of PER. This can simplify the implementation of priority computation and sampling.
References and Reference Books
Details of reinforcement learning are described in “Theories and Algorithms of Various Reinforcement Learning Techniques and Their Python Implementations. Please also refer to this page.
A reference book is “Reinforcement Learning: An Introduction, Second Edition.
“Deep Reinforcement Learning with Python: With PyTorch, TensorFlow and OpenAI Gym“
コメント