Knowledge Sharing for Reinforcement Learning:
Writing a BOOK
Simyung Chang1
, YoungJoon Yoo2
, Jaeseok Choi1
, Nojun Kwak1
Seoul National University
1{timelighter, jaeseok.choi, nojunk}@snu.ac.kr, 2yjyoo3312@gmail.com
Abstract
This paper proposes a novel deep reinforcement learning (RL) method integrating
the neural-network-based RL and the classical RL based on dynamic programming.
In comparison to the conventional deep RL methods, our method enhances
the convergence speed and the performance by delving into the following two
characteristic features in the training of conventional RL: (1) Having many credible
experiences is important in training RL algorithms, (2) Input states can be
semantically clustered into a relatively small number of core clusters, and the
states belonging to the same cluster tend to share similar Q-values given an action.
By following the two observations, we propose a dictionary-type memory that
accumulates the Q-value for each cluster of states as well as the corresponding
action, in terms of priority. Then, we iteratively update each Q-value in the memory
from the Q-value acquired from the network trained by the experiences stored in
the memory. We demonstrate the effectiveness of our method through training RL
algorithms on widely used game environments from OpenAI.