Shixiang Gu1 2 3 SG717@CAM.AC.UK
Timothy Lillicrap4 COUNTZERO@GOOGLE.COM
Ilya Sutskever3 ILYASU@GOOGLE.COM
Sergey Levine3 SLEVINE@GOOGLE.COM
1 University of Cambridge 2Max Planck Institute for Intelligent Systems 3Google Brain 4Google DeepMind
Abstract
Model-free reinforcement learning has been successfully applied to a range of challenging problems, and has recently been extended to handle large neural network policies and value functions. However, the sample complexity of model free algorithms, particularly when using high dimensional function approximators, tends to limit their applicability to physical systems. In this paper, we explore algorithms and representations to reduce the sample complexity of deep reinforcement learning for continuous control tasks. We propose two complementary techniques for improving the efficiency of such algorithms. First, we derive a continuous variant of the Q-learning algorithm, which we call normalised advantage functions (NAF), as an alternative to the more commonly used policy gradient and actor-critic methods. NAF representation allows us to apply Q-learning with experience replay to continuous tasks, and substantially improves performance on a set of simulated robotic control tasks. To further improve the efficiency of our approach, we explore the use of learned models for accelerating model-free reinforcement learning. We show that iteratively refitted local linear models are especially effective for this, and demonstrate substantially faster learning on domains where such models are applicable.
