(T) The DeepMind Technologies team in London has just published a paper in Nature: “human-level control through deep reinforcement learning”. The Deep Neural Network called  Deep Q-Network (DQN) combines a Reinforcement Learning algorithm with Experience Replay to play old good Atari games:

“DQN incorporated several key features that…enabled the power of Deep Neural Networks (DNN) to be combined in a scalable fashion with Reinforcement Learning (RL) — a machine learning framework that prescribes how (software) agents should act in an environment in order to maximize future cumulative reward (e.g., a game score). Foremost among these was a neurobiologically inspired mechanism, termed “experience replay,” whereby during the learning phase DQN was trained on samples drawn from a pool of stored episodes — a process physically realized in a brain structure called the hippocampus through the ultra-fast reactivation of recent experiences during rest periods (e.g., sleep). Indeed, the incorporation of experience replay was critical to the success of DQN: disabling this function caused a severe deterioration in performance.”

A quick explanation of reinforcement learning

• Reinforcement learning at the same time explores (learns about the game from past experiences) and exploits (maximizes rewards from known past experiences) by evaluating future game actions and optimizing the next action

A quick summary of DQN basic features

• DQN learns directly from interactive trial-and-error game playing

• DQN architecture is based on convolutional nets

• DQN inputs are the game screen pixels and previous game rewards/scores, DQN outputs are game actions/joysticks

• DQN aims to optimize the agent’s behavior, e.g. game output actions, based on future cumulative rewards through an optimal action-value function

• The same DQN algorithm is used across all games

• DQN is implemented with Torch (that got recently a contribution from Facebook)

• According to DeepMind, DQN performs at more than 75% of the level of a professional human player

I found the references to the paper extremely useful.

Inside DeepMind’s quest in playing games with Deep Learning algorithms:

References about the DeepMind project

Playing Atari with Deep Reinforcement Learning (a previous paper from DeepMind on the same topics)
Human-Level Control Through Deep Reinforcement Learning
Source code for the project: https://sites.google.com/a/deepmind.com/dqn/

References about Reinforcement Learning

A class about reinforcement learning from David Silver, one of the contributors to this project:
http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html

Stanford University Professor Andrew Ng’s class about reinforcement learning:

Note: The picture above is Atari’s Space Invaders game.

Copyright © 2005-2015 by Serge-Paul Carrasco. All rights reserved.
Contact Us: asvinsider at gmail dot com.