LSTM for Uniform Credit Assignment to Deep Networks

LSTM (Long Short-Term Memory) networks are learning systems that can be understood as a special type of recurrent artificial neural networks with an explicit memory architecture consisting of so-called memory cells and gates that control the inflow and outflow to and from the memory cells. LSTMs scan sequences of inputs (e.g. text, biological sequences) sequentially and automatically learn to recognize and memorize particular inputs or patterns of inputs that occur throughout the sequences by storing them in their memory cells. Recently, LSTM networks have emerged as the best-performing technique in speech and language processing. Recent conferences in these fields have been dominated by LSTM-based approaches, e.g. the flagship conference ICASSP. Recent benchmark records were achieved with LSTM, often by major IT companies like Google, IBM, Microsoft, and Baidu. The success of LSTM networks stems from its capability to perform uniform credit assignment to inputs, that is, LSTM allows for treating all inputs on the same level. For example, when a sentence is processed, the first word may be as important for learning as the last word. Uniform credit assignment considers all input information equally no matter where it is located in the input sequence. If learning is biased to the most recent inputs, sub-optimal solutions are often obtained. In this project, we want to go beyond uniform credit assignment to simple inputs like words. The goal of this project is to develop LSTM networks and corresponding learning techniques for uniform credit assignment to so-called deep networks which pre-process complex inputs, such as, images, speech, or chemical compounds. Such networks can be applied to the classification of actions in videos, where single frames may not convey sufficient information. That may also include photo series that show the same object from different angles, with the aim to extract features that are not visible on single images. High-content imaging of cells in drug design is another application in which a high-resolution image is split into multiple sub- images that are presented sequentially to the classification system. A further application is to predict the toxicity of a mixture of chemical compounds (e.g. a soil sample), where an unknown number of chemical structures are presented sequentially to the network. The new architectures and new approaches to LSTM-based uniform credit assignment to deep networks will be applied to the following tasks and will be benchmarked and tested on corresponding data sets: 1) video activity recognition and video description; 2) classification of large images which are split into sub-images; 3) classification of mixtures of compounds with unknown number of components, where the components are sequentially presented to the LSTM network.

Reinforcement learning (RL) is an area of machine learning in which an agent has to learn to interact with an environment such that it achieves a goal. For example, the agent can be a deep artificial neural network that has to learn how to play a computer game and maximize it's score (="reward") in the game. The field of RL is recently gaining increased public attention, with success stories such as AlphaGo, an RL program developed by DeepMind/Google, which beat the worlds best player in the board game GO, and OpenAI-Five, an RL program developed by OpenAI, which is able to competitively play the multiplayer-online-battle-arena video game Dota2. However, despite these progresses, RL is still struggling with real-world tasks and more complex strategy games, such as starcraft-II. In this project we were able to identify one of the fundamental limitations of current RL methods and were able to provide a solution to overcome this limitation. We were able to show that current methods severely suffer from long delays between actions and resulting rewards. However, real-world tasks and strategy games typically include long delays between actions and rewards. After analyzing this problem for current RL methods, we derived a novel method RUDDER, which removes the delays in rewards. It does so by training a deep learning model to predict the accumulated rewards at the end of completed state-action sequences. In other words, it predicts the outcome of a sequence by looking at the complete sequence.This is a supervised learning task and can be solved using e.g. long short-term memory networks. By doing so, the model learns which actions cause the reward. We then apply contribution analysis to determine the contributions of each action to the prediction of the model. If the model is able to predict the accumulated reward at the end of the sequence, these contributions will correspond to the contribution of the actions to the accumulated reward. As such, we can replace the original reward with these contributions, effectively moving (="redistributing") delayed rewards to the actions that caused these rewards. This redistributed reward without delays can then be used to train an RL agent, resulting in exponential speed-ups for training. As an intuitive example, let us consider the case of trying to learn how to play the piano. Current RL approaches can be compared to telling the student how good or bad the play was at the end of the play. RUDDER, however, moves the feedback from the end of the play directly to the points in the play where the student was good or bad. The student gets immediate feedback after doing something good or bad, which makes learning much easier.