Foundations and Trends® in Systems and Control > Vol 8 > Issue 1–2

Synchronous Reinforcement Learning-Based Control for Cognitive Autonomy

Kyriakos G. Vamvoudakis, Georgia Institute of Technology, USA, kyriakos@gatech.edu , Nick-Marios T. Kokolakis, Georgia Institute of Technology, USA, nmkokolakis@gatech.edu
 
Suggested Citation
Kyriakos G. Vamvoudakis and Nick-Marios T. Kokolakis (2020), "Synchronous Reinforcement Learning-Based Control for Cognitive Autonomy", Foundations and Trends® in Systems and Control: Vol. 8: No. 1–2, pp 1-175. http://dx.doi.org/10.1561/2600000022

Publication Date: 12 Nov 2020
© 2020 Kyriakos G. Vamvoudakis and Nick-Marios T. Kokolakis
 
Subjects
Optimal Control,  Reinforcement learning,  Algorithmic game theory
 

Free Preview:

Download extract

Share

Download article
In this article:
1. Introduction
2. Optimal Regulation
3. Game-Theoretic Learning
4. Model-Free RL with Q-Learning
5. Model-Based and Model-Free Intermittent RL
6. Bounded Rationality and Non-Equilibrium RL in Games
7. Applications to Autonomous Vehicles
8. Concluding Remarks
Acknowledgements
References

Abstract

This monograph provides an exposition of recently developed reinforcement learning-based techniques for decision and control in human-engineered cognitive systems. The developed methods learn the solution to optimal control, zero-sum, non zero-sum, and graphical game problems completely online by using measured data along the system trajectories and have proved stability, optimality, and robustness. It is true that games have been shown to be important in robust control for disturbance rejection, and in coordinating activities among multiple agents in networked teams. We also consider cases with intermittent (an analogous to triggered control) instead of continuous learning and apply those techniques for optimal regulation and optimal tracking. We also introduce a bounded rational model to quantify the cognitive skills of a reinforcement learning agent. In order to do that, we leverage ideas from behavioral psychology to formulate differential games where the interacting learning agents have different intelligence skills, and we introduce an iterative method of optimal responses that determine the policy of an agent in adversarial environments. Finally, we present applications of reinforcement learning to motion planning and collaborative target tracking of bounded rational unmanned aerial vehicles.

DOI:10.1561/2600000022
ISBN: 978-1-68083-744-5
186 pp. $99.00
Buy book (pb)
 
ISBN: 978-1-68083-745-2
186 pp. $280.00
Buy E-book (.pdf)
Table of contents:
1. Introduction
2. Optimal Regulation
3. Game-Theoretic Learning
4. Model-Free RL with Q-Learning
5. Model-Based and Model-Free Intermittent RL
6. Bounded Rationality and Non-Equilibrium RL in Games
7. Applications to Autonomous Vehicles
8. Concluding Remarks
Acknowledgements
References

Synchronous Reinforcement Learning-Based Control for Cognitive Autonomy

This monograph describes the use of principles of reinforcement learning (RL) to design feedback policies for continuous-time dynamical systems that combine features of adaptive control and optimal control. In a control engineering context, RL bridges the gap between traditional optimal control and adaptive control algorithms.

The authors give an insightful introduction to reinforcement learning techniques that can address various control problems. In this context, they give a detailed description of techniques such as Game-Theoretic Learning, Q-learning, and Intermittent RL; with each chapter providing a self-contained exposition of the topic and giving the reader suggestions for further reading. Finally, the authors demonstrate the application of the techniques in autonomous vehicles.

This review of a topic that is rapidly becoming ubiquitous in many engineering systems enables to reader dip in and out of the topic to quickly understand the essentials and provides the starting point for further research.

 
SYS-022