Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control

Kendall Lowrey, Aravind Rajeswaran, Sham M. Kakade, Emanuel Todorov, Igor Mordatch

Type: Preprint

Publication Date: 2018-11-05

Citations: 62

View

Locations

arXiv (Cornell University) - View

Similar Works

Action	Title	Year	Authors
+	Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control	2018	Kendall Lowrey Aravind Rajeswaran Sham M. Kakade Emanuel Todorov Igor Mordatch
+	COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL	2023	Xiyao Wang Ruijie Zheng Yanchao Sun Ruonan Jia Wichayaporn Wongkamjan Huazhe Xu Furong Huang
+	Blending MPC & Value Function Approximation for Efficient Reinforcement Learning	2020	Mohak Bhardwaj Sanjiban Choudhury Byron Boots
+	Blending MPC & Value Function Approximation for Efficient Reinforcement Learning	2020	Mohak Bhardwaj Sanjiban Choudhury Byron Boots
+	Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning	2020	Michelle A. Lee Carlos Florensa Jonathan Tremblay Nathan Ratliff Animesh Garg Fábio Ramos Dieter Fox
+	Exploration via Planning for Information about the Optimal Trajectory	2022	Viraj Mehta Ian Char Joseph Abbate Rory Conlin Mark D. Boyer Stefano Ermon Jeff Schneider Willie Neiswanger
+	Improving Model-Based Control and Active Exploration with Reconstruction Uncertainty Optimization	2018	Norman Di Palo Harri Valpola
+	Model-Based Offline Planning	2020	Arthur Argenson Gabriel Dulac-Arnold
+ PDF Chat	Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning	2020	Michelle A. Lee Carlos Florensa Jonathan Tremblay Nathan Ratliff Animesh Garg Fábio Ramos Dieter Fox
+	Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning	2020	Michelle A. Lee Carlos Florensa Jonathan Tremblay Nathan Ratliff Animesh Garg Fábio Ramos Dieter Fox
+	A survey on policy search algorithms for learning robot controllers in a handful of trials	2018	Konstantinos Chatzilygeroudis Vassilis Vassiliades Freek Stulp Sylvain Calinon Jean-Baptiste Mouret
+	Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics	2021	Ingmar Schubert Danny Driess Ozgur S. Oguz Marc Toussaint
+ PDF Chat	Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics	2021	Ingmar Schubert Danny Driess Ozgur S. Oguz Marc Toussaint
+	Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics	2021	Ingmar Schubert Danny Driess Ozgur S. Oguz Marc Toussaint
+	Model-Based Offline Planning	2020	Arthur Argenson Gabriel Dulac-Arnold
+	Model-based Lookahead Reinforcement Learning	2019	Zhang-Wei Hong Joni Pajarinen Jan Peters
+	LVIS: Learning from Value Function Intervals for Contact-Aware Robot Controllers	2018	Robin Deits Twan Koolen Russ Tedrake
+	LVIS: Learning from Value Function Intervals for Contact-Aware Robot Controllers	2018	Robin Deits Twan Koolen Russ Tedrake
+	Regularizing Model-Based Planning with Energy-Based Models	2019	Rinu Boney Juho Kannala Alexander Ilin
+	Model Learning for Look-ahead Exploration in Continuous Control	2018	Arpit Agarwal Katharina Muelling Katerina Fragkiadaki

Cited by (52)

Action	Title	Year	Authors
+	Deep Dynamics Models for Learning Dexterous Manipulation	2019	Anusha Nagabandi Kurt Konoglie Sergey Levine Vikash Kumar
+	Model-Augmented Actor-Critic: Backpropagating through Paths	2020	Ignasi Clavera Violet Fu Pieter Abbeel
+	Safe Interactive Model-Based Learning	2019	Marco Gallieri Seyed Sina Mirrazavi Salehian Nihat Engin Toklu Alessio Quaglino Jonathan Masci Jan Koutník Faustino Gomez
+	Learning Physics-Based Manipulation in Clutter: Combining Image-Based Generalization and Look-Ahead Planning	2019	Wissam Bejjani Mehmet R. Doğar Matteo Leonetti
+ PDF Chat	Learning Physics-Based Manipulation in Clutter: Combining Image-Based Generalization and Look-Ahead Planning	2019	Wissam Bejjani Mehmet R. Doğar Matteo Leonetti
+	Improving Model-Based Control and Active Exploration with Reconstruction Uncertainty Optimization	2018	Norman Di Palo Harri Valpola
+	Multi-Objective Graph Heuristic Search for Terrestrial Robot Design	2021	Jie Xu Andrew Spielberg Allan Zhao Daniela Rus Wojciech Matusik
+	AI and Wargaming	2020	James Goodman Sebastian Risi Simon M. Lucas
+	Planning to Explore via Self-Supervised World Models	2020	R. Sekar Oleh Rybkin Kostas Daniilidis Pieter Abbeel Danijar Hafner Deepak Pathak
+ PDF Chat	Comparison of Deep Reinforcement Learning and Model Predictive Control for Adaptive Cruise Control	2020	Yuan Lin John McPhee Nasser L. Azad
+	Learning Off-Policy with Online Planning	2020	Harshit Sikchi Wenxuan Zhou David Held
+	Regularizing Trajectory Optimization with Denoising Autoencoders	2019	Rinu Boney Norman Di Palo Mathias Berglund Alexander Ilin Juho Kannala Antti Rasmus Harri Valpola
+	Project Thyia: A Forever Gameplayer	2019	Raluca D. Gaina Simon M. Lucas Diego Pérez-Liébana
+	Discriminator Augmented Model-Based Reinforcement Learning	2021	Behzad Haghgoo Allan Zhou Archit Sharma Chelsea Finn
+	ABC-LMPC: Safe Sample-Based Learning MPC for Stochastic Nonlinear Dynamical Systems with Adjustable Boundary Conditions	2020	Brijen Thananjeyan Ashwin Balakrishna Ugo Rosolia Joseph E. Gonzalez Aaron D. Ames Ken Goldberg
+	Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrians	2021	Bruno Brito Michael Everett Jonathan P. How Javier Alonso‐Mora
+	Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs	2020	Jianzhun Du Joseph Futoma Finale Doshi‐Velez
+	Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning	2022	Jinxin Liu Donglin Wang Qiangxing Tian Zhengyu Chen
+	From proprioception to long-horizon planning in novel environments: A hierarchical RL model.	2020	Nishad Gothoskar Miguel Lázaro-Gredilla Dileep George
+ PDF Chat	Multi-Objective Graph Heuristic Search for Terrestrial Robot Design	2021	Jie Xu Andrew Spielberg Allan Zhao Daniela Rus Wojciech Matusik
+	Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration	2021	Oliver Groth Markus Wulfmeier Giulia Vezzani Vibhavari Dasagi Tim Hertweck Roland Hafner Nicolas Heess Martin Riedmiller
+ PDF Chat	Safety Augmented Value Estimation From Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks	2020	Brijen Thananjeyan Ashwin Balakrishna Ugo Rosolia Felix Li Rowan McAllister Joseph E. Gonzalez Sergey Levine Francesco Borrelli Ken Goldberg
+	Dream to Control: Learning Behaviors by Latent Imagination	2019	Danijar Hafner Timothy Lillicrap Jimmy Ba Mohammad Norouzi
+	Safe Model-Based Off-Policy Reinforcement Learning for Eco-Driving in Connected and Automated Hybrid Electric Vehicles	2022	Zhaoxuan Zhu Nicola Pivaro Shobhit Gupta Abhishek Gupta Marcello Canova
+	Safely Learning Dynamical Systems from Short Trajectories	2020	Amir Ali Ahmadi Abraar Chaudhry Vikas Sindhwani Stephen Tu
+	Self-Adapting Recurrent Models for Object Pushing from Learning in Simulation	2020	Cong Lin Michael Görner Philipp Ruppel Hongzhuo Liang Norman Hendrich Jianwei Zhang
+	Blending MPC & Value Function Approximation for Efficient Reinforcement Learning	2020	Mohak Bhardwaj Sanjiban Choudhury Byron Boots
+	Continuous Control With Ensemble Deep Deterministic Policy Gradients	2021	Piotr Januszewski Mateusz Olko Michał Królikowski Jakub Świątkowski Marcin Andrychowicz Łukasz Kuciński Piotr Miłoś
+	Optimization of the model predictive control meta-parameters through reinforcement learning	2023	Eivind Bøhn Sébastien Gros Signe Moe Tor Arne Johansen
+ PDF Chat	Mini Cheetah, the Falling Cat: A Case Study in Machine Learning and Trajectory Optimization for Robot Acrobatics	2022	Vince Kurtz He Li Patrick M. Wensing Hai Lin
+	Reinforcement Learning of the Prediction Horizon in Model Predictive Control	2021	Eivind Bøhn Sébastien Gros Signe Moe Tor Arne Johansen
+	Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning	2021	Andrew S. Morgan Daljeet Nandha Georgia Chalvatzaki Carlo D’Eramo Aaron M. Dollar Jan Peters
+	Disentangled Attention as Intrinsic Regularization for Bimanual Multi-Object Manipulation.	2021	Minghao Zhang Pingcheng Jian Yi Wu Huazhe Xu Xiaolong Wang
+	A novel approach to model exploration for value function learning	2019	Zlatan Ajanović Halil Beglerovic Bakir Lačević
+	Information Theoretic Regret Bounds for Online Nonlinear Control	2020	Sham M. Kakade Akshay Krishnamurthy Kendall Lowrey Motoya Ohnishi W. Sun
+	Information Theoretic Model Predictive Q-Learning	2020	Mohak Bhardwaj Ankur Handa Dieter Fox Byron Boots
+	Dream to Control: Learning Behaviors by Latent Imagination.	2019	Danijar Hafner Timothy Lillicrap Jimmy Ba Mohammad Norouzi
+ PDF Chat	Self-Adapting Recurrent Models for Object Pushing from Learning in Simulation	2020	Cong Lin Michael Grner Philipp Ruppel Hongzhuo Liang Norman Hendrich Jianwei Zhang
+ PDF Chat	Learning Coordinated Terrain-Adaptive Locomotion by Imitating a Centroidal Dynamics Planner	2022	Philémon Brakel Steven Bohez Leonard Hasenclever Nicolas Heess Konstantinos Bousmalis
+	Reset-Free Lifelong Learning with Skill-Space Planning	2020	Kevin Lü Aditya Grover Pieter Abbeel Igor Mordatch

Citing (18)

Action	Title	Year	Authors
+	High-Dimensional Continuous Control Using Generalized Advantage Estimation	2015	John Schulman Philipp Moritz Sergey Levine Michael I. Jordan Pieter Abbeel
+ PDF Chat	A contextual-bandit approach to personalized news article recommendation	2010	Lihong Li Wei Chu John Langford Robert E. Schapire
+	VIME: Variational Information Maximizing Exploration	2016	Rein Houthooft Xi Chen Yan Duan John Schulman Filip De Turck Pieter Abbeel
+	Unifying Count-Based Exploration and Intrinsic Motivation	2016	Marc G. Bellemare Sriram Srinivasan Georg Ostrovski Tom Schaul David Saxton Rémi Munos
+	Curiosity-driven Exploration by Self-supervised Prediction	2017	Deepak Pathak Pulkit Agrawal Alexei A. Efros Trevor Darrell
+	Thinking Fast and Slow with Deep Learning and Tree Search	2017	Thomas Anthony Tian Zheng David Barber
+	Parameter Space Noise for Exploration	2017	Matthias Plappert Rein Houthooft Prafulla Dhariwal Szymon Sidor Richard Y. Chen Xi Chen Tamim Asfour Pieter Abbeel Marcin Andrychowicz
+	Noisy Networks for Exploration	2017	Meire Fortunato Mohammad Gheshlaghi Azar Bilal Piot Jacob Menick Ian Osband Alex Graves Vlad Mnih Rémi Munos Demis Hassabis Olivier Pietquin
+	Universal Planning Networks	2018	Aravind Srinivas Allan Jabri Pieter Abbeel Sergey Levine Chelsea Finn
+	Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learning	2018	Wen Sun J. Andrew Bagnell Byron Boots
+	Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion	2018	Jacob Buckman Danijar Hafner George Tucker Eugene Brevdo Honglak Lee
+ PDF Chat	Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning	2018	Anusha Nagabandi Gregory Kahn Ronald S. Fearing Sergey Levine
+	A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning	2010	Stéphane Ross Geoffrey J. Gordon J. Andrew Bagnell
+ PDF Chat	Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system	2018	Kendall Lowrey Svetoslav Kolev Jeremy Dao Aravind Rajeswaran Emanuel Todorov
+	Asynchronous Methods for Deep Reinforcement Learning	2016	Volodymyr Mnih Adrià Puigdomènech Badia Mehdi Mirza Alex Graves Tim Harley Timothy Lillicrap David Silver Koray Kavukcuoglu
+ PDF Chat	Learning dexterous in-hand manipulation	2019	OpenAI Marcin Andrychowicz Bowen Baker Maciek Chociej Rafał Józefowicz Bob McGrew Jakub Pachocki Arthur J Petron Matthias Plappert Glenn Powell Alex Ray
+ PDF Chat	Sim-to-Real Transfer of Robotic Control with Dynamics Randomization	2018	Xue Bin Peng Marcin Andrychowicz Wojciech Zaremba Pieter Abbeel
+	Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models	2015	Bradly C. Stadie Sergey Levine Pieter Abbeel