An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods

Yanli Liu, Kaiqing Zhang, Tamer Başar, Wotao Yin

Type: Preprint

Publication Date: 2022-01-01

Citations: 28

DOI: https://doi.org/10.48550/arxiv.2211.07937

View Publication

Locations

arXiv (Cornell University) - View - PDF
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policies	2023	Ilyas Fatkhullin Anas Barakat Anastasia Kireeva Niao He
+	PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation	2022	Matilde Gargiani Andrea Zanelli Andrea Martinelli Tyler Summers John Lygeros
+	Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction	2024	Jie Feng Wei Ke Jinchi Chen
+	Stochastic Recursive Momentum for Policy Gradient Methods	2020	Huizhuo Yuan Xiangru Lian Ji Liu Yuren Zhou
+ PDF Chat	Stochastic Variance-Reduced Policy Gradient	2018	Matteo Papini Damiano Binaghi Giuseppe Canonaco Matteo Pirotta Marcello Restelli
+	The Role of Baselines in Policy Gradient Optimization	2023	Jincheng Mei Wesley Chung Valentin Thomas Bo Dai Csaba Szepesvári Dale Schuurmans
+	A general sample complexity analysis of vanilla policy gradient	2021	Rui Yuan Robert M. Gower Alessandro Lazaric
+ PDF Chat	Bregman Gradient Policy Optimization	2021	Feihu Huang Shangqian Gao Heng Huang
+	Bregman Gradient Policy Optimization	2021	Feihu Huang Shangqian Gao Heng Huang
+	From Importance Sampling to Doubly Robust Policy Gradient	2019	Jiawei Huang Nan Jiang
+	From Importance Sampling to Doubly Robust Policy Gradient	2019	Jiawei Huang Nan Jiang
+	Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds	2024	Gabor Paczolay Matteo Papini Alberto Maria Metelli István Harmati Marcello Restelli
+	Homotopic Policy Mirror Descent: Policy Convergence, Implicit Regularization, and Improved Sample Complexity	2022	Yan Li Tuo Zhao Guanghui Lan
+	Optimal Estimation of Off-Policy Policy Gradient via Double Fitted Iteration	2022	Chengzhuo Ni Ruiqi Zhang Ji Xiang Xuezhou Zhang Mengdi Wang
+	On the Global Convergence of Momentum-based Policy Gradient.	2021	Yuhao Ding Junzi Zhang Javad Lavaei
+	On the Global Optimum Convergence of Momentum-based Policy Gradient	2021	Yuhao Ding Junzi Zhang Javad Lavaei
+	Zeroth-order Deterministic Policy Gradient.	2020	Harshat Kumar Dionysios S. Kalogerias George J. Pappas Alejandro Ribeiro
+	Zeroth-order Deterministic Policy Gradient	2020	Harshat Kumar Dionysios S. Kalogerias George J. Pappas Alejandro Ribeiro
+	An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient	2019	Pan Xu Felicia Gao Quanquan Gu
+	An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradient	2019	Pan Xu Felicia Gao Quanquan Gu

Works That Cite This (24)

Action	Title	Year	Authors
+ PDF Chat	Recent Advances in Reinforcement Learning in Finance	2021	Ben Hambly Renyuan Xu Huining Yang
+	Finite-Sample Analysis of Off-Policy Natural Actor–Critic With Linear Function Approximation	2022	Zaiwei Chen Sajad Khodadadian Siva Theja Maguluri
+	Local Analysis of Entropy-Regularized Stochastic Soft-Max Policy Gradient Methods	2023	Yuhao Ding Junzi Zhang Javad Lavaei
+	On the Global Convergence of Momentum-based Policy Gradient.	2021	Yuhao Ding Junzi Zhang Javad Lavaei
+ PDF Chat	Smoothed functional-based gradient algorithms for off-policy reinforcement learning: A non-asymptotic viewpoint	2021	Nithia Vijayan L. A. Prashanth
+ PDF Chat	Independent Natural Policy Gradient Methods for Potential Games: Finite-time Global Convergence with Entropy Regularization	2022	Shicong Cen Fan Chen Yuejie Chi
+ PDF Chat	On the Linear Convergence of Natural Policy Gradient Algorithm	2021	Sajad Khodadadian Prakirt Raj Jhunjhunwala Sushil Mahavir Varma Siva Theja Maguluri
+ PDF Chat	Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settings	2022	Matthew S. Zhang Murat A. Erdogdu Animesh Garg
+ PDF Chat	A Robust and Constrained Multi-Agent Reinforcement Learning Electric Vehicle Rebalancing Method in AMoD Systems	2023	Sihong He Yue Wang Shuo Han Shaofeng Zou Fei Miao
+ PDF Chat	Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm	2023	Qinbo Bai Amrit Singh Bedi Vaneet Aggarwal

Works Cited by This (18)

Action	Title	Year	Authors
+	Robust Stochastic Approximation Approach to Stochastic Programming	2009	Arkadi Nemirovski Anatoli Juditsky Guanghui Lan Alexander Shapiro
+ PDF Chat	Conditioning of quasi-Newton methods for function minimization	1970	David F. Shanno
+ PDF Chat	A family of variable-metric methods derived by variational means	1970	Donald Goldfarb
+ PDF Chat	A new approach to variable metric algorithms	1970	R. Fletcher
+	The Convergence of a Class of Double-rank Minimization Algorithms 1. General Considerations	1970	C. G. Broyden
+	The Exponential Family and Statistical Applications	2011	Anirban Dasgupta
+	Benchmarking Deep Reinforcement Learning for Continuous Control	2016	Yan Duan Xi Chen Rein Houthooft John Schulman Pieter Abbeel
+	Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator	2018	Maryam Fazel Rong Ge Sham M. Kakade Mehran Mesbahi
+	Acceleration of SVRG and Katyusha X by Inexact Preconditioning	2019	Yanli Liu Fei Feng Wotao Yin
+	SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives	2014	Aaron Defazio Francis Bach Simon Lacoste-Julien