Tuesday, May 26, 2020. 12:00 PM. Link to Zoom for Online Seminar.
Thodoris Lykouris -- Corruption robust exploration in episodic reinforcement learning.
Abstract: We initiate the study of multi-stage episodic reinforcement learning under adversarial corruptions in both the rewards and the transition probabilities of the underlying system extending recent results for the special case of stochastic bandits. We provide a framework which modifies the aggressive exploration enjoyed by existing reinforcement learning approaches based on "optimism in the face of uncertainty", by complementing them with principles from "action elimination". Importantly, our framework circumvents the major challenges posed by naively applying action elimination in the RL setting, as formalized by a lower bound we demonstrate. Our framework yields efficient algorithms which (a) attain near-optimal regret in the absence of corruptions and (b) adapt to unknown levels corruption, enjoying regret guarantees which degrade gracefully in the total corruption encountered. To showcase the generality of our approach, we derive results for both tabular settings (where states and actions are finite) as well as linear-function-approximation settings (where the dynamics and rewards admit a linear underlying representation). Notably, our work provides the first sublinear regret guarantee which accommodates any deviation from purely i.i.d. transitions in the bandit-feedback model for episodic reinforcement learning.
Bio: Thodoris Lykouris is a postdoctoral researcher in the machine learning group of Microsoft Research NYC. His research focus is on online decision-making spanning across the disciplines of machine learning, theoretical computer science, operations research, and economics. He completed his Ph.D. in 2019 from Cornell University where he was advised by Eva Tardos. During his Ph.D. years, his research has been generously supported by a Google Ph.D. Fellowship and a Cornell University Fellowship. He was also a finalist in the INFORMS Nicholson and Applied Probability Society best student paper competitions.