When should agents explore?

When should a reinforcement learning agent explore, and when should it exploit? This video reviews and discusses the paper When Should Agents Explore, which examines the limitations of conventional exploration strategies such as ε-greedy and argues that effective exploration should be structured, temporally extended, and state-dependent. The paper draws inspiration from biological systems, where exploration often occurs in sustained, directed phases rather than as random, moment-to-moment deviations. The discussion focuses on mechanisms for triggering and terminating exploration, as well as alternative strategies for selecting exploratory actions once exploration has begun. The paper analyzes how these choices influence learning efficiency, stability, and long-term performance, and situates the proposed framework within the broader landscape of intrinsic motivation and adaptive exploration methods. Connections are made to related ideas such as intrinsic reward signals, bandit-style adaptation, and empirical observations from animal behavior studies, highlighting how these perspectives inform more principled exploration strategies in reinforcement learning. Referenced paper: * When Should Agents Explore: https://arxiv.org/abs/2108.11811 Additional references: * Bumblebees learning foraging routes: https://royalsocietypublishing.org/doi/10.1098/rsif.2019.0103 * Random Network Distillation (RND): https://arxiv.org/abs/1810.12894 * Bandit-based adaptation methods: https://arxiv.org/abs/2003.13350 Presentation material: * Slides used in this episode: https://dry-peak.cloudvent.net/Additional-Post-Files/Youtube/WhenShouoldAgentsExplore.pdf Related videos: * Human-level control through deep reinforcement learning: https://www.youtube.com/watch?v=A0OVmImyFEA * Deterministic Policy Gradient Algorithms: https://www.youtube.com/playlist?list=PL3PIRsM7569PX_TzvDG6FaZz4KwRbIzsy