Verification of Reinforcement Learning Models:

Abstract

In recent years of advancements in reinforcement learning (RL), utilizing neural network based models to make decisions in dynamic and complex environments has emerged as a powerful paradigm. In particular, model based reinforcement learning has been widely used for its ability to increase learning efficiency and performance. By constructing an environment model beforehand, the agent attains a prior knowledge of the dynamics of the model to take informed decisions and converge fast to optimal policies. Real-world environments are often intricate and subject to external disturbances, posing substantial challenges for accurate modeling. Addressing these challenges requires the application of sophisticated neural network-based models that can effectively approximate the underlying environment dynamics. In this work, we develop and evaluate extensive neural network models, specifically focusing on Gaussian Ensemble models, Bayesian neural networks, and Monte Carlo Dropout techniques, to approximate various standard gym environments. These models are trained on different numbers of samples to understand their efficiency and accuracy in capturing environment dynamics. Once trained, the neural network models are used to construct Markov Decision Processes (MDPs) with various discretization strategies. The constructed MDPs are then analyzed and compared to evaluate the performance of each neural network approach. The purpose of this thesis is to present a comprehensive study on the construction of environment models using advanced neural network techniques. We aim to approximate the standard environments in the reinforcement learning setup, utilizing a variety of neural networks and compare the efficiency based on the reconstruction of MDPs.