Scenic4RL: Programmatic Modeling and Generation of Reinforcement\n Learning Environments

The capability of a reinforcement learning (RL) agent heavily depends on the\ndiversity of the learning scenarios generated by the environment. Generation of\ndiverse realistic scenarios is challenging for real-time strategy (RTS)\nenvironments. The RTS environments are characterized by intelligent\nentities/non-RL agents cooperating and competing with the RL agents with large\nstate and action spaces over a long period of time, resulting in an infinite\nspace of feasible, but not necessarily realistic, scenarios involving complex\ninteraction among different RL and non-RL agents. Yet, most of the existing\nsimulators rely on randomly generating the environments based on predefined\nsettings/layouts and offer limited flexibility and control over the environment\ndynamics for researchers to generate diverse, realistic scenarios as per their\ndemand. To address this issue, for the first time, we formally introduce the\nbenefits of adopting an existing formal scenario specification language,\nSCENIC, to assist researchers to model and generate diverse scenarios in an RTS\nenvironment in a flexible, systematic, and programmatic manner. To showcase the\nbenefits, we interfaced SCENIC to an existing RTS environment Google Research\nFootball(GRF) simulator and introduced a benchmark consisting of 32 realistic\nscenarios, encoded in SCENIC, to train RL agents and testing their\ngeneralization capabilities. We also show how researchers/RL practitioners can\nincorporate their domain knowledge to expedite the training process by\nintuitively modeling stochastic programmatic policies with SCENIC.\n