PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems
Authors: David Biagioni, Xiangyu Zhang, Dylan Wald, Deepthi Vaidhynathan, Rhoit Chintala, Jennifer King, Ahmed S. Zamzam
Corresponding author: David Biagioni
All authors are with the National Renewable Energy Laboratory (NREL).
Description
PowerGridworld provides users with a lightweight, modular, and customizable framework for creating power-systems-focused, multi-agent Gym environments that readily integrate with existing training frameworks for reinforcement learning (RL). Although many frameworks exist for training multi-agent RL (MARL) policies, none can rapidly prototype and develop the environments themselves, especially in the context of heterogeneous (composite, multidevice) power systems where power flow solutions are required to define grid-level variables and costs. PowerGridworld is an opensource software package that helps to fill this gap. To highlight PowerGridworld’s key features, we include two case studies and demonstrate learning MARL policies using both OpenAI’s multi-agent deep deterministic policy gradient (MADDPG) and RLLib’s proximal policy optimization (PPO) algorithms. In both cases, at least some subset of agents incorporates elements of the power flow solution at each time step as part of their reward (negative cost) structures.
Please refer to our preprint on arXiv for more details. Data and run scripts used to generate figures in the preprint are available in the paper directory.
Basic installation instructions
Env setup:
conda create -n gridworld python=3.8 -y
conda activate gridworld
git clone [email protected]:NREL/PowerGridworld.git
cd PowerGridWorld
pip install -e .
pip install -r requirements.txt
Run the pytests to sanity check:
pytest tests/
pytests --nbmake examples/envs
Examples
Examples of running various environments and MARL training algorithms can be found in examples.
Funding Acknowledgement
This work was authored by the National Renewable Energy Laboratory (NREL), operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. This work was supported by the Laboratory Directed Research and Development (LDRD) Program at NREL.
Citation
If citing this work, please use the following:
@article{biagioni2021powergridworld,
title={PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems},
author={Biagioni, David and Zhang, Xiangyu and Wald, Dylan and Vaidhynathan, Deepthi and Chintala, Rohit and King, Jennifer and Zamzam, Ahmed S},
journal={arXiv preprint arXiv:2111.05969},
year={2021}
}
1 Dec 17, 2021
56 Nov 15, 2022
8 Sep 30, 2022
1.1k Jan 2, 2023
669 Dec 29, 2022
1 May 15, 2022
0 Jul 26, 2022
183 Dec 27, 2022
8 Nov 7, 2022
184 Dec 12, 2022
2 Nov 28, 2021
21 Apr 09, 2022
4.4k Dec 27, 2022
8 Apr 11, 2022
6.1k Dec 30, 2022
2.3k Jan 09, 2023
730 Jan 09, 2023
4 Jan 24, 2022
594 Jan 05, 2023
242 Dec 12, 2022
36 Dec 12, 2022
28 Dec 09, 2022
656 Dec 29, 2022
5 Jun 16, 2022
4 Feb 15, 2022
211 Dec 31, 2022
67 Nov 22, 2022
1 Oct 30, 2021
3 Nov 30, 2021