Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning

Daniel Tabas, Ahmed Zamzam, Baosen Zhang

Research output: Contribution to conferencePaperpeer-review


Constrained multiagent reinforcement learning (C-MARL) is gaining importance as MARL algorithms find new applications in real-world systems ranging from energy systems to drone swarms. Most C-MARL algorithms use a primal-dual approach to enforce constraints through a penalty function added to the reward. In this paper, we study the structural effects of this penalty term on the MARL problem. First, we show that the standard practice of using the constraint function as the penalty leads to a weak notion of safety. However, by making simple modifications to the penalty term, we can enforce meaningful probabilistic (chance and conditional value at risk) constraints. Second, we quantify the effect of the penalty term on the value function, uncovering an improved value estimation procedure. We use these insights to propose a constrained multiagent advantage actor critic (C-MAA2C) algorithm. Simulations in a simple constrained multiagent environment affirm that our reinterpretation of the primal-dual method in terms of probabilistic constraints is effective, and that our proposed value estimate accelerates convergence to a safe joint policy.

Original languageAmerican English
Number of pages13
StatePublished - 2023
Event5th Annual Conference on Learning for Dynamics and Control, L4DC 2023 - Philadelphia, United States
Duration: 15 Jun 202316 Jun 2023


Conference5th Annual Conference on Learning for Dynamics and Control, L4DC 2023
Country/TerritoryUnited States

Bibliographical note

See NREL/CP-5D00-84649 for preprint

NREL Publication Number

  • NREL/CP-5D00-87858


  • chance constraints
  • conditional value at risk
  • Multiagent reinforcement learning
  • primal-dual methods


Dive into the research topics of 'Interpreting Primal-Dual Algorithms for Constrained Multiagent Reinforcement Learning'. Together they form a unique fingerprint.

Cite this