A Parallel Hub-and-Spoke System for Large-Scale Scenario-Based Optimization Under Uncertainty

Efficient solution of stochastic programming problems generally requires the use of parallel computing resources. Here, we describe the open source package mpi-sppy, in which efficient and scalable parallelization is a central feature. We describe the overall architecture and provide computational examples and results showing scalability to the largest instances that we know of for the well-known unit commitment problem. In addition we demonstrate novel combinations of methods for accelerating convergence. The mpi-sppy package is written in Python, leverages the widely used Pyomo (http://www.pyomo.org) library for mathematical programming, builds on existing MPI and numpy implementations to ensure efficiency and scalability, and is available via http://github.com/Pyomo/mpi-sppy. We report computational experiments that demonstrate the ability to solve very large stochastic programming problems - including mixed-integer variants - in minutes of wall clock time, effectively and scalably leveraging significant parallel computing resources. We report results for the largest known instances of stochastic mixed-integer unit commitment problems, solving to provably tight optimality gaps in minutes of wall clock time.