NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning

Integrating cloud services into advanced computing facilities provides significant new capabilities and offers several advantages over focusing solely on traditional high performance computing (HPC) workloads. The integration of cloud services is especially potent for workflows that fuse data streams, modeling and simulation ("modsim"), and machine learning. A key challenge to adopting a hybrid edge-cloud-HPC model is aligning optimal capability, data, and user intent on the right resources for each step in a workflow. The National Renewable Energy Laboratory (NREL) Stratus service provides a basis for this alignment. Stratus layers the capabilities needed to make cloud services accessible to a lab-based scientific community on commercial offerings, and currently supports upward of 200 projects, ranging from Internet of Things (IoT) integration to traditional modeling and simulation. This provides a real-world inventory of scientific workflow elements, which enables placing these elements appropriately between the edge, cloud, and traditional HPC. This paper outlines a vision via reference architecture and the application of that architecture in a typical workflow. We highlight multiple components, including sensor data intake, cleaning and transforming (edge/cloud suitable), generation of synthetic data through modsim, computationally heavy machine learning training and hyperparameter optimization (HPC suitable), and inference and deployment (cloud ideal). Every step in such a workflow involves a cost-benefit analysis of the data movement, computational efficiency, availability, latency, and resource capabilities. The reference architecture and examples outlined in this paper allow for better understanding of new opportunities in the context of emerging workflows that combine IOT, cloud, and HPC to bolster scientific productivity.