NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Integrating cloud services into advanced computing facilities provides significant new capabilities and offers several advantages over focusing solely on traditional high performance computing (HPC) workloads. The integration of cloud services is especially potent for workflows that fuse data streams, modeling and simulation (“modsim”), and machine learning. A key challenge to adopting a hybrid edge-cloud-HPC model is aligning optimal capability, data, and user intent on the right resources for each step in a workflow. The National Renewable Energy Laboratory (NREL) Stratus service provides a basis for this alignment. Stratus layers the capabilities needed to make cloud services accessible to a lab-based scientific community on commercial offerings, and currently supports upward of 200 projects, ranging from Internet of Things (IoT) integration to traditional modeling and simulation. This provides a real-world inventory of scientific workflow elements, which enables placing these elements appropriately between the edge, cloud, and traditional HPC. This paper outlines a vision via reference architecture and the application of that architecture in a typical workflow. We highlight multiple components, including sensor data intake, cleaning and transforming (edge/cloud suitable), generation of synthetic data through modsim, computationally heavy machine learning training and hyperparameter optimization (HPC suitable), and inference and deployment (cloud ideal). Every step in such a workflow involves a cost-benefit analysis of the data movement, computational efficiency, availability, latency, and resource capabilities. The reference architecture and examples outlined in this paper allow for better understanding of new opportunities in the context of emerging workflows that combine IOT, cloud, and HPC to bolster scientific productivity.

Original languageEnglish
Title of host publicationDriving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation - 21st Smoky Mountains Computational Sciences and Engineering, SMC 2021, Revised Selected Papers
Editors[given-name]Jeffrey Nichols, [given-name]Arthur ‘Barney’ Maccabe, James Nutaro, Swaroop Pophale, Pravallika Devineni, Theresa Ahearn, Becky Verastegui
PublisherSpringer Science and Business Media Deutschland GmbH
Pages227-246
Number of pages20
ISBN (Print)9783030964979
DOIs
StatePublished - 2022
Event21st Smoky Mountains Computational Sciences and Engineering Conference, SMC 2021 - Virtual, Online
Duration: 18 Oct 202120 Oct 2021

Publication series

NameCommunications in Computer and Information Science
Volume1512 CCIS
ISSN (Print)1865-0929
ISSN (Electronic)1865-0937

Conference

Conference21st Smoky Mountains Computational Sciences and Engineering Conference, SMC 2021
CityVirtual, Online
Period18/10/2120/10/21

Bibliographical note

Publisher Copyright:
© 2022, Springer Nature Switzerland AG.

Keywords

  • Cloud
  • Computing
  • Data streams
  • Edge
  • HPC
  • Machine learning
  • Modeling simulation
  • NREL

Fingerprint

Dive into the research topics of 'NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning'. Together they form a unique fingerprint.

Cite this