28 NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning: Preprint

Research output: Contribution to conferencePaper

Abstract

Integrating cloud services into advanced computing facilities provides significant new capabilities over focusing solely on traditional high performance computing (HPC) workloads. This brings complementary capabilities as well as enabling new focused roles for HPC. They are especially potent for workflows that fuse data streams, modeling and simulation ('modsim') and machine learning. A key challenge to adopting a hybrid edge-cloud-HPC model is to align optimal capability, data, and user intent on the right resources for each step in a workflow.?The NREL Stratus service provides a basis for this: Stratus layers capabilities needed to make?cloud services accessible to a lab-based scientific community on commercial offerings, and; currently supports upwards of 200 projects ranging from IOT integration to traditional modeling and simulation. This provides a real-world inventory of scientific workflow elements. A growing knowledge base enables placing these elements appropriately between the edge, cloud, and traditional HPC. This paper outlines a vision via reference architecture and the application of that architecture in a typical workflow highlighting multiple components: sensor data intake, cleaning and transforming (edge/cloud suitable); generation of synthetic data through modsim, computationally heavy ML training and hyperparameter optimization (HPC suitable), and; inference and deployment (cloud ideal). Every step in such a workflow involves a cost-benefit analysis regarding the data movement, computational efficiency, availability, latency, and resource capabilities. The reference architecture and examples outlined allow for understanding new opportunities in the context of emerging workflows that combine IOT, cloud, and HPC to bolster scientific productivity.
Original languageAmerican English
Number of pages17
StatePublished - 2022
EventSmokey Mountains Computational Sciences and Engineering Conference - Kingsport, Tennessee
Duration: 24 Aug 202126 Aug 2021

Conference

ConferenceSmokey Mountains Computational Sciences and Engineering Conference
CityKingsport, Tennessee
Period24/08/2126/08/21

NREL Publication Number

  • NREL/CP-2C00-80214

Keywords

  • AI
  • cloud computing
  • cloud vision
  • compute workflow
  • data modeling
  • edge computing
  • edge inference
  • high performance computing
  • HPC
  • IoT
  • machine learning
  • modsim
  • NREL Stratus

Fingerprint

Dive into the research topics of '28 NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning: Preprint'. Together they form a unique fingerprint.

Cite this