Heterogeneous Machine Learning on High Performance Computing for End to End Driving of Autonomous Vehicles

Robert Patton, Shang Gao, Spencer Paullssen, Nicholas Haas, Brian Jewell, Xiangyu Zhang, Peter Graf

Research output: Contribution to conferencePaperpeer-review


Current artificial intelligence techniques for end to end driving of autonomous vehicles typically rely on a single form of learning or training processes along with a corresponding dataset or simulation environment. Relatively speaking, success has been shown for a variety of learning modalities in which it can be shown that the machine can successfully "drive" a vehicle. However, the realm of real-world driving extends significantly beyond the realm of limited test environments for machine training. This creates an enormous gap in capability between these two realms. With their superior neural network structures and learning capabilities, humans can be easily trained within a short period of time to proceed from limited test environments to real world driving. For machines though, this gap is guarded by at least two challenges: 1) machine learning techniques remain brittle and unable to generalize to a wide range of scenarios, and 2) effective training data that enhances generalization and generates the desired driving behavior. Further, each challenge can be computationally intensive on its own thereby exasperating the gap. Moreover, is has not yet been shown that a single form of learning or training is capable of addressing a large range of scenarios. As a result, solving the first challenge does not inherently solve the second and vice versa. The work described here discusses an approach to address the first challenge that would also provide a foundation for solving the second. Our approach utilizes a combination of conditional imitation learning with a static dataset, reinforcement learning with a simulation environment, and high-performance computing to train a neural network. As a result, this reduces the "time to solution" from to the existing techniques for autonomous driving and provides an extensible framework to address the second key challenge.

Original languageAmerican English
Number of pages11
StatePublished - 14 Apr 2020
EventSAE 2020 World Congress Experience, WCX 2020 - Detroit, United States
Duration: 21 Apr 202023 Apr 2020


ConferenceSAE 2020 World Congress Experience, WCX 2020
Country/TerritoryUnited States

Bibliographical note

Publisher Copyright:
© 2020 U.S. Department of Energy.

NREL Publication Number

  • NREL/CP-2C00-77207

Other Report Number

  • SAE Technical Paper No. 2020-01-0739


  • autonomous vehicles
  • machine learning


Dive into the research topics of 'Heterogeneous Machine Learning on High Performance Computing for End to End Driving of Autonomous Vehicles'. Together they form a unique fingerprint.

Cite this