Chapter 18: Learning and Tracking Ad Hoc Fiducial Markers in Spatial Augmented Reality

Emma Gould, Stephen Guerin, Cody Smith, Steve Smith, Brian Bush, Kenny Gruchalla

Research output: Chapter in Book/Report/Conference proceedingChapter


We describe a spatial augmented reality system with a tangible user interface used to control computer simulations of complex systems. In spatial augmented reality, the user’s physical space is augmented with projected imagery, blending real objects with projected information, and a tangible user interface enables users to manipulate physical objects as controllers for interactive visualizations. Our system learns ad hoc objects in the user’s environment as fiducial markers (i.e., objects that are visually recognized and tracked). When combined with simulation and visualization tools, these interfaces allow the user to control simulations or ensembles of simulations via physical objects using apt metaphors. While other research has leveraged the use of depth cameras, our system enables the use of standard cameras in readily available smartphones and webcams and has an implementation that runs completely in JavaScript in the web browser. We discuss the prerequisite object-recognition requirements for such tangible user interfaces and describe computer-vision and machine-learning algorithms meeting those requirements. We conclude by presenting example applications, which are also available online.
Original languageAmerican English
Title of host publicationAdvances in Computers and Information in Engineering Research, Volume 2
EditorsJ. G. Michopoulos, C. J. J. Paredis, D. W. Rosen, J. M. Vance
StatePublished - 2021

NREL Publication Number

  • NREL/CH-6A20-70202


  • energy distribution
  • energy production
  • fiducial markers
  • learning
  • object recognition
  • simulations
  • software framework
  • spatial augmented reality
  • tracking
  • user interfaces


Dive into the research topics of 'Chapter 18: Learning and Tracking Ad Hoc Fiducial Markers in Spatial Augmented Reality'. Together they form a unique fingerprint.

Cite this