Multimodal Sensor Fusion Framework for Residential Building Occupancy Detection: Article No. 111828

Sin Tan, Margarite Jacoby, Homagni Saha, Anthony Florita, Gregor Henze, Soumik Sarkar

Research output: Contribution to journalArticlepeer-review

16 Scopus Citations


For several years now, smart building energy systems have been a research area of intensive activity. In light of the increasing need for sustainable buildings and energy systems, this trend motivates an increasing need for a solution to reduce carbon dioxide emissions and improve energy efficiency. This paper proposes a high-performing and transferable occupancy detection framework that combines sensor data from different data modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic energy data using ensemble method. To draw out the best prediction performance in each modality, the proposed framework was developed, including various models that were designed to learn the occupancy patterns reflected in the physical data streams. To tackle the time series environmental data, we designed two variants of an occupancy detection spatiotemporal pattern network (Occ-STPN) that performs both feature level and decision level fusion, respectively. We also propose a new metric; the fading memory mean square error (FMMSE), that provides a fair evaluation and penalization of delayed occupancy predictions. Multiple open-sourced datasets, including the Electricity Consumption and Occupancy and the University of California, Irvine's (UCI) building occupancy detection dataset, along with our own real data collected from six different houses, were used to validate the algorithms' performance. The experimental results presented herein break down the performance for each sensing modality, and a detailed analysis of the performance is also discussed.
Original languageAmerican English
Number of pages16
JournalEnergy and Buildings
StatePublished - 2022

NREL Publication Number

  • NREL/JA-5500-82141


  • deep learning
  • ensemble
  • feature extraction
  • feature selection
  • few-shot learning
  • machine learning
  • multimodal sensor fusion
  • neural network
  • occupancy detection
  • time series forecasting


Dive into the research topics of 'Multimodal Sensor Fusion Framework for Residential Building Occupancy Detection: Article No. 111828'. Together they form a unique fingerprint.

Cite this