Learning appliance usage in homes

Problem: detect when different appliances are turned on.

Why learning appliance usage?

  • Useful for energy saving and improving energy efficiency
    (consumer, industry, and policy benefits are discussed in Armel et al 2013).
  • Health sensing by understanding user habits and activities at home (e.g., for seniors living alone)
  • Provide behavioral analytics

Past work has looked at the problem using energy data from the utility meter. These meters sample the total energy consumed by a home over time. The challenge is that the input data is a mix of multiple patterns.


In this work, we stay in the unsupervised domain -- we do NOT assume knowing the energy patterns of the appliances, or any human labels. However, we take a new perspective: our idea is to use in-door location data to provide self-supervising information. Our solution combines utility meter's total energy data with residents' location data from a passive radio sensor.

We show that our design allows us to detect when a particular appliance is used, and the location of the appliance in the home, all in a self-supervised manner, without any labeled data.

Challenge: designing a model for weakly-related data streams.

  • Location data does not imply appliance usage (people can be next to an appliance without using it, and we do not assume knowing the locations of appliances).
  • For energy data, many background cycling events (e.g., fridge, HVAC) are unrelated to residents' locations.
  • For location data, there could be multiple people in the environment, and not all are related to appliance activation.
  • Cannot learn to encode the two modalities into a shared space (as common in past work), because the two streams are unrelated most of the time.

Our solution: learning cross-modal predictability

The intuition is that when an appliance is activated, location data becomes predictable (because someone is likely to be near the appliance). Therefore, our design:

  1. learns to predict locations given the energy event
  2. clusters activation events with the learned predictability and cross-modal relation


Self-Supervised Learning of Appliance Usage
Chen-Yu Hsu, Abbas Zeitoun, Guang-He Lee, Dina Katabi, Tommi Jaakkola
International Conference on Learning Representations (ICLR) 2020
[PAPER] [TALK (5 minutes)]


Sapple was covered by: Tech Crunch, Engadget, DailyMail, MIT news, and other media outlets.

Also check out

Enabling Identification and Behavioral Sensing in Homes using Radio Reflections
Chen-Yu Hsu, Rumen Hristov, Guang-He Lee, Mingmin Zhao, Dina Katabi
CHI 2019

Zero-Effort In-Home Sleep and Insomnia Monitoring using Radio Signals
Chen-Yu Hsu, Aayush Ahuja, Shichao Yue, Rumen Hristov, Zachary Kabalec, Dina Katabi
Ubicomp 2017

Extracting Gait Velocity and Stride Length from Surrounding Radio Signals
Chen-Yu Hsu, Yuchen Liu, Zachary Kabelac, Rumen Hristov, Dina Katabi, Christin Liu
CHI 2017

Capturing the Human Figure Through a Wall
Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, Fredo Durand
SIGGRAPH Asia 2015

Smart Homes that Monitor Breathing and Heart Rate
Fadel Adib, Hongzi Mao, Zachary Kabelac, Dina Katabi, Robert C. Miller
ACM CHI 2015

3D Tracking via Body Radio Reflections
Fadel Adib, Zachary Kabelac, Dina Katabi, Robert C. Miller
Usenix NSDI 2014