Power-Efficient and Adaptive Duty-Cycling of Smartphone Activity Recognition Systems Presented by: Vijay Srinivasan (Samsung Research) Collaborators: Thomas Phan (Samsung Research) Kiran Rachuri, Theus Hossmann, Cecilia Mascolo (University of Cambridge)

Smartphone activity recognition Sensors include microphone, accelerometer, GPS, etc. Walking Driving Biking Idle Running Current generation on-device sensors Sensor sampling Signal processing Classification model Classification engine Feature extraction What is sar? What sensors are used? Real-time, on-device activity recognition

Applications Physical activity monitoring Automatic Driving UI Two example applications Explain them Automatic Driving UI

Interesting wellness visualizations Instead of a boring pie-chart, researchers have come up with more interesting visualizations. For exam Ubifit from Intel, 2008 BeWell from Dartmouth, 2011 Social motivation in the future

Battery drain is a problem Continuous activity recognition should not reduce the Smartphone's battery life Aim to keep battery drain per day due to activity recognition to less than 20% Avoid heavy use of power-hungry sensors GPS, sound, etc.

Battery drain is a problem Using accelerometers is power-efficient. What percentage of your phone’s battery charge is drained per day due to always-on activity recognition? * Keeping the smartphone CPU awake to collect and process data: 72.5% ! Total battery drain per day: 88.4% ! Data acquisition: 13.9% Data processing: 2.0% * Android 2.3.4 on Galaxy SII

Reducing battery drain Do not keep the activity recognition system running all the time. Less time spent in power-hungry ON state Example of duty-cycling User activities: Idle Walking Define wake time here Idle Idle Walking Walking ON power: ~ 250 mW Power Activity recognition ON OFF power: ~ 9 mW Activity recognition OFF Time

Two key questions How much sensor data is needed in the ON state to accurately infer the user’s current activity? Sampling duration How long can the OFF state be maintained before we miss the user’s next activity transition? Sleep duration User activities: Idle Walking Idle Idle Walking Walking Sleep duration Power Activity recognition ON Activity recognition OFF Time Sampling duration

Goals Reduce sampling duration and accurately infer user’s current activity Increase sleep duration as much as possible while missing as few activity transitions as possible In other words, reduce the wake time of the activity recognition system Wake time is the proportion of all time when system is ON User activities: Idle Walking Idle Idle Walking Walking Sleep duration Power Activity recognition ON Activity recognition OFF Time Sampling duration

Adaptive duty-cycling Adaptive sleep durations Longer sleep durations for missable (Idle) event stream Shorter sleep durations for unmissable (Walking, Running, etc.) event stream User activities: Idle Walking Idle Idle Walking Walking Power Activity recognition ON Activity recognition OFF Time

Adaptive duty-cycling Adaptive sampling durations Shorter sampling durations for idle events Longer sampling durations for active events User activities: Idle Walking Idle Idle Walking Walking Power Activity recognition ON Activity recognition OFF Time

Adaptive duty-cycling Leverage user’s natural idle periods to save power Adaptive duty-cycling Achieves 90% of the accuracy of an always-on system Reduces wake time by 96.7% Reduces power consumption by 81.9% User activities: Idle Walking Idle Idle Walking Walking Power Activity recognition ON Activity recognition OFF Time

Outline Related work Our approach Experimental Setup Evaluation Activity Recognition Adaptive Duty-Cycling Experimental Setup Evaluation Example applications Future directions

Related work on activity recognition Early work on mobile sensor-based activity recognition used specialized on-body sensors Bao 2004, Lester 2006, Tapia 2007, Consolvo 2008 Smartphones are fast becoming the platform of choice for mobile activity recognition research Miluzzo 2008 (CenceMe), Lu 2010 (Jigsaw), Reddy 2010 We treat battery use as a first-class challenge for accelerometer-based activity recognition

Related work on power-efficient smartphone activity recognition First work to our knowledge to systematically address the key bottleneck for accelerometer-based activity recognition Approach Idea Reduces power from data acquisition? (13.9% battery) per day) Reduces power from data processing? (2% battery per day) Reduces power from keeping smartphone awake? (72.5% battery per day) Triggered sensing (Lu et al, Sensys 2010) Trigger high power sensor such as GPS based on low power accelerometer Yes No Activity-adaptive approach (Yan et al, ISWC 2012) Vary sampling rate and set of features used (time-domain vs. frequency domain) to save power Admission control (Rachuri et al, Ubicomp 2010; Lu et al, Sensys 2010) Use heuristics to determine if sensor data window should be sent to classifier

Outline Related work Our approach Experimental Setup Evaluation Activity Recognition Adaptive Duty-Cycling Experimental Setup Evaluation Example applications Future directions

Overview of activity recognition Supervised classification with C4.5 decision trees Offline training phase to create decision tree (stored as a JSON file) Real-time classification uses decision tree from JSON file

Feature extraction Derive three orientation independent data streams For each stream: Segment into fixed-size time windows Compute time and frequency domain features for each window

Feature extraction Time-domain features Frequency-domain features Mean, standard deviation, entropy, etc. Frequency-domain features Energy, highest magnitude frequency, etc.

Evaluation on scripted activities Scripted activity data collection from 17 test subjects Over 6 hours of data Research lab members, late-20s to 50s Activities: walking (on a at surface), walking up stairs, walking down stairs, running, bicycling, driving, standing, and sitting.

Evaluation on scripted activities Perform 10-fold cross-validation to evaluate recognition accuracy Per window recognition accuracy is 98.4% (for 4-second windows) More rigorous field evaluation in the future

Outline Related work Our approach Experimental Setup Evaluation Activity Recognition Adaptive Duty-Cycling Experimental Setup Evaluation Example applications Future directions

Adaptive sleep scheduling algorithm Maintain a probability of sensing On each activity output by classifier: If interesting (say walking), set If missable (say idle), set Ramp-up and ramp-down factors User activities: Idle Walking Idle Idle Walking Walking Power Activity recognition ON Activity recognition OFF Time

Adaptive sleep scheduling algorithm Initialize sleep interval Repeatedly increment by step interval until a uniform random number from is greater than the probability of sensing Key parameters Ramp-up and ramp-down factors Step interval Minimum sleep interval

Adaptive sampling durations with a two-tier classifier Tier I Tier II Two key insights Performing accurate idle vs. non-idle classification requires a smaller wake time compared to full activity classification Leverage user’s natural idle periods to save power User activities: Idle Walking Idle Idle Walking Walking Power Activity recognition ON Activity recognition OFF Time

Adaptive sampling durations with a two-tier classifier Train two decision tree classifiers offline with two different sampling windows Tier I Idle classifier Use idle window size of 0.5-1 seconds Recognizes only idle vs. non-idle (transform training labels to idle or non-idle) Tier II activity classifier Use active window size of 4-8 seconds Recognizes entire set of physical activities such as walking, driving, idle, biking etc. Tier I Tier II Two-Tier classifier

Smartphone implementation Acquire wake lock to ensure that activity recognition is not interrupted Sample tri-axial accelerometer data at 32 Hz Perform feature extraction and two-tier activity classification Pass activity result to adaptive sleep scheduling algorithm to compute sleep duration T Set an alarm T seconds into the future Release wake lock to allow smartphone to enter low power sleep mode And repeat! System implemented on both Android and Tizen Set alarm and release wakelock Alarm triggered Acquire wakelock Power Activity recognition ON Phone in sleep mode Time

Outline Related work Our approach Experimental Setup Evaluation Activity Recognition Adaptive Duty-Cycling Experimental Setup Evaluation Example applications Future directions

Naturalistic data collection Goal: Collect long-term naturalistic smartphone accelerometer data to evaluate adaptive duty-cycling Continuous accelerometer data at 65 Hz ~ 35 MB per day after compression No ground truth collected Challenging to obtain long-term, continuous ground truth

Evaluation metric for accuracy Use always-on activity recognition output as ground truth Interpret accuracy as a percentage of always-on classifier accuracy. Four coarse-grained activities: walking, running, idle, and driving. Metric: Aggregate per class accuracy For each class (say driving), compute time slice accuracy: Percentage of seconds that are correctly classified to the target class (driving) across all users Aggregate per class accuracy is the average time slice accuracy across all classes.

Evaluation metrics for power Wake time: Proportion of time when activity recognition system is ON Platform independent measure Empirical power consumption in mW Platform dependent May vary greatly for the same wake time measure depending on sleep and wakeup overhead

Schemes implemented for analysis Four variants implemented for analysis: Fixed sleep duration with fixed sample duration Adaptive sleep duration with fixed sample duration Fixed sleep duration with adaptive sample duration Adaptive sleep duration with adaptive sample duration

Parameter exploration in each scheme Goal: Given an accuracy lower bound, compute the lowest wake time or power consumption achievable by each scheme over the parameter space For example, the best power consumption achievable by scheme X to achieve an accuracy greater than 90% is 200 mW

Outline Related work Our approach Experimental Setup Evaluation Activity Recognition Adaptive Duty-Cycling Experimental Setup Evaluation Example applications Future directions

Wake Time vs. Accuracy Adaptive duty-cycling performs best with a wake time of less than 4% for a 90% accuracy bound Adaptive sample durations outperform adaptive sleep scheduling

Wake time and power consumption Significant additional overhead for each wakeup and sleep Ramp-up costs during wakeup Lag time during sleep Ramp-up cost

Wake time and power consumption When comparing duty-cycling schemes, better wake time does not imply better power consumption Schemes that wake up fewer times are better able to handle the wakeup and sleep overhead Ramp-up cost

Power consumption evaluation Goal: For each duty-cycling scheme, evaluate best power consumption achievable given an accuracy lower bound Compute power consumption by extrapolating empirical measurements of sleep and wakeup overhead Simple linear model to capture the relationship between the energy of each wakeup and the duration of the wakeup

Power consumption vs. Accuracy Adaptive duty-cycling achieves a power consumption of ~45 mW at 90% accuracy Reduces always-on power (253 mW) by 81.9%! Adaptive sleep scheduling outperforms adaptive sample durations, by waking up fewer times and reducing the wakeup/sleep overhead

Conclusions Adaptive duty-cycling schemes leverage the user’s natural idle periods to save power Achieves 90% of the accuracy of an always-on system Reduces wake time by 96.7% Reduces power consumption by 81.9% Battery drain for activity recognition is greatly reduced 99.5% battery drain per day for always-on system 17.7% battery drain per day for adaptive duty-cycling

Outline Related work Our approach Experimental Setup Evaluation Activity Recognition Adaptive Duty-Cycling Experimental Setup Evaluation Example applications Future directions

Log visualization Visualization application on Tizen Without duty-cycling, battery does not last a single day

Narrative generation We log: User activities User places on major activity transitions (reverse geocoding) Use natural language generation techniques to create an activity narrative Intelligent GPS triggering

Narrative generation Example narrative: Potential applications: On December 9, I took a 2 hour, 23 minute round trip from my home. I started the day at 10:31 AM by driving for 10 miles from my home to Samsung R&D Center where I stayed for 31 minutes. I then went 1 mile from there to Subway Sandwiches on First Street where I remained for 28 minutes. After that, I drove to Valley Fair Mall where I hung out for 37 minutes. I finally drove for 12 miles to my home in San Jose. Potential applications: Activity summarization for the blind Multimedia captions for descriptive story-telling

Outline Related work Our approach Experimental Setup Evaluation Activity Recognition Adaptive Duty-Cycling Experimental Setup Evaluation Example applications Future directions

Future directions Exploit relationships between sensor types and activity predicates to save power E.g.) At home -> Not driving, At work -> Not sleeping Low power cores dedicated for sensor data acquisition and processing Lower overhead with each wakeup/sleep Potentially execute Tier I on low-power core and Tier II if necessary on higher power core

Thanks Questions?

Backup slides

Spurious classification pruning