Power-Efficient User Activity Recognition and Longitudinal Pattern Mining on Smartphones Presented by: Vijay Srinivasan (Samsung Research America) Contributors: Thomas Phan, Saeed Abbasi Moghaddam, Abhishek Mukherjee, Emmanuel Munguia Tapia (Samsung Research America) Suyambulingam Rathinasamy Muthupandi (Samsung Research India) Kiran Rachuri, Theus Hossmann, Cecilia Mascolo (University of Cambridge)

Smartphone Sensing and Processing Calls, App usage, SMS, Battery usage, Email, Browsing history Physical sensors Location logs Soft sensor logs Quad core, 1400 MHz, ARM Cortex-A9 Powerful Processing Capabilities 1024 MB RAM ARM Mali-400/MP4 Graphics Processor

Activity Recognition Real-time, on-device activity recognition Walking Driving Biking Idle Running Current generation on-device sensors Sensor sampling Signal processing Classification model Classification engine Feature extraction Real-time, on-device activity recognition

Activity Recognition Automatic Driving UI Physical activity monitoring

Co-Occurrence Engine: Goals and Use Cases Longitudinal Pattern Mining Mine for co-occurring context events that occur frequently together in time and location. Mine all available user activity logs. Contact Alex & Joan Pre-loading for Improved User Experience Open Project Folder Intuitive UI Co-Occurrence Engine Co-Occurrence Patterns Smart Reminders for Energy Management User Activity Logs Video Play Log Music Play Log Setting Log Place Log Call Log Message Log App Usage Log Physical Activity Log Search Log Web Browser Log eBook Log

Outline Power-Efficient Activity Recognition Longitudinal On-Device Pattern Mining Motivation Approach Evaluation Applications

Battery drain is a problem Continuous activity recognition should not drain the Smartphone's battery Aim to keep battery drain per day due to activity recognition to less than 20% Avoid heavy use of power-hungry sensing and processing GPS, sound, etc.

Battery drain is a problem Using accelerometers is power-efficient. What percentage of your phone’s battery charge is drained per day due to always-on accelerometer-based activity recognition? * Keeping the smartphone CPU awake to collect and process data: 72.5% ! Total battery drain per day: 88.4% ! Data acquisition: 13.9% Data processing: 2.0% * Android 2.3.4 on Galaxy SII

Reducing battery drain Do not keep the activity recognition system running all the time – perform duty-cycling. Less time spent in power-hungry ON state Example of duty-cycling User activities: Idle Walking Define wake time here Idle Idle Walking Walking ON power: ~ 250 mW Power Activity recognition ON OFF power: ~ 9 mW Activity recognition OFF Time

Two key questions How much sensor data is needed in the ON state to accurately infer the user’s current activity? Sampling duration How long can the OFF state be maintained before we miss the user’s next activity transition? Sleep duration User activities: Idle Walking Idle Idle Walking Walking Sleep duration Power Activity recognition ON Activity recognition OFF Time Sampling duration

Goals Reduce sampling duration and accurately infer user’s current activity Increase sleep duration as much as possible while missing as few activity transitions as possible In other words, reduce the wake time of the activity recognition system Wake time is the proportion of all time when system is ON User activities: Idle Walking Idle Idle Walking Walking Sleep duration Power Activity recognition ON Activity recognition OFF Time Sampling duration

Outline Power-Efficient Activity Recognition Longitudinal On-Device Pattern Mining Motivation Approach Evaluation Applications

Overview of activity recognition Supervised classification with C4.5 decision trees Offline training phase to create decision tree (stored as a JSON file) Real-time classification uses decision tree from JSON file

Feature extraction Derive three orientation independent data streams For each stream: Segment into fixed-size time windows Compute time and frequency domain features for each window

Evaluation on scripted activities Scripted activity data collection from 17 test subjects Research lab members, late-20s to 50s Perform 10-fold cross-validation to evaluate recognition accuracy Per window recognition accuracy is 98.4% (for 4-second windows) More rigorous field evaluation in the future

Adaptive sleep scheduling algorithm Adaptive sleep durations Longer sleep durations for missable (Idle) event stream Shorter sleep durations for unmissable (Walking, Running, etc.) event stream User activities: Idle Walking Idle Idle Walking Walking Power Activity recognition ON Activity recognition OFF Time

Adaptive sleep scheduling algorithm Maintain a probability of sensing (used to set sleep duration) On each activity output by classifier: If interesting (say walking), set If missable (idle), set Ramp-up and ramp-down factors User activities: Idle Walking Idle Idle Walking Walking Power Activity recognition ON Activity recognition OFF Time

Adaptive sampling durations with a two-tier classifier Tier I Tier II Two key insights Performing accurate idle vs. non-idle classification requires a smaller wake time compared to full activity classification Leverage user’s natural idle periods to save power User activities: Idle Walking Idle Idle Walking Walking Power Activity recognition ON Activity recognition OFF Time

Adaptive sampling durations with a two-tier classifier Train two decision tree classifiers offline with two different sampling windows Tier I Idle classifier Use idle window size of 0.5-1 seconds Tier II activity classifier Use active window size of 4-8 seconds Tier I Tier II Two-Tier classifier

Smartphone implementation System implemented on both Android and Tizen (using wakelock, alarm APIs) Acquire wake lock to ensure that activity recognition is not interrupted Sample tri-axial accelerometer data Perform feature extraction and two-tier activity classification Pass activity result to adaptive sleep scheduling algorithm to compute sleep duration T Set an alarm T seconds into the future Release wake lock to allow smartphone to enter low power sleep mode And repeat! Set alarm and release wakelock Alarm triggered Acquire wakelock Power Activity recognition ON Phone in sleep mode Time

Outline Power-Efficient Activity Recognition Longitudinal On-Device Pattern Mining Motivation Approach Evaluation Applications

Naturalistic data collection Goal: Collect long-term naturalistic smartphone accelerometer data to evaluate adaptive duty-cycling Continuous accelerometer data at 65 Hz 8 users and a total of over 36 days of data ~ 35 MB per day after compression

Evaluation metric for accuracy No ground truth collected Challenging to obtain long-term, continuous ground truth Use always-on activity recognition output as ground truth Measure accuracy as a percentage of always-on classifier accuracy. Four coarse-grained activities: walking, running, idle, and driving. Metric: Average per class accuracy across all users Ensures that accuracy is not biased by majority class (idle)

Evaluation metrics for power Wake time: Proportion of time when activity recognition system is ON Platform independent measure Empirical power consumption in mW Platform dependent May vary across platforms depending on sleep and wakeup overhead

Schemes implemented for analysis Four variants implemented for analysis: Fixed sleep duration with fixed sample duration Adaptive sleep duration with fixed sample duration Fixed sleep duration with adaptive sample duration Adaptive sleep duration with adaptive sample duration

Parameter exploration in each scheme Goal: Given an accuracy lower bound, compute the lowest wake time or power consumption achievable by each scheme over the parameter space For example, the best power consumption achievable by scheme X to achieve an accuracy greater than 90% is 200 mW

Wake Time vs. Accuracy Adaptive duty-cycling achieves a wake time of less than 4% for a 90% accuracy bound Adaptive sample durations outperform adaptive sleep scheduling

Wake time and power consumption Significant additional overhead for each wakeup and sleep on typical smartphones Ramp-up costs during wakeup Lag time during sleep Ramp-up cost

Wake time and power consumption Better wake time does not imply better power consumption Schemes that wake up fewer times are better able to handle the wakeup and sleep overhead Ramp-up cost

Power consumption vs. Accuracy Adaptive duty-cycling achieves a power consumption of ~45 mW at 90% accuracy Reduces always-on power (253 mW) by 81.9%! Adaptive sleep scheduling outperforms adaptive sample durations, by waking up fewer times and reducing the wakeup/sleep overhead

Summary Adaptive duty-cycling schemes leverage the user’s natural idle periods to save power Achieves 90% of the accuracy of an always-on system Reduces wake time by 96.7% Reduces power consumption by 81.9% Battery drain for activity recognition is greatly reduced at a 90% accuracy bound 99.5% battery drain per day for always-on system 48.6% battery drain per day with fixed duty-cycling 17.7% battery drain per day for adaptive duty-cycling

Outline Power-Efficient Activity Recognition Longitudinal On-Device Pattern Mining Motivation Approach Evaluation Applications

Activity visualization Visualization application on Tizen Implemented with HTML5 and Javascript Without duty-cycling, a full battery does not last a single day

Narrative generation Use natural language generation techniques to create an activity narrative for the user We log: User activities User places on major activity transitions (reverse geocoding) Intelligent GPS triggering

Narrative generation Example narrative: Potential applications: On December 9, I took a 2 hour, 23 minute round trip from my home. I started the day at 10:31 AM by driving for 10 miles from my home to Samsung R&D Center where I stayed for 31 minutes. I then went 1 mile from there to Subway Sandwiches on First Street where I remained for 28 minutes. After that, I drove to Valley Fair Mall where I hung out for 37 minutes. I finally drove for 12 miles to my home in San Jose. Potential applications: Activity summarization for the blind Multimedia captions for descriptive story-telling

Future directions Adapt to low power cores dedicated for sensor data acquisition and processing Lower overhead with each wakeup/sleep Execute simpler Tier I on low-power core and Tier II if necessary on higher power core Explore more aggressive, user-specific sleep scheduling based on longitudinal patterns E.g.) User always sleeps from 11 PM to 6 AM Exploit longitudinal patterns between sensor types and activity predicates to save power E.g.) At home -> Not driving, At work -> Not sleeping

Outline Power-Efficient Activity Recognition Longitudinal On-Device Pattern Mining Motivation Approach Evaluation Applications

Co-Occurrence Engine: Goals and Use Cases On-Device Pattern Mining Advantages of pattern mining Patterns are general and can be used in multiple services Exhaustive patterns are flexible in handling missing contexts Contact Alex & Joan Alice typically opens project folder while at work in the afternoon Pre-loading for Improved User Experience Open Project Folder John typically reads news while at home in the morning having breakfast Intuitive UI Bob typically charges phone every night before going to sleep Smart Reminders for Energy Management

On-Device Pattern Mining Advantages of on-device operation Preserves personal data privacy No data uploading costs to cloud to mine patterns Makes use of significant idle cycles on smartphones Assume: No penalty to user as long as engine runs when phone is idle, charging, and battery level > 80%

Outline Power-Efficient Activity Recognition Longitudinal On-Device Pattern Mining Motivation Approach Evaluation Applications

Co-Occurrence Engine Overview Time User Behavioral Logs Morning Afternoon Mobile Itemset Extractor At Home At work Weighted Mining of Temporal Patterns (WeMiT) Play Jazz Itemsets Call contact2 Read book1 SMS contact1 Frequent Itemsets Pattern Generator Co-occurrence itemsets: [Morning, At Home, Play Jazz, Read book1] [Morning, At work] [Afternoon, At work, SMS contact1] [Afternoon, At work, Call contact2] Context Patterns SMS contact1 whenever I am at work in the afternoon Pattern Query Interface Scheduler Post-condition Pre-condition Association rules

Mobile Itemset Extractor Configurable to use additional features through API. Examples: Last contact called or app used Events from last K minutes Compress itemsets due to the repetitive nature of mobile user context data Significantly reduces size of input data Co-occurrence itemsets: [Morning, At Home, Call Contact2, Last Call Contact1] [Morning, At work] [Afternoon, At work, SMS contact1] Frequency weighted Co-occurrence itemsets: [Morning, At Home, Call Contact2, Last Call Contact1] (2) [Morning, At work] (1) [Afternoon, At work, SMS contact1] (2)

Weighted Mining of Temporal Patterns (WeMiT) Based on Apriori Uses downward closure property of support to prune candidate itemsets at each step Uses frequency weighted itemsets to significantly reduce running time Up to 95% reduction in number of co-occurrence itemsets in some users Distribution of itemset frequencies

Weighted Mining of Temporal Patterns (WeMiT) Based on Apriori Uses downward closure property of support to prune candidate itemsets at each step Uses frequency weighted itemsets to significantly reduce running time Up to 95% reduction in number of co-occurrence itemsets in some users Example patterns generated

Prediction applications Intuitive UI. Predict: Next app used by user Next contact called by user

Prediction using co-occurrence engine Higher prediction ranking for: Higher confidence values Higher degree of overlap between current context and pre-condition of association rules Example: Prediction output: Context: Outside, Driving, Evening Target action: - Outgoing calls Call Alice Call Bob Call John

Outline Power-Efficient Activity Recognition Longitudinal On-Device Pattern Mining Motivation Approach Evaluation Applications

Naturalistic Data collection Data from 8 users for 48-77 days Logs mined by the co-occurrence engine: SMS, Call, App usage, Battery level, Phone Charging, Location

Evaluation Evaluate running time for increasing support threshold Support threshold represents minimum number of times each pattern must occur in the data set to be considered frequent WeMiT is 40-60 times faster than Apriori for mobile context data

App prediction evaluation Prediction accuracy for co-occurrence engine with 3 recommendations Better than majority classifier Improves with higher selectivity Similar results for outgoing call contact prediction

Outline Power-Efficient Activity Recognition Longitudinal On-Device Pattern Mining Motivation Approach Evaluation Applications

Other applications Smart battery notifications Activity Recommendations Smart charging reminders

Future directions Develop extensible mining engine to efficiently mine for co-occurrence patterns over: Multiple time and location bins Multiple history windows (short-term vs. long-term patterns) Improve prediction results Explore additional features Leverage existing work on association rule-based classification Develop extensible itemset extractor to support multiple applications User profile summarization Battery reminders