1. A Framework of Energy Efficient Mobile Sensing for Automatic User State Recognition
Y. Wang, et al.
Dept. of Electrical Engineering,
University of Southern California

2. Outline INTRODUCTION RELATEDWORK SENSOR MANAGEMENT METHODOLOGY
EEMSS DESIGN AND IMPLEMENTATION – A CASE STUDY
EEMSS: The Overview
EEMSS: Architecture and Implementation
ENERGY CONSUMPTION MEASUREMENT AND SENSOR DUTY CYCLES
Energy Consumption Measurement
Sensor Duty Cycle Assignment and Computation Time
SENSOR INFERENCE AND CLASSIFICATION
GPS Sensing and Mode of Travel Classification
WiFi Scanning and Usage
Real-time Motion Classification Based on Accelerometer Sensing
Real-time Background Sound Recognition
PERFORMANCE EVALUATION
Method
Results
CONCLUSIONS AND FUTUREWORK DIRECTIONS

3. INTRODUCTION Background Applications using user state recognition
Moore's law : doubling the number of transistors in unit area every 18 months
Integrating complex sensing capabilities on mobile devices
WiFi, Bluetooth, GPS, audio, video, light sensors, accelerometers and so on
Mobile phone : a powerful environmental sensing unit
Monitoring a user's ambient context, both unobtrusively and in real time
Applications using user state recognition
Mobile cooperative services
Real time traffic monitoring
Social networking applications
Facebook and MySpace
AI and data mining techniques
Learning interactions between user's behavior and the environment
A combination of diverse features
Motion, location and background condition
Automatically adjusting ring tone profile to appropriate volume

4. INTRODUCTION Limited battery capacity of mobile devices
A big hurdle for context detection
Embedded sensors → major sources of power consumption
Fully charged battery on Nokia N95 mobile phone
Only telephone conversation : 10 hours
GPS receiver is turned on : 6 hours
Energy efficient mobile sensing system (EEMSS)
Hierarchical sensor management scheme for power management
Automatic user state recognition
Motion (such as running and walking), location (such as staying at home or on a freeway) and background environment (such as sound and crowd level)
User states and state transition rules definition by an XML
Three sensor management scheme
State descriptor to support flexibility for application requirements
Minimum set of sensors assignment to achieve energy efficiency,
Invoking new sensors when state transitions happen

5. INTRODUCTION Hierarchical sensor management EEMSS Field study
Reconfiguration of active sensors dynamically
EEMSS
Implementation on Nokia N95 devices
Using accelerometer, WiFi detector, GPS, and micro phone on the N95
Incorporation of algorithms for real-time user activity and sound recognition
User states of EEMSS can currently
“Walking", “Vehicle", “Resting", “Home talking",”Hom entertaining", “Working", “Meeting", ”Loud_office", “Quiet place", “Speech place", and “Loud place”
Field study
10 users at two different university campuses
92.56% accuracy
Improvement of the battery lifetime by over 75%

6. RELATEDWORK Multi-sensor mobile applications and services
Combining a diverse set of sensors (Gellersen et al. [5])
Monitoring and recognizing human activities with motion sensors
Car manufacturing (Stiefmeier et al. [6])
Supporting a production or maintenance worker by recognizing the worker's actions
Delivering just-in-time information about activities to be performed
Development of a jacket with various sensors
Accelerometer
A common low cost sensor used for detecting motion
Training data collection with one or more accelerometer in a certain period
CenceMe [13]
Sharing “buddies” presence with social networks in a secure manner
Using sensors to capture the users' status
Activity, disposition, habits and surroundings
CenceMe prototype implementation on Facebook [14]

7. RELATEDWORK
Sensay [12]
Context-aware mobile phone using data from a number of sources
Dynamic change of cell phone ring tone, alert type depending on user states
Using a large number of sensors from different fields
Urban / paticipatory sensing, activity recognition and health monitoring [13, 14, 11, 17, 18, 19, 20]
GPS, Bluetooth, WiFi detector, oxygen sensor, accelerometer, electrocardio-graph sensor, temperature sensor, light sensor, microphone, camera, and so on
Providing context information for higher layer applications
Human status tracking, social networking, and location based services
The problem of power management on mobile devices
Survey of methods for saving power on hand-held devices(Viredaz et al. [21])
Dynamic frequency/voltage scaling [22] to reduce power consumption
Suitable for lower-level system design rather than application development
Event driven power-saving method to reduce extra power consumption
Reducing the idle power (Shih et. al. [23])in a “standby” mode
Power on only when there is an incoming or outgoing call

8. RELATEDWORK Turdecken [24] Proposed method
Event-driven, a hierarchical power management system
Proposed method
Effective power management for sensors on mobile devices
Hierarchical approach for managing sensors
Maintaining accuracy in sensing the user's state
Similar with “SeeMon" system [25] by only performing context recognition when changes

9. SENSOR MANAGEMENT METHODOLOGY
Energy efficient mobile sensing
Managing sensors in a hierarchical way based on the user's current state
User's real-time condition
One's motion (such as running and walking), location (such as staying at home or on a freeway) and background environment (such as sound and crowd level)
Sensor management
Core component of EEMSS
Including user state recognition
Ex) “meeting in office”
Existence of speech :yes
Current location : office area
Sensor assignment depending user state
Specifying an XML-format state descriptor
Sensor management rules for each state
Control of sensors based on real-time system feedback

10. SENSOR MANAGEMENT METHODOLOGY
General format of a state descriptor
User state definition between “<State>”and “</State>” tags
Sensor speciation by “<Sensor>” tags
State transition criteria satisfaction → a new state “<NextState>”
Ex) If the user is at “state2" and “sensor2" returns “sensor reading 2" which is not sufficient for state transition, “sensor3"will be turned on immediately to further detect the user's status in order to identify state transition
Three major advantages of xml as state descriptor
(1) representation of hierarchical relationship among sensor in a clear manner
(2) easy modification by someone with limited programming experience
(3) easy parsing by modern programming languages (Java and Python)
Current implementation of EEMSS design
Manually configured sensor management rules
Determination of XML state descriptor and sensor sampling intervals and duty cycles

11. SENSOR MANAGEMENT METHODOLOGY
Future extension of EEMSS design
Automated sensor assignment mechanism rather than manual setting

12. EEMSS DESIGN AND IMPLEMENTATION – A CASE STUDY

13. EEMSS: The Overview EEMSS (energy efficient mobile sensing system)
Implementation on Nokia N95 devices
N95 built-in sensors
GPS, WiFi detector, accelerometer, and embedded microphone
Goal of case study
Prototype implementation
Evaluation of state recognition accuracy, detection latency, and energy efficiency
Set of states
User's daily activities

14. EEMSS: The Overview
State transition → a new set of sensors to recognize new activity
Example of of the user states (Walking)
State transition detection when the user is walking outdoor
Hierarchical decision rules for walking
Periodically sampling GPS when the user is walking
Significant amount of speed increase : riding a vehicle
Wireless access point sets : one's frequently visited places (home, cafeteria, office)

15. EEMSS: Architecture and Implementation
Main components of EEMSS
Sensor management
Activity classification
Implementation on J2ME on Nokia N95 devices
Current version of J2ME
Not providing APIs to allow direct access to some of the sensors
Python program
Gathering and sharing sensor data over a local socket connection
Layered architecture
Sensor management module
Classification module
Sensor control interface
Turning sensors on and off, and obtaining sensed data
Other components for debugging and evaluation
Real-time user state updates, logging, and user interfaces

16. EEMSS: Architecture and Implementation
Sensor management module
Major control unit of the system
Parsing a state description file
Sensor management scheme
Control of sensors based on user state and state transition conditions
Classification module
Consumer of the sensor raw data
Processing the raw sensing data into desired format
Magnitude of 3-axis accelerometer sensing data
FFT on sound clips to conduct frequency domain signal analysis
Returning user activity and position feature
“Moving fast”, “walking”, “home wireless access point detected” and “loud environment”
Forwarding user states to the sensor management module
Sensor interface APIs
Providing direct access to the sensors
Support of sensor readings and instruct sensors to switch on/off
GPS and embedded microphone through J2ME APIs
Accelerometer and Wi-Fi detector through Python APIs.

17. ENERGY CONSUMPTION MEASUREMENT AND SENSOR DUTY CYCLES
Energy consumption of each sensor in the Nokia N95
Best coordination of sensors
Energy consumption measurements on different built-in sensors
GPS, WiFi detector, microphone and accelerometer
Duty cycling mechanisms on the sensors to reduce the energy cost

18. Energy Consumption Measurement
Two class of sensors on a mobile phone
First class : accelerometer and microphone
Continuous operation and requirement of an explicit signal to be turned off
Need for activation for a period of time to obtain meaningful sensing data
Second class : GPS, WiFi detector, and Bluetooth scanner
Gathering instantaneous samples
Automatically turn off when the sampling interval is over
Energy cost of the sensors
Instant power drain + the operating duration
Ex) (API and hardware limitations) GPS on Nokia N95s
Requirement of a certain amount of time to synchronize with satellites
Remaining active for about 30 seconds after a location query
Assisted-GPS : satellite synchronization time to less than 10 seconds
Wi-Fi scan : less than 2 seconds to finish
Bluetooth scan : around 10 seconds to complete

19. Energy Consumption Measurement
Measurement of sensor energy consumptions
Using Nokia Energy Profiler [26]
A stand-alone application to test and monitor application energy usage in real time
Measurement results
Accelerometer : the least amount of power
Capturing the change of body movement
Indicator of state transition with high probability
GPS : the large power drain and long initialization time
Location tracking and mode of travel classification

20. Sensor Duty Cycle Assignment and Computation Time
Sensor management scheme
Minimum set of sensors at any specific time
Assigning an appropriate duty cycle to each sensor
Periodic sensing and sleeping instead of being sampled continuously
Reducing the energy consumption
Duty cycles in EEMSS
Saving energy cost by reducing sensing intervals
Too sampling period : insufficient representation of the real condition
Longer sensing period : increase of the robustness of state recognition with more energy
Longer sleep interval : reducing power battery consumption, increase of detection latency
Two reasons for longer duty cycles to the microphone versus the accelerometer
Accelerometer : less power, and more frequent sampling
Accelerometer : capturing user motion change with less detection delay compared to identifying background sound type

21. Sensor Duty Cycle Assignment and Computation Time
Periodically querying GPS
Providing outdoor location and speed information
5 minutes, a relatively long duration for the GPS to lock satellite signal
Wi-Fi scanning
Using event-based approach
Two scenarios for Wi-Fi scan
(1) when the user is detected as moving, a Wi-Fi scan is conducted to check if the user has left his or her recent range
(2) when the user has arrived at a new place, we compare the nearby wireless access points set with known ones in order to identify the user's current location
Duty cycle parameters through extensive empirical tests
No optimization or dynamic adjustment

22. SENSOR INFERENCE AND CLASSIFICATION
Potential human activities inference
Using sensors including GPS and Wi-Fi detector
Implementation of classification algorithms
Identification of user activities
Identification of background sound types
Using accelerometer and microphone readings

23. GPS Sensing and Mode of Travel Classification
Real-time location tracking
Detecting the user's basic mode of travel
Using Geo-coordinates and the moving speed of the user
Combining velocity information and distance of travel
Distinguishing one's basic mode of travel such as walking or riding a vehicle
Classification of mode of travel
Checking the recent moving distance and speed
Training a classifier by several location tracking records of user
Using certain threshold values
Occurrence of location request timeout
Indication of entering a building or other indoor environment
Indication of locations in which satellite signals are not reachable

24. Wi-Fi Scanning and Usage
Performing a Wi-Fi scan
Returning MAC address of visible wireless access points around the user
Tagging a particular location by the set of access points visible
Automatic identification of current location by checking nearby access points
EEMSS implementation
Ex) User is at home if the Wi-Fi scan result matches a user’s home
Memorization of wireless access points feature of the user's home and office
Monitoring a user's moving range with Wi-Fi scan
Covering an area of radius 20-30m using wireless access point
Classification if the user has moving out of that range
W-Fi scan for GPS sampling start
Sampling location information immediately after Wi-Fi scan
When out of recent range by Wi-Fi scanning

25. Real-time Motion Classification Based on Accelerometer Sensing
Activity classification based on accelerometer readings
Widely using various machine learning tools
Problems in most of the previous works
Requirement of attachment of accelerometers on specific body positions
Requirement of several data features not suitable for real-time implementation
Accelerometer readings from mobile phone
Placement at various locations due to individual habit
Extremely difficult to perform fully detailed motion classification [10]
Standard deviation of accelerometer magnitude values
One of the independent features of phone placement
Suitable feature for real-time motion classification
Accelerometer data collection
53 different experiments distributed in two weeks

26. Real-time Motion Classification Based on Accelerometer Sensing
Accelerometer data collection
53 different experiments distributed in two weeks
Lengths of experiment : several minutes to hours
Tagging ground truth of activity for analysis and comparison purposes
Off-line calculation of standard deviation for different activities
Standard deviation threshold values
Stable, walking, running, and vehicle mode
No explicit requirement of where the phone should be placed
Computation per 6 seconds

27. Real-time Motion Classification Based on Accelerometer Sensing
Experimental results
26 experiments containing a combination of different user motions
Very good for extreme conditions such as stable and running
Confusing with walking and vehicle mode due to feature overlap
above 70% accuracy
Using acceleration motion as a trigger for GPS or Wi-Fi detector

28. Real-time Background Sound Recognition
Two steps for sound classification
Measuring the energy level of the audio signal
Classification whether environment is silent or loud
x(n) : time domain signal
Loud environment → speech recognition with time and frequency domains
Speech → higher silence ratio (SR) [28]
SR : Ratio between the amount of silent time and the total amount of the audio data
No speech → “loud” or “noisy”
No further classification algorithm to distinguish music, noise, etc.

29. Real-time Background Sound Recognition
Fast Fourier Transform
Implementation on the mobile device
Frequency domain features of four types of audio clips
Male's speech, a female's speech, a noise clip and a music clip
Speech signals : more weight on low frequency spectrum from 300Hz to 600Hz
SSCH (Subband Spectral Centroid Histogram) algorithm [29]
Speech detection
Comparison with speech peak frequency thresholds (300Hz - 600Hz)

30. Real-time Background Sound Recognition
Sound data set
1085 speech clips, 86 music clips and 336 noise clips
4 seconds length for each clip
Classification accuracy with different SR thresholds
Increase of SR threshold → decrease of false positives
Decrease of speech detection accuracy
SR = 0.7 : more than 90% of detection accuracy and less than 20% false positive

31. PERFORMANCE EVALUATION

32. Method Evaluation of EEMSS Phase I - Lab Study Phase II - User Trial
State recognition accuracy
State transition detection latency
Energy efficiency
Phase I - Lab Study
Lab study over 1.5 months with our team members
Calibration of classification parameters
Parameters used to determine mode of travel based on GPS readings
Duty cycles for different sensors
e.g. the sampling frequency of accelerometer, GPS and microphone
Duty cycles for system energy consumption measurement
Phase II - User Trial
User trial in November 2008 at two different universities
Test of EEMSS system in a real setting
Recruit of 10 users
Undergraduate, graduate students, faculties and family members

33. Method Phase II - User Trial
Introduction about basic operation of mobile device
Manual recording diary for two days per user
A standardized booklet (three simple questions)
Motion (e.g.: walking, in vehicle, etc), location, sound (e.g.: quiet, loud, speech, etc)
260 running hours of EEMSS application
More than 300 state transitions

34. Results State Recognition Records
Tracking the user's location by recording geo-coordinates of the user
Location information per 20 seconds
Daily traces captured by EEMSS
Two different participants from CMU and USC on two campus maps
Different modes of travel
Dashed ones : vehicle mode
Solid ones : walking mode
Comparison with ground truth diaries
Dashed curves : bus routes
Solid curves : a user’s traces between home and bus station

35. Results State Recognition Records State Recognition Accuracy
Probing background sound
Quiet, loud and containing speech
User activity inference at some places
Working, meeting, resting, etc.
Combining detected background condition with location obtained by Wi-Fi scan
State Recognition Accuracy
Comparing the system recognized state log with the ground truth records
Ratio of correctly recognized records over total records
Average recognition accuracy over all users : 92.56%
Standard deviation of 2.53%

36. Results State Recognition Accuracy State Transition Detection Latency
Three super states for recognition accuracy
“Walking”, “Vehicle” and “At some place” by location and mode of travel
Percentage of recognition accuracy
Column : ground truth
Row : recognized states of EEMSS.
Staying at some place : very high accuracy
Home, office, etc.
False positives (12.64% of walking time and 10.59% of vehicle time)
Regular slow motions of vehicles → walking
State Transition Detection Latency

37. Results State Transition Detection Latency Device Lifetime Test
Vehicle mode
Quick detection with only one or two GPS queries
At some place mode
Less than 5 minutes for GPS location request timeout, and Wi-Fi scan
Detecting the transition from riding a vehicle to walking
Longer time to distinguish slow motion of vehicle and walking
Sampling acceleration data per 6 seconds
Background sound change detection : less than 3 minutes (duty cycle)
Device Lifetime Test
Lab studies : two researchers for 12 days
Average device lifetime with EEMSS : hours

38. Results Device Lifetime Test Power Usage
Turning on GPS, accelerometer and microphone with same sampling frequency
WiFi scanning per 5 minutes
Less than 5 hours regardless of user activity
Power Usage

39. CONCLUSIONS AND FUTUREWORK DIRECTIONS
Mobile device based sensing platform
Rich contextual information about users
Environment for higher layer applications
Considering limited battery capacities
EEMSS (sensor management scheme for mobile devices)
Selectively turning on minimum set of sensors to monitor user state
Triggering new set of sensors to achieve state transition detection
Shutting down unnecessary sensors at any particular time
Implementation on Nokia N95 devices
Evaluation with 10 users from two universities
Future work
More sophisticated algorithms to dynamically assign sensor duty cycles
Implementing sensor management scheme on more complex sensing applications