1. A Framework of Energy Efficient Mobile Sensing for Automatic User State Recognition
Y. Wang, et al.
Dept. of Electrical Engineering,
University of Southern California

2. Background Background Diverse applications for user state
Moore's law : doubling the number of transistors in unit area every 18 months
Integrating complex sensing capabilities on mobile devices
Wi-Fi, Bluetooth, GPS, audio, video, light sensors, accelerometers and so on
Mobile phone : a powerful environmental sensing unit
Monitoring a user's ambient context, both unobtrusively and in real time
Diverse applications for user state
Mobile cooperative services
Real time traffic monitoring
Social networking applications
Facebook and MySpace
User state recognition
AI and data mining techniques
Learning interactions between user's behavior and the environment
A combination of diverse features
Motion, location and background condition
Automatically adjusting ring tone profile to appropriate volume

3. Problems and Proposal Limited battery capacity of mobile devices
A big hurdle for context detection
Embedded sensors → major sources of power consumption
Fully charged battery on Nokia N95 mobile phone
Only telephone conversation : 10 hours
GPS receiver is turned on : 6 hours
Energy efficient mobile sensing system (EEMSS)
Hierarchical sensor management scheme for power management
Automatic user state recognition
Motion (such as running and walking), location (such as staying at home or on a freeway) and background environment (such as sound and crowd level)
User states and state transition rules definition by an XML
Three sensor management scheme
State descriptor to support flexibility for application requirements
Minimum set of sensors assignment to achieve energy efficiency
Invoking new sensors when state transitions happen

4. Related Works (1/2) Multi-sensor mobile applications and services
Combining a diverse set of sensors (Gellersen et al. [5])
Monitoring and recognizing human activities with motion sensors
Car manufacturing (Stiefmeier et al. [6])
Supporting a production or maintenance worker by recognizing the worker's actions
Delivering just-in-time information about activities to be performed
Development of a jacket with various sensors
CenceMe [13]
Sharing “buddies” presence with social networks in a secure manner
Using sensors to capture the users' status
Activity, disposition, habits and surroundings
CenceMe prototype implementation on Facebook [14]
Sensay [12]
Context-aware mobile phone using data from a number of sources
Dynamic change of cell phone ring tone, alert type depending on user states

5. Related Works (2/2)
Using a large number of sensors from different fields
Activity recognition and health monitoring [13, 14, 11, 17, 18, 19, 20]
GPS, Bluetooth, WiFi detector, oxygen sensor, accelerometer, electrocardio-graph sensor, temperature sensor, light sensor, microphone, camera, and so on
Providing context information for higher layer applications
Human status tracking, social networking, and location based services
Studies for power management on mobile devices
Survey of methods for saving power on hand-held devices(Viredaz et al. [21])
Dynamic frequency/voltage scaling [22] to reduce power consumption
Suitable for lower-level system design rather than application development
Event driven power-saving method to reduce extra power consumption
Reducing the idle power (Shih et. al. [23])in a “standby” mode
Power on only when there is an incoming or outgoing call
Turdecken [24]
Event-driven, a hierarchical power management system

6. Sensor Management Methodology
Energy efficient mobile sensing
Managing sensors in a hierarchical way based on the user's current state
User's real-time condition
Motion (such as running and walking)
Location (such as staying at home or on a freeway)
Background environment (such as sound and crowd level)
Sensor management
Core component of EEMSS
Including user state recognition
Ex) “meeting in office”
Existence of speech :yes
Current location : office area
Sensor assignment depending user state
Specifying an XML-format state descriptor
Sensor management rules for each state
Control of sensors based on real-time system feedback

7. Sensor Management Using XML
General format of a state descriptor
User state definition between “<State>”and “</State>” tags
Sensor speciation by “<Sensor>” tags
State transition criteria satisfaction → a new state “<NextState>”
Ex) If the user is at “state2" and “sensor2" returns “sensor reading 2" which is not sufficient for state transition, “sensor3"will be turned on immediately to further detect the user's status in order to identify state transition
Three major advantages of xml as state descriptor
(1) representation of hierarchical relationship among sensor in a clear manner
(2) easy modification by someone with limited programming experience
(3) easy parsing by modern programming languages (Java and Python)
Current implementation of EEMSS design
Manually configured sensor management rules
Determination of XML state descriptor and sensor sampling intervals and duty cycles

8. EEMSS: The Overview EEMSS (energy efficient mobile sensing system)
Implementation on Nokia N95 devices
Using N95 built-in sensors
GPS, WiFi detector, accelerometer, and embedded microphone
Goals of study
Prototype implementation
Evaluation of state recognition accuracy, detection latency, and energy efficiency
Set of states

9. EEMSS: Hierarchical Decision Rules
Example of hierarchical decision rules for walking
Periodically sampling GPS when the user is walking
Significant amount of speed increase : riding a vehicle
Wireless access point sets : one's frequently visited places (home, cafeteria, office)

10. EEMSS: Architecture and Implementation (1/2)
Main components of EEMSS
Sensor management module
Classification module
Sensor control interface
Turning sensors on and off
Collecting sensed data
Components for debugging and evaluation
Real-time user state updates
Logging, and user interfaces
Current version of J2ME
No direct access to some of the sensors
Python program
Gathering and sharing sensor data

11. EEMSS: Architecture and Implementation (2/2)
Sensor management module
Major control unit of the system
Parsing a state description file
Sensor management scheme
Control of sensors based on user state and state transition conditions
Classification module
Consumer of the sensor raw data
Processing the raw sensing data into desired format
Magnitude of 3-axis accelerometer sensing data
FFT on sound clips to conduct frequency domain signal analysis
Returning user activity and position feature
“Moving fast”, “walking”, “home wireless access point detected” and “loud environment”
Forwarding user states to the sensor management module
Sensor interface APIs
Support of sensor readings and instruct sensors to switch on/off
GPS and embedded microphone through J2ME APIs
Accelerometer and Wi-Fi detector through Python APIs.

12. Energy Consumption Measurement (1/2)
Two class of sensors on a mobile phone
First class : accelerometer and microphone
Continuous operation and requirement of an explicit signal to be turned off
Need for activation for a period of time to obtain meaningful sensing data
Second class : GPS, Wi-Fi detector, and Bluetooth scanner
Gathering instantaneous samples
Automatically turn off when the sampling interval is over
Energy cost of sensors
Instant power drain + operating duration
Ex) (API and hardware limitations) GPS on Nokia N95s
Requirement of a certain amount of time to synchronize with satellites
Remaining active for about 30 seconds after a location query
Assisted-GPS : satellite synchronization time to less than 10 seconds
Wi-Fi scan : less than 2 seconds to finish
Bluetooth scan : around 10 seconds to complete

13. Energy Consumption Measurement (2/2)
Measurement of sensor energy consumptions
Using Nokia Energy Profiler [26]
A stand-alone application to test and monitor application energy usage in real time
Measurement results
Accelerometer : the least amount of power
Capturing the change of body movement
Indicator of state transition with high probability
GPS : the large power drain and long initialization time
Location tracking and mode of travel classification

14. Sensor Duty Cycle and Computation Time
Sensor management scheme
Minimum set of sensors at any specific time
Assigning an appropriate duty cycle to each sensor
Periodic sensing and sleeping instead of being sampled continuously
Reducing the energy consumption
Duty cycles in EEMSS
Saving energy cost by reducing sensing intervals
Too sampling period : insufficient representation of the real condition
Longer sensing period : increase of the robustness of state recognition with more energy
Longer sleep interval : reducing power battery consumption, increase of detection latency
Two reasons for longer duty cycles to the microphone versus the accelerometer
Accelerometer : less power, and more frequent sampling
Accelerometer : capturing user motion change with less detection delay compared to identifying background sound type

15. Sensor Duty Cycle and Computation Time
Periodically querying GPS
Providing outdoor location and speed information
5 minutes, a relatively long duration for the GPS to lock satellite signal
Wi-Fi scanning
Using event-based approach
Two scenarios for Wi-Fi scan
(1) when the user is detected as moving, a Wi-Fi scan is conducted to check if the user has left his or her recent range
(2) when the user has arrived at a new place, we compare the nearby wireless access points set with known ones in order to identify the user's current location
Duty cycle parameters through extensive empirical tests
No optimization or dynamic adjustment

16. GPS Sensing and Mode of Travel
Real-time location tracking
Detecting the user's basic mode of travel
Using Geo-coordinates and the moving speed of the user
Combining velocity information and distance of travel
Distinguishing one's basic mode of travel such as walking or riding a vehicle
Classification of mode of travel
Checking the recent moving distance and speed
Training a classifier by several location tracking records of user
Using certain threshold values
Occurrence of location request timeout
Indication of entering a building or other indoor environment
Indication of locations in which satellite signals are not reachable

17. Wi-Fi Scanning and Usage
Performing a Wi-Fi scan
Returning MAC address of visible wireless access points around the user
Tagging a particular location by the set of access points visible
Automatic identification of current location by checking nearby access points
EEMSS implementation
Ex) User is at home if the Wi-Fi scan result matches a user’s home
Memorization of wireless access points feature of the user's home and office
Monitoring a user's moving range with Wi-Fi scan
Covering an area of radius 20-30m using wireless access point
Classification if the user has moving out of that range
Wi-Fi scan for GPS sampling start
Sampling location information immediately after Wi-Fi scan
When out of recent range by Wi-Fi scanning

18. Real-time Motion Classification
Accelerometer readings from mobile phone
Placement at various locations due to individual habit
Extremely difficult to perform fully detailed motion classification [10]
Accelerometer data collection
53 different experiments distributed in two weeks
Lengths of experiment : several minutes to hours
Tagging ground truth of activity for analysis and comparison purposes
Standard deviation threshold values
Stable, walking, running, and vehicle mode
No explicit requirement of where the phone should be placed
Computation per 6 seconds

19. Real-time Motion Classification
Experimental results
26 experiments containing a combination of different user motions
Very good for extreme conditions such as stable and running
Confusing with walking and vehicle mode due to feature overlap
above 70% accuracy
Using acceleration motion as a trigger for GPS or Wi-Fi detector

20. Real-time Background Sound Recognition
Two steps for sound classification
Measuring the energy level of the audio signal
Classification whether environment is silent or loud
x(n) : time domain signal
Loud environment → speech recognition with time and frequency domains
Speech → higher silence ratio (SR) [28]
SR : Ratio between the amount of silent time and the total amount of the audio data
No speech → “loud” or “noisy”
No further classification algorithm to distinguish music, noise, etc.

21. Real-time Background Sound Recognition
Fast Fourier Transform
Implementation on the mobile device
Frequency domain features of four types of audio clips
Male's speech, a female's speech, a noise clip and a music clip
Speech signals : more weight on low frequency spectrum from 300Hz to 600Hz
SSCH (Subband Spectral Centroid Histogram) algorithm [29]
Speech detection
Comparison with speech peak frequency thresholds (300Hz - 600Hz)

22. Real-time Background Sound Recognition
Sound data set
1085 speech clips, 86 music clips and 336 noise clips
4 seconds length for each clip
Classification accuracy with different SR thresholds
Increase of SR threshold → decrease of false positives
Decrease of speech detection accuracy
SR = 0.7 : more than 90% of detection accuracy and less than 20% false positive

23. Evaluation Methods Evaluation of EEMSS Phase I - Lab Study
State recognition accuracy
State transition detection latency
Energy efficiency
Phase I - Lab Study
Lab study over 1.5 months with our team members
Calibration of classification parameters
Parameters used to determine mode of travel based on GPS readings
Duty cycles for different sensors
e.g. the sampling frequency of accelerometer, GPS and microphone
Duty cycles for system energy consumption measurement
Phase II - User Trial
User trial in November 2008 at two different universities
Test of EEMSS system in a real setting
Recruit of 10 users
Undergraduate, graduate students, faculties and family members

24. Evaluation Methods Phase II - User Trial
Introduction about basic operation of mobile device
Manual recording diary for two days per user
A standardized booklet (three simple questions)
Motion (e.g.: walking, in vehicle, etc), location, sound (e.g.: quiet, loud, speech, etc)
260 running hours of EEMSS application
More than 300 state transitions

25. Results State Recognition Records
Tracking the user's location by recording geo-coordinates of the user
Location information per 20 seconds
Daily traces captured by EEMSS
Two different participants from CMU and USC on two campus maps
Different modes of travel
Dashed ones : vehicle mode
Solid ones : walking mode
Comparison with ground truth diaries
Dashed curves : bus routes
Solid curves : a user’s traces between home and bus station

26. Results State Recognition Records State Recognition Accuracy
Probing background sound
Quiet, loud and containing speech
User activity inference at some places
Working, meeting, resting, etc.
Combining detected background condition with location obtained by Wi-Fi scan
State Recognition Accuracy
Comparing the system recognized state log with the ground truth records
Ratio of correctly recognized records over total records
Average recognition accuracy over all users : 92.56%
Standard deviation of 2.53%

27. Results State Recognition Accuracy State Transition Detection Latency
Three super states for recognition accuracy
“Walking”, “Vehicle” and “At some place” by location and mode of travel
Percentage of recognition accuracy
Column : ground truth
Row : recognized states of EEMSS.
Staying at some place : very high accuracy
Home, office, etc.
False positives (12.64% of walking time and 10.59% of vehicle time)
Regular slow motions of vehicles → walking
State Transition Detection Latency

28. Results State Transition Detection Latency Device Lifetime Test
Vehicle mode
Quick detection with only one or two GPS queries
At some place mode
Less than 5 minutes for GPS location request timeout, and Wi-Fi scan
Detecting the transition from riding a vehicle to walking
Longer time to distinguish slow motion of vehicle and walking
Sampling acceleration data per 6 seconds
Background sound change detection : less than 3 minutes (duty cycle)
Device Lifetime Test
Lab studies : two researchers for 12 days
Average device lifetime with EEMSS : hours

29. Results Device Lifetime Test Power Usage
Turning on GPS, accelerometer and microphone with same sampling frequency
WiFi scanning per 5 minutes
Less than 5 hours regardless of user activity
Power Usage

30. CONCLUSIONS AND FUTUREWORK DIRECTIONS
Mobile device based sensing platform
Rich contextual information about users
Environment for higher layer applications
Considering limited battery capacities
EEMSS (sensor management scheme for mobile devices)
Selectively turning on minimum set of sensors to monitor user state
Triggering new set of sensors to achieve state transition detection
Shutting down unnecessary sensors at any particular time
Implementation on Nokia N95 devices
Evaluation with 10 users from two universities
Future work
More sophisticated algorithms to dynamically assign sensor duty cycles
Implementing sensor management scheme on more complex sensing applications