DARWIN PHONES: THE EVOLUTION OF SENSING AND INFERENCE ON MOBILE PHONES PRESENTED BY: BRANDON OCHS Emiliano Miluzzo, Cory T. Cornelius, Ashwin Ramaswamy, Tanzeem Choudhury, Zhigang Liu, Andrew T. Campbell, "Darwin phones: the evolution of sensing and inference on mobile phones," In Proc. of 8th ACM Conference on Mobile Systems, Applications, and Services (MobiSys), 2010, pp

What does Darwin do?  A Smartphone platform for urban sensing  Proof of concept model uses microphone  Communicates with other local devices to improve inference accuracy (collaborative inference)  Framework can be expanded to gather information using a range of sensor data

What about battery life?  Communicates with backend server to do the CPU- intensive machine learning algorithms  Local devices share models rather than re- computing them  Sensing is enabled/disabled as the system sees fit

Common Urban Sensing Challenges  Human burden of training classifiers  Ability to perform reliably in different environments (indoor vs outdoor)  The ability to scale to a large number of phones without hurting usability and battery life.  Darwin overcomes all of these through classifier/model evolution, model pooling, and collaborative inference

Types of Learning  Supervised: Given a fully-labeled training set  Semi-Supervised: Given a small training set that is evolved  Unsupervised: No training set is given

Darwin Steps  Evolution, Pooling, and Collaborative Inference  These represent Darwin’s novel evolve-pool-collaborate model implemented on mobile phones

Classifier Evolution  Automated approach to updating models over time  Needs to account for variability in sensing conditions and settings  Variability in background noise and phone location require separate models

Model Pooling  Reuses models that have already been built and evolved on other phones  Exchange classification models whenever the model is available from another phone  Classifiers do not need to be retrained, which increases scalability  Can pool models from backend servers

Collaborative Inference  Combines results from multiple phones  Run inference algorithms in parallel on the same classifiers  System is more robust to degradation in sensing quality  Increases accuracy

Darwin Design: Computation  Reduces the on-the-phone computation by offloading some of the work to backend servers  Backend server uses a machine learning algorithm to compute a Gaussian Mixture Model (2 hours for 15 seconds of audio)  Feature vectors are computed locally

Darwin Design: Context  Context (in/out of pocket, in/out of bag) will impact the sensing and inference capability  Classifier evolution makes sure the classifier of an event is robust across different environments

Darwin Design: Co-location  Accounts for a group of co-located phones running the same classification algorithm and sensing the same event but computing different inference results  Phones pool classification models when collocated or from backend servers  Compares against its own model and the co-located model  Drastically reduces classification latency  Exploits diversity of different phone sensing context viewpoints

Speaker Recognition  Attempts to identify a speaker by analyzing the microphone’s audio stream  Suppresses silence, low amplitude audio, and chunks that do not contain human voice  Reduce false positives by pre-processing in 32ms blocks

Speaker Modeling  Feature vector consisting of Mel Frequency Cepstral Coefficients  Each speaker is modeled with 20 Gaussians  An initial speaker model is built by collecting a short training sample

Classifier Evolution: Training Step  Short training phase (30 seconds) used to build a model which is later evolved  First 15 seconds used as the training set  Last 15 seconds used as baseline for evolution

Classifier Evolution: Evolution Step  Semi-supervised learning strategy  If the likelihood of the incoming audio stream is much lower than any of the baselines then a new model is evolved

Collaborative Inference  Local inference phase can be broken into three steps:  Local inference operated by each individual phone  Propagation of the result of the local inference to the neighboring phones  Final inference based on the neighboring mobile phones local inference results  Each node individually operates inference on the sensed event  Results and confidence broadcasted

Privacy and Trust  Raw sensor data is not stored on or leaves the mobile phone  The content of a conversation or raw audio data is never disclosed  Users can choose to opt out of Darwin

Experimental Results  Tested using a mixture of five N97 and iPhones used by eight people over a period of two weeks  Audio recorded in different locations  Classifier trained indoors

Experiment 1 Parameters  Three people walk along a sidewalk of a busy road and engage in conversation  The speaker recognition application without the Darwin components runs on each of the phones carried by the people

Experiment 1 Results: Without Evolution

Experiment 2 Parameters  Meeting setting in an office environment where 8 people are involved in conversation  The phones are located at different distances from people in the meeting, some on the table and some in people’s pockets

Experiment 2 Results

Experiment 3 Parameters  Five phones in a noisy restaurant  Three of the five people are engaged in conversation  Two of the five phones are placed on the table  Phone 4 Is the closest phone to speaker 4 and also the closest phone to another group of people having a loud conversation

Experiment 3 Results

Experiment 4 Parameters  Five people walk along a sidewalk and three of them are talking  The greatest improvement is observed by speaker 1, whose phone is clipped to their belt

Experiment 4 Results

Time and Energy Measurements  Baselines for power use determined  Measurements performed using the Nokia Energy Profiler tool  No data gathered for the iPhone  Smart duty cycling required later to save battery life

Time and Energy Measurements

Possible Applications  Virtual square application  Social application for a group of friends  Place discovery application  Use collaborative inference to determine location  Friend Tagging application  Exploit face recognition to tag friends on pictures

Future Work  Duty cycling for improved battery life  Simplified classification techniques

Improvements On The Paper  Studies don’t show conclusive evidence; there should be separate control models for each of the scenarios

Conclusion  The Darwin system combines classifier evolution, model pooling, and collaborative inference  Results indicate that the performance boost offered by Darwin off sets problems with sensing context  The Darwin system provides a scalable framework that can be used for other urban sensing applications

References  [1] Emiliano Miluzzo, Cory T. Cornelius, Ashwin Ramaswamy, Tanzeem Choudhury, Zhigang Liu, Andrew T. Campbell, "Darwin phones: the evolution of sensing and inference on mobile phones," In Proc. of 8th ACM Conference on Mobile Systems, Applications, and Services (MobiSys), 2010, pp  [2] H. Ezzaidi and J. Rouat. Pitch and MFCC Dependent GMM Models for Speaker Identification systems. In Electrical and Computer Engineering, Canadian Conference on, volume 1, 2004  [3] H. Ezzaidi and J. Rouat. Pitch and MFCC Dependent GMM Models for Speaker Identification systems. In Electrical and Computer Engineering, Canadian Conference on, volume 1, 2004.