SoundSense: Scalable Sound Sensing for People-Centric Application on Mobile Phones Hon Lu, Wei Pan, Nocholas D. lane, Tanzeem Choudhury and Andrew T. Campbell Department of Computer Science, Dartmouth College

Motivation: Utilizing the microphone sensor to detect personalized sound events. Sound captured by mobile phone’s microphone is a rich source of information for surrounding environment, social environment, conversation, activity, location, dietary etc.

What is SoundSense? Scalable Sound Sensing Framework: Capable of identifying any meaningful sound events of a user’s daily life. Implemented for resource limited devices, Apple iPhone. System solely runs in mobile phone

Contribution First general purpose sound event classification system designed for large number of events. Able to address significant sound event’s for individual user’s environment Implemented the whole system architecture and algorithm in Apple iPhone

Design Consideration Building a scalable sound classification system so that it can detect all type of sound events for different users. Privacy Issue: Record and Processing audio data happens all in the Mobile phone. Light weight signal processing and classification of sound.

Design Consideration Phone context condition  RMS good approximation of volume.  30% range of variation for different contextual position.

SoundSense Architecture Remove Frames that are silent or hard to classify

SoundSense Architecture 1. Collect features that are insensitive to volume. 2. Detect coarse-grain category of sound: Voice, music and ambient sound. 3. Multilevel Classification: Decision Tree and Markov Model based classifier. 4. Two level of classification to make the output smoothing.

SoundSense Architecture 1. Use previously established audio signal processing technique 2. In this stage speech recognition, speaker identification and music genre classification is applied

SoundSense Architecture 1. Detect only ambient sound (sound other then voice and music) 2.Unsuprvised learning technique 3. Detect meaningful ambient sound. ( assumption: sound occurrence and duration indicates its importance) 4. Maintain a SoundRank: ranking of the meaningful sound based on their importance 5. Prompt user, if a new sound exceed the threshold value of minimum sound rank.

Implementation Implemented in C,C++ and Objective C Developed for Apple iPhone Duty cycle 0.64 second during lack of acoustic event

Parameters Selection Increasing the buffer size (Sequence Length) increase the accuracy. However, responsiveness of the system also increases. Optimal buffer size is 5. Decision tree Classifier Buffered in FIFO queue Markov model classifier

Parameters Selection Precision is the number of frames that are correctly classified divided by all frames. Recall is define as the recognized occurrence of a frame type divided by the number of overall occurrence of that frame MFCC frame length This Precision and Recall plot is for ambient sound

Evaluation 1. When acoustic event detected CPU usage increase to 25%. In idle situation CPU usage is less then 5% 2. Processing time of a frame (64 ms) is around 20-30ms.

Evaluation Only Decision Tree Classifier With Markov model Classification accuracy improved 10% for music and speech and 3% for ambient sound

Evaluation No reliable sound to represent bus riding

Applications Audio Daily Diary: Log everyday events for a users. – To make query, how much time spend in certain event Music Detector based on Participatory Sensing: – Provides user a way to discover event that are associated with music being played.

Friday Saturday Some music and voice samples are incorrectly classified as ambient sound

Conclusion General Sound Classification – Light-weight – Hierarchical Flexible and Scalable. All task implemented in mobile Phone. Able to identify new sound. Can be used in personalized context.

Thank you Question?