A Novel Approach for Recognizing Auditory Events & Scenes Ashish Kapoor
Problem Description How can we represent arbitrary environments, so that we can: –Label scene elements –Classify environments –Synthesize environmental sounds Example: Coffee Shop –Basic spectral texture –Glasses clinking, doors opening, etc.
Outline of Our Approach Create a palette of sounds –Epitomes (Jojic et al) for audio Given an audio segment –Generate distributions over the palette Use the distribution for classification/detection etc
Representation: Palette of Sounds Palette World Input Audio Features To Represent Audio
Epitomes for images Epitome –Jojic, Frey, and Kannan, ICCV 2003 –Developed for images
Epitomes for Audio 1-D signal –2-D representation, but little vertical self-similarity Lots of redundancy (silence, repeated background) Much longer inputs, bigger ratio of input to epitome size –Hours of data => second epitome
Informative Sampling of Patches Original epitome: take patches at random Our approach: try to maximize coverage –reduce sampling likelihood of patches similar to those we have covered t probability of patch selection f t t*
Examples: Toy Sequence 600 frame (10 sec) epitome from 3700 frames (2 min) Informative SamplingRandom Sampling
Random Vs Informative Simulation on the toy dataset –2 secs long epitome –Likelihood Vs # of patches –Averaged over 10 runs
Examples: Outdoor Sequence 1800 frame (1 min) epitome from frames (8 min)
Classification of Events/Scenes Cars Speech P(T|e,c=1) ??? classifying c’: P(T|e,c=2) P(T|e,c=c’) Look at distributions over the epitome –Given a audio segment to classify For all the patches in the audio –Recover the transformations given the epitome Look at the distribution of the transformations to classify Cafe Highway
Experiments 3 Different Environments –Highway, Kitchen, Outdoor Parking 6 Minutes of data to train 30 sec long epitome 4 Events to Detect (manually segmented) Speech (22 examples) Car (17 examples) Utensil: Knife Chopping Vegetables (29 examples) Bird Chirp (24 examples) None of the above (30 examples)
Speech Car Knife/UtensilChirp
Detection Example Speech Detection (hard case) –Very noisy environment (148 th Ave) –Only 5 labeled examples of speech
Performance Comparison Mixture of Gaussians –For each audio segment to classify Classify every frame using the mixture Vote among the results Nearest Neighbor –Same method as for mixture of Gaussians –Computationally too expensive!
Performance Vs Amount of Training Data
Speech Knife/Utensil CarChirp
Contributions Framework for Acoustic Event Detection and Scene Classification Epitomes for Audio –Informative Sampling (Can be applied to any domain) Distributions over epitomic indexes for discrimination
Future Work Informative Sampling –Maximizing the Minimum Likelihood Discriminative Epitomes Novel Scene Classification –Rich Representation using Epitomes –Boosting, other ensemble techniques Hierarchical Acoustic Sound Analysis –Same Model for: Acoustic Event Detection, Scene Classification & Synthesis –clustering mechanisms for scene retrieval
Acknowledgments Sumit Basu Nebojsa Jojic My friends and fellow interns