Tanja Schultz, Alan Black, Bob Frederking Carnegie Mellon University West Palm Beach, March 28, 2003 Towards Dolphin Recognition

Outline Ê Speech-to-Speech Recognition è Brief Introduction è Lab, Research è Data Requirements è Audio data è ‘Transcriptions’ Ë Towards Dolphin Recognition è Applications è Current Approaches è Preliminary Results

Part 1 Ê Speech-to-Speech Recognition è Brief Introduction è Lab, Research è Data Requirements è Audio data è ‘Transcriptions’ Ë Towards Dolphin Recognition è Applications è Current Approaches è Preliminary Results

Speech Processing Terms l Speech Recognition Converts spoken speech input into written text output l Natural Language Understanding (NLU) Derives the meaning of the spoken or written input l (Speech-to-speech) Translation Transforms text / speech from language A to text / speech of language B l Speech Synthesis (Text-To-Speech=TTS) Converts written text input into audible output

Speech Recognition hellohello Hello Hale Bob Hallo : TTS Speech Input - Preprocessing Decoding / Search Postprocessing - Synthesis

Fundamental Equation of SR hellohello P(W/x) = [ P(x/W) * P(W) ] / P(x) Am AE M Are A R I AI you J U we VE I am you are we are : Acoustic Model Pronunciation Language Model A-b A-m A-e

SR: Data Requirements Audio Data Sound set Units built from sounds Text Data  Am AE M Are A R I AI you J U we VE I am you are we are : Acoustic Model Pronunciation Language Model A-b A-m A-e

Janus Speech Recognition Toolkit (JRTk) l Unlimited and Open Vocabulary l Spontaneous and Conversational Human-Human Speech l Speaker-Independent l High Bandwidth, Telephone, Car, Broadcast l Languages: English, German, Spanish, French, Italian, Swedish, Portuguese, Korean, Japanese, Serbo-Croatian, Chinese, Shanghai, Arabic, Turkish, Russian, Tamil, Czech l Best Performance on Public Benchmarks Õ DoD, (English) DARPA Hub-5 Test ‘96, ‘97 (SWB-Task) Õ Verbmobil (German) Benchmark ’95-’00 (Travel-Task)

Mobil Device for Translation&Navigation

Multi-lingual Meeting Support The Meeting Browser is a powerful tool that allows us to record a new meeting, review or summarize an existing meeting or search a set of existing meetings for a particular speaker, topic, or idea.

Multilingual Indexing of Video View4You / Informedia: Automatically records Broadcast News and allows the user to retrieve video segments of news items for different topics using spoken language input Non-cooperative speaker on video Cooperative user Indexing requires only low quality translation

Part 2 Ê Speech-to-Speech Recognition è Brief Introduction è Lab, Research è Data Requirements è Audio data è ‘Transcriptions’ Ë Towards Dolphin Recognition è Applications è Current Approaches è Preliminary Results

Towards Dolphin Recognition ? ? Whose voice is this? ? ? ? Whose voice is it? Identification ? Is this Bob¡¯s voice? ? Is it Nippy’s voice? Verification/Detection Segmentation and Clustering Speaker B Speaker A Which segments are from the same speaker? Which segments are from the same speaker? Where are speaker changes? Where are speaker changes? Speaker B Speaker A Which segments are from the same speaker? Which segments are from the same dolphin? Where are speaker changes? Where are dolphins changing?

Applications ‘off-line’ applications (off the water, off the boat, off season) l Data Management and Indexing l Automatic Assignment/Labeling of already recorded (archived) data l Automatic Post-Processing (Indexing) for later retrieval l Towards Important/Meaningful Units = DOLPHONES l Segmentation and Clustering of similar sounds/units l Find out about unit frequencies l Find out about correlation between sounds and other events l Whistles correlated to Family Relationship l Who belongs to whom l Find out about the family tree? l Can we find out more about social structure?

Applications ‘on-line’ applications l Identification and Tracking l Who is currently speaking l Who is around l Towards Important/Meaningful Units l Find out about correlation between sounds and other events l Whistles correlated to Family Relationship l Who belongs to whom éWide-range identification, tracking, and observation (since sound travels longer distances than image)

Common Approaches Two distinct phases Feature extraction Model training Training speech for each dolphin Nippy Havana Model for each dolphin Havana Nippy Training Phase Feature extraction Detection decision Hypothesis: Havana Detection Phase ? xyz

Current Approaches l A likelihood ratio test is used for the detection decision Feature extraction Dolphin model Background model Background model   l p(X|dolph) is the likelihood for the dolphin model when the features X = (x1,x2,…) are given l p(X|dolph) is an alternative or so called background model trained on all data but that of the dolphin in question  = p(X|dolph) / p(X|dolph)

First Experiments - Setup l Take the data we got from Denise l Alan did the labeling of about 160 files l Labels: Õ dolphin sounds ~370 tokens Õ electric noise (machine, clicks, others) ~180 tokens Õ pauses ~ 220 tokens l Derive Dolphin ID from file name (educ. Guess) (Caroh, Havana, Lag, Lat, LG, LH, Luna, Mel, Nassau, Nippy) l Train one model per dolphin, one ‘garbage’ model for the rest l Recognize incoming audio file; hypotheses consist of list of dolphin and garbage models l Count number of models per audio file and return the name of dolphin with the highest count as the one being identified

First Experiments - Results

Next steps l Step 1: To build a ‘real’ system we need l MORE audio data MORE audio data MORE... l Labels (the more accurate the better) l Idea 1: Automatic labeling, live with the errors l Idea 2: Manual labeling l Idea 3: Automatic labeling and post-editing l Step 2: Given more data l Automatic clustering l Try first steps towards unit detection l Step 3: Build a working system, make it small and fast enough for deployment