Audio/Speech CS376: November 4, 2004 as presented by Jessica Kuo

Audio/Speech Readings Speech Acts: A Spoken-Language Framework The Audio Notebook: Paper and Pen Interaction with Structured Speech A Confederation of Tools for Capturing and Accessing Collaborative Activity

SpeechActs What is it?  Platform for developing spoken natural language applications What are its goals?  Accessibility to developers who are not speech experts  Context awareness  Technology independence

SpeechActs Telephone-based applications for business travelers  reading, calendar lookup, currency conversion, weather forecast  Dictating s  Hands-free usage Compare to: palmOne Treo Smartphone

SpeechActs Overall Structure

SpeechActs Challenges of simulating human conversation  Conversational pacing Is it pausing because it’s broken or thinking?  Explicit error corrections  User prompting Possible user actions are hidden

SpeechActs (Questions) Stellar examples of voice interaction systems?  or less than stellar… (AT&T, airline reservations) Last comments?

Audio Notebook User Scenarios  Student taking notes during class  Reporter taking notes during interview Problem  Capturing information during an audio presentation  Dividing attention between two sources

Audio Notebook Synchronization of handwritten notes and audio Mechanism  Ordinary paper notebook  Digitizing tablet & pen, digital audio recorder Structured audio recording  By user through note-taking  By talker through changes in pitch, pauses, energy

Audio Notebook Features  Audio scrollbar of 80 LED lights (green-red-orange)  Audio Snap-to-Grid using phrase detection “-stormed about it a little bit” “and we brainstormed about it a little bit”  Topic suggestions “uh really neat that’s called phase conjugation” “okay number 4 is going to be metrology and microscopy, and uh as an example of this… that’s going to be uh really neat that’s called phase conjugation”

Audio Notebook Fieldwork Results (2 students, 2 reporters)  Taking notes normally vs. as an outline  Selective re-listening for areas of confusion  Listening at 2x speed  Locating and transcribing quotes  Benefits realized when Audio Notebook not available for use

Audio Notebook (Questions) How might this change the way you take notes? What happens when you tear off some of your notes? How useful is “Snap-to-grid” phrase detection? How useful are topic suggestions and the control over their granularity? Does the audio scrollbar make sense when my recording covers a long time? (? LEDs/minute) Last comments?

Capturing & Accessing Collaborative Activity Coral - confederation of tools Goals  Support activities during collaborative work  Capture records of activities  Provide ways for activities to be indexed  Provide access to indexed activities to make the record an effective resource

Capturing & Accessing Collaborative Activity Types of indices  Intentional annotations  Side-effect indices  Derived indices  Post hoc indices Coral focuses on intentional annotations and side- effect indices  Goal: extract indices from structure of activity instead of imposing regularity upon process

Capturing & Accessing Collaborative Activity Process of assessing and managing invention proposals (IPs) at PARC  Researchers submit IPs  Meeting to evaluate IPs (discussion, conclusion, summary documentation) Setup  LiveBoard + microphones + manager with laptop  Discussion: notes are beamed to the LiveBoard  Conclusion: manager notes action items on LiveBoard  Documentation: manager re-listens to audio

Capturing & Accessing Collaborative Activity Compare audio access with Audio Notebook

Capturing & Accessing Collaborative Activity (Q…) Coral recorded audio only. Would it have been beneficial to have video? Is using speaker identification as indices helpful? Contrast with Audio Notebook  Single user vs. Collaborative environment Constraints due to unique domain it was designed in? Last comments?