1. Building the Design Studio of the Future Aaron Adler Jacob Eisenstein Michael Oltmans Lisa Guttentag Randall Davis October 23, 2004

2. AARON ADLERBuilding the Design Studio of the Future Sketching Is Not the Whole Picture

3. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Speech and Gestures Express more precise spatial and structural relationships Relate sketches that are drawn from different perspectives Describe temporal events or paths of motion Disambiguate using multiple modalities

4. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Our Vision Previous work focused on one or two modalities Step back and take a look at the big picture Design Studio of the Future –Understand users’ sketching, speech, gesture inputs –Respond naturally, like talking to another human

5. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Roadmap Exploratory Studies Multimodal Communication Domain Modeling Dialogue Management Recognition Components WoZ Dialogue Studies Domain Reasoning Multimodal Understanding Multimodal Intelligent Design Studio Usability Human Dialogue - Out-of-vocabulary terms - Illuminate unclear parts - Possible design mistakes

6. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Outline Sketching is not the whole picture Roadmap Trimodal Studies Preliminary Components –Speech and Sketching –Gesture and Speech

7. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Outline Sketching is not the whole picture Roadmap Trimodal Studies Preliminary Components –Speech and Sketching –Gesture and Speech

8. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Interesting Questions to Study Sketch –What information is best communicated with sketching? –How do people sketch when they’re explaining things? –How do they sketch when they’re taking notes?

9. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Interesting Questions to Study Speech –What information is best communicated with speech? –How do people talk when they’re drawing? How can ASR better support speech in the context of other modalities? –How should computers generate and understand back-channel feedback?

10. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Interesting Questions to Study Gesture –What information is best communicated with gesture? –Which features of gesture are idiosyncratic? Which are universal? –What does gesture tell us about discourse structure? –Are gestures cues for disfluencies? Sentence breaks? –How can we build embodied agents with realistic body language?

11. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Trimodal Interaction Study How are speech, gesture, and sketching used together? –15 pairs of participants –Two video cameras, a Mimio, and a Tablet PC –4.5 hours of data

12. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Looking Ahead to Recognition Captured digital video Colored gloves used for future hand tracking Headset microphones Time-stamped sketch data

13. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Diagrams and objects

14. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Trimodal Setup Listener Speaker

15. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Trimodal Setup Listener Setup Speaker Setup

16. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Details to Observe Structure then function Back channel communication Collaboration between speaker and listener

17. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Movie Clip I

18. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Movie Clip II

19. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Movie Clip III

20. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Early Results We identified two phases in the explanations: –Structure phase –Function phase Participants used back channel communication during the explanations The amount of sketching the listener did varied

21. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Outline Sketching is not the whole picture Roadmap Trimodal Studies Preliminary Components –Speech and Sketching –Gesture and Speech

22. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Results: Speech and Sketching Filled pauses important: “ahh”, “umm” Key phrases: “there are”, “and”

23. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Multimodal Understanding and Integration Combining speech and sketching we can make it easier to perform some operations “There are three identical pendulums.” “There are three touching pendulums.”

24. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Speech-Gesture Integration Align referential pronouns and gestures to the diagram “ this piece is made of wood” “it moves back and forth like this” deictictracingtrajectory Time thispiecegoesherethis…andso

25. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Results: Speech and Gesture Corpus-based approach –Feature set based on linguistic knowledge Improves performance –84% F-measure for baseline system –95% F-measure for optimization system

26. October 23, 2004AARON ADLERBuilding the Design Studio of the Future Conclusion Intelligent Design Studio of the Future Trimodal Corpus Empirical Results Initial Components