1. Mixed Reality: A Model of Mixed Interaction Céline Coutrix, Laurence Nigay User Interface Engineering Team CLIPS-IMAG Laboratory, University of Grenoble

2. Introduction: Mixed Systems Merging Physical and Digital Worlds In many application fields 1

3. Introduction: Designers Needs A framework for : Understanding and capitalizing on existing Mixed Systems Descriptive power Creating designsGenerative power Assessing multiple design alternatives and increasing usability Comparative power 2

4. Mixed Interaction Model Descriptive Power Generative and Comparative Powers Conclusion and Future Work Outline 3

5. Mixed Interaction Model Mixed object Mixed interaction Descriptive Power Generative and Comparative Powers Conclusion and Future Work Outline 4

6. Example 5

7. Mixed Interaction Model: Mixed Object mixed object tooltask object 6

8. Mixed Interaction Model: Mixed Object physical object physical properties digital object digital properties mixed object physical properties digital properties ? Interaction modality m = (d, l) speech = (microphone, natural language) 7

9. Mixed Interaction Model: Mixed Object Linking modality mixed object physical properties digital properties input device output device output language acquired physical data input language generated physical data 8

10. Mixed Interaction Model: Mixed Object mixed drawing physical properties shapes camera projectorselection image computer vision image 9

11. Mixed Interaction Model: Mixed Object a music bottle physical properties music part loud speakers identity audio sequence screen table lighting colored sequence pitch and volume analysis 10

12. Mixed Interaction Model: Mixed Object mixed object physical properties digital properties Linking modalities tooltask object 11

13. tool task object action reaction elementary task response feedback Mixed Interaction Model: Mixed Interaction 12

14. Interaction modality m = (d, l) Mixed Interaction Model: Mixed Interaction 13

15. mixed tool physical properties digital properties input device output device output language acquired physical data input language generated physical data interaction output language interaction input language task object Mixed Interaction Model: Mixed Interaction 14

16. Mixed Interaction Model: Mixed Interaction 15

17. mixed tool paper button pressed word, place on drawing cameraimage computer vision translation drawing mixed tool Mixed Interaction Model: Mixed Interaction 16

18. Mixed Interaction Model Descriptive Power Generative and Comparative Powers Conclusion and Future Work Outline 17

19. Schemes for classification and better understanding of mixed systems Descriptive Power 18

20. Mixed Interaction Model Descriptive Power Generative and Comparative Powers Conclusion and Future Work Outline 19

21. RAZZLE: Case Study 20

22. RAZZLE: Case Study mixed piece physical properties Identifier Location Orientation location sensor head mounted display 3D projection numeric values geometric transforma° image orienta° sensor numeric values geometric transforma° 21

23. Generative Power mixed tool ?? ?? ? 22

24. RAZZLE: Gestural Modality 23

25. RAZZLE: Gestural Modality mixed tool physical properties recognized gestures cameraimage computer vision translation 24

26. RAZZLE: Tactile Modality 25

27. RAZZLE: Tactile Modality mixed tool physical properties click touchpaddata input language translation 26

28. Mixed Interaction Model Descriptive Power Generative and Comparative Powers Conclusion and Future Work Outline 27

29. Comparative Power Criteria within the model for assessing multiple design alternatives Observability Spatial continuity 28

30. Comparative Power: Observability task object physical properties digital properties input device output device output language acquired physical data input language generated physical data mixed tool physical properties digital properties input device output device output language acquired physical data input language generated physical data interaction output language interaction input language 29

31. Comparative power: Observability mixed tool physical properties recognized gestures cameraimage input language translation 30

32. Comparative Power: Observability mixed tool physical properties click touchpaddata input language translation 31

33. task object physical properties digital properties input device output device output language acquired physical data input language generated physical data Comparative Power: Spatial Continuity mixed tool physical properties digital properties input device output device output language acquired physical data input language generated physical data interaction output language interaction input language 32

34. Comparative Power: Spatial Continuity mixed tool physical properties recognized gestures cameraimage computer vision translation physical properties Identifier Location Orientation head mounted display 3D projection image mixed piece location sensor numeric values geometric transforma° orienta° sensor numeric values geometric transforma° 33

35. Comparative Power: Spatial Continuity mixed tool physical properties click touchpaddata input language translation physical properties Identifier Location Orientation head mounted display 3D projection image mixed piece location sensor numeric values geometric transforma° orienta° sensor numeric values geometric transforma° 34

36. Mixed Interaction Model Descriptive Power Generative and Comparative Powers Conclusion and Future Work Outline 35

37. Definition of a Mixed Interaction Model which Capitalizes existing approaches to describe mixed systems Allows designers to create new interaction modalities Allows them to choose between multiple design alternatives Conclusion 36

38. Improving comparative power by identifying more criteria Future Work 37

39. Providing a development environment based on ICARE Future Work 38

40. Mixed Reality: A Model of Mixed Interaction User Interface Engineering Team CLIPS-IMAG Laboratory, University of Grenoble Céline Coutrix, Laurence Nigay