1. Ravin Balakrishnan ravin@dgp.toronto.edu www.dgp.toronto.edu/~ravin Physics, projection, piles, and pens: user interfaces in the post GUI era.

2. 2 DGP lab Since 1968 6 faculty: 3 HCI, 3 Graphics ~50 graduate students, ~5 postdocs Several tech/admin staff My group: 15 graduate students, 2 postdocs

3. 3 Projects BumpTop: Desktop with Physics and Piles (CHI 2006) Hybrid pointing (UIST 2006) Handheld projector interaction (UIST 2006) Volumetric Display Interaction (UIST 2004, UIST 2006, AVI 2006) Mnemonic rendering (UIST 2006) Under the table interaction (UIST 2006)

4. 4 Physical vs. GUI Desktops GUIs lack rich expressiveness Casual but meaningful placement Loose vs. rigid organization

5. 5 Physically Inspired Desktop Enrich desktop metaphor with real world characteristics Support casual organization, Piling Pen centric

6. 6 Support for Piling Ethnographic studies of office organization (Malone 83, Whittaker et al. 01) Filing Cognitively difficult, multiple categories Premature, duplicate filing Piling Lightweight, easy to maintain Deferred classification Reminds, accessibility of recent information

7. 7 Related Work Ethnographic Studies of Office Organization Malone 1983 Whittaker et al. 2001 Piling in GUIs Mander et al. 1992 Presto 1999 DynaPad 2004 Physically Inspired Interfaces WebBook, Web Forager 1996 Beaudouin-Lafon 2001 Fold’n’Drop 2004 3D Paper-flyer metaphor 2003 2½D, 3D Spatial Organization Data Mountain 1998 Task Gallery 2000

8. 8 Design Goals Realistic Feel Disable Physics as Necessary Tangible, Paper-like Icons Optimize for Pen Interaction Discoverability

9. 9 Video Demonstration Object Movement Pile Interaction Paper-like Icons Arbitrarily sized objects Beyond Physics

10. 10 LassoMenu Fluid selection and parameter adjustment in one stroke Advantages: Avoids tricky gestures, memorization Doesn’t interrupt flow with pen lift, etc.

11. 11 Real and Virtual Piles Items pulled out for emphasis

12. 12 Pressure Trigger interaction with PressureLock Pressure Cursor States: 0% Pressure 75% Pressure100% Pressure PressureLock Hollow cursor, indicates functionality

13. 13 Piling in GUIs We significantly expand on previous designs Pile-pile, intra-pile interaction Emphasize discoverability Pile widgets Integrate browsing advances Transition to pile Physical simulation (Mander et al. 92) DynaPad 2004Presto 1999

14. 14 Addional Interaction Techniques Manipulation Sort, Delete, Move Emphasize contents Insertion: Tossing, Drag’n’Cross Paper-like Icons Crease, Crumple, Pin up Enlarge, Reduce size Shelving Arbitrarily sized objects Beyond Physics Enforcing Axis Alignment Hierarchical Piles

15. 15 Initial User Evaluation 12 participants (5 female, 7 male) 1 hr videotaped think-aloud sessions Simulated tasks Realistic Feel Novices found realistic techniques empowering Playfully rearranging, balancing items Optimize for Pen Interaction Crossing “smoother”, more realistic Crossing deteriorated at screen edges Drag’n’Cross troublesome Discoverability All users discovered interaction techniques Improved repetition of tasks

16. 16 Some Next Steps Convey meta-data as physical attributes Other physics: fields, magnets, cloth Beyond reality? Deployable applications

17. 17 Hybrid Pointing

18. 18 Hybrid Pointing

19. 19 Hybrid Pointing Evaluation

20. 20 Handheld Projectors

21. 21 Prototype

22. 22 Video

23. 23 Volumetric Displays Points illuminated in true 3D 360 ° viewing angle Available commercially

24. 24 Implementation - Display Perspecta Spatial 3D System 10" spherical image 198 2D slices, 768x768 each 24Hz refresh rate

25. 25 Implementation – Finger Tracking Vicon motion tracking system 5 Cameras Tracked 5mm markers Real Time, 120Hz, sub mm precision

26. 26 Issues & Assumptions Objects within arm’s reach Objects inside enclosure Direct link between input and display Volumetric display as exclusive platform

27. 27 Video

28. 28 Acknowledgements DGP Lab members John Hancock

29. 29 Prior Physically-Inspired Interfaces Windows as paper Beaudouin-Lafon 2001 WebBook, Web Forager 1996 3D Paper flyer metaphor 2003 Fold’n’Drop 2004

30. 30 Prior Spatial Organization Benefit of 2½ D, 3D spatial memory Data Mountain (1998) Task Gallery (2000)