Sketch­based interface on a handheld augmented reality system Rhys Moyne Honours Minor Thesis Supervisor: Dr. Christian Sandor

TINT

Augmented Reality (AR) Adding virtual information to the real world Aids user in understanding the world Merges real world (camera image) and virtual objects Tracking is needed to calculate location of objects and user in the environment

AR Display Technologies Head-mounted Handheld

Motivation Handheld AR systems require different ways of interaction due to size constraint Exisiting methods such as pinch gloves or wrist pads are not suitable Future AR devices such as mobile phones are likely to have touch screens Sketch-based input in handheld augmented reality is largely unexplored

Direct Manipulation and Sketch- based Interface

Direct Manipulation

User actions affect object immediately Example: driving a car User turns steering wheel left and wheel moves left No command “TURN LEFT” “Matching user's gestures with the observed virtual motion” (Dragicevic et al., p. 2)

Sketch-based interface

Allows user to directly interact similar to pen and paper More natural Has been explored in areas such as modelling, animation, user interface prototyping Gestures

Research Question Is a sketch­based interface a suitable interaction method in a handheld augmented reality system?

Research Approach Literature Review Iterative prototypes Informal qualitative feedback on techniques (User study)

Development Approach Create a demo illustrating interactive exploration in a simple test scene Make use of TINT framework  Add tracking  Add animation Direct manipulation interface (complete by 18 September)  More complicated physics (18 September)  Gesture Recognition (18 September) Port to Mobile Augmented Reality (30 September) Projection of camera image onto virtual objects

Demo This can be used as a base to control visualisations in TINT Illustrates how to control objects to explore

TINT (This is not TINMITH) TINMITH – HMD prototyping platform TINT – Handheld augmented reality prototyping platform Used to prototype AR applications that may be possible on future mobile phones Written in Python allowing fast development

Implementation

ARToolkit Tracking Calculates where the marker is relative to the camera position Uses computer vision techniques to identify marker Allows the virtual objects to be placed in correct location

Compiz Physics

$1 Gesture Recognizer

Summary How to interactively control visualisations in handheld augmented reality?

Thanks, Questions?

References Compiz.org, 2009, 'Compiz',, accessed 3 September Dragicevic, P, Ramos, G, Bibliowitcz, J, Nowrouzezahrai, D, Balakrishnan, R & Singh, K 2008, 'Video browsing by direct manipulation', in Twenty­ sixth annual SIGCHI conference on Human factors in computing systems, ACM, Florence, Italy, pp. 237­246. Igarashi, T, Matsuoka, S & Tanaka, H 1999, 'Teddy: a sketching interface for 3D freeform design' in Proceedings of the 26th annual conference on computer graphics and interactive techniques, ACM, pp. 409­416. Kato, H & Billinghurst, M 1999, 'Marker tracking and HMD calibration for a video­based augmented reality conferencing system', in Proceedings of Augmented Reality 1999 (IWAR '99), IEEE, San Francisco, CA, pp. 85­94. Piekarski, W & Thomas, BH 2001, 'Tinmith­Metro: new outdoor techniques for creating city models with an augmented reality wearable computer', in Proceedings of Fifth International Symposium on Wearable Computers, IEEE, Zurich, pp. 31­ 38. Sandor, C, Cunningham, A, Eck, U, Urquhart, D, Jarvis, D, Dey, A, Barbier, S, Marner, M & Rhee, S 2009, 'Egocentric space­distorting visualizations for rapid environment exploration in mobile mixed reality', in 8th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'09), ACM, Orlando, Florida. Shneiderman, B 1983, 'Direct Manipulation: a step beyond programming languages', Computer, vol. 16, no. 8, pp sleepygeek.org,n.d., '$1 gesture recognizer in python',, accessed 3 September Wobbrock, JO, Wilson, AD & Li, Y 2007, 'Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes' in Proceedings of the 20th annual ACM symposium on user interface software and technology, ACM, Newport, Rhode Island, pp. 159­168.