Collaborative Mixed Reality Visualization of an Archaeological Excavation Hrvoje Benko Department of Computer Science Columbia University MIT talk – January 12, 2005

2 Multidisciplinary Team

3 Motivation Excavation is destructive and physically unreconstructable process Excavation is destructive and physically unreconstructable process Need to preserve as much data as possible for analysis Need to preserve as much data as possible for analysis Data interpretation happens off-site Data interpretation happens off-site Current tools focus on 2D data and do not incorporate 3D information Current tools focus on 2D data and do not incorporate 3D information Many expertscollaboration is a must! Many expertscollaboration is a must!

4 Archaeological Excavation at Monte Polizzo, Sicily, Summer 2003 Ian Morris, Director (Stanford University)

5 Working in the field!

6 Meshed 3D Model (13 scans)

7 We have a 3D site model, now what? RealVirtual

8 High Resolution Images Drawings Field Notes Panoramic Images Videos GIS Data

9 High Resolution Images Drawings Field Notes Panoramic Images 3D Object Models 3D Site Model Videos GIS Data Database

10 Two Problems How to combine all this data in one seamless environment? How to combine all this data in one seamless environment? How to make it easy to interact with? How to make it easy to interact with?

11 VITA: Visual Interaction Tool for Archaeology Multiple users Multiple users Multiple displays Multiple displays Projected tabletop Projected tabletop Handheld Handheld High-resolution monitor High-resolution monitor See-through head-worn See-through head-worn Multiple interaction devices Multiple interaction devices MERL DiamondTouch table MERL DiamondTouch table EssentialReality P5 gloves EssentialReality P5 gloves Speech input Speech input 6DOF tracker 6DOF tracker

12 Design Considerations Use the most appropriate display for the given data Use the most appropriate display for the given data Facilitate both human-system and human-human interaction Facilitate both human-system and human-human interaction

13 Modular Architecture AR ModuleDT Module SCREEN Module HANDHELD Module AR Module … Database Message Facilitator 2D Modules 3D Modules

14 AR Module Components Head Tracker Hand Tracker (Intersense IS900) Sony Head-Worn Display (LDI-D100B) Microphone P5 Glove (Essential Reality) DiamondTouch Table Connector (MERL)

15 Life-size Immersive Exploration

16 3D Multimodal Interaction Provide natural interaction mechanism for our 3D environment Provide natural interaction mechanism for our 3D environment Modalities Modalities Speech: IBM ViaVoice 10 Speech: IBM ViaVoice 10 Gestures: EssentialReality P5 glove Gestures: EssentialReality P5 glove Selection statistics: SenseShapes Selection statistics: SenseShapes Focus on selection Focus on selection Based on collaboration with Phil Cohen et al. (ICMI 2003) and SenseShapes (ISMAR 2003) Based on collaboration with Phil Cohen et al. (ICMI 2003) and SenseShapes (ISMAR 2003)

17 VirtualTray

18 Desktop Components Touch-sensitive Projective Display (MERL DiamondTouch) High-resolution Display Handheld Display

19 World-In-Miniature

20 Harris Matrix

21 Enhanced Harris Matrix

22 Enhanced Harris Matrix

23 Cross-Dimensional Hybrid Gestures Synchronized 2D and 3D gestures Synchronized 2D and 3D gestures Facilitate seamless transition across dimensions Facilitate seamless transition across dimensions To appear in IEEE VR 2005

24 Cross-Dimensional Hybrid Gestures Pull Push To appear in IEEE VR 2005

25 Cross-Dimensional Hybrid Gestures Pull Pin Drag Rotate Push To appear in IEEE VR 2005

26 Cross-Dimensional Hybrid Gestures Pull Connect Scale Disconnect Push To appear in IEEE VR 2005

27 Handheld Focus-in-Context Display Movable high- resolution inset Movable high- resolution inset Tracked by DiamondTouch Tracked by DiamondTouch Projection suppressed in its bounds Projection suppressed in its bounds Physical magic lens Physical magic lens

28 Tabletop Interaction

29 User Feedback Overall very positive reaction Overall very positive reaction Archaeologists benefited from: Archaeologists benefited from: Temporal–Spatial connection Temporal–Spatial connection Aggregated collection of all data Aggregated collection of all data Accurate 3D model Accurate 3D model Simple touch-based interactions Simple touch-based interactions Potential for increased collaboration Potential for increased collaboration

30 Room for Improvement Reduce wires Reduce wires Reduce weight Reduce weight Eye occlusion hinders communication Eye occlusion hinders communication Missing data: Missing data: More objects, features, notes and pictures More objects, features, notes and pictures More scans during excavation (time-lapse spatial record) More scans during excavation (time-lapse spatial record) Missing features: Missing features: Virtual scale measure (implemented since) Virtual scale measure (implemented since) Variable site model scaling Variable site model scaling Improved selection in world-in-miniature Improved selection in world-in-miniature

31 Current and Future Work Larger Site: Larger Site: Summer Thulamela, South Africa Summer Thulamela, South Africa Personalized user experience based on expertise Personalized user experience based on expertise Environment management Environment management

32 Acknowledgments Special thanks to: Special thanks to: Peter Allen, Alejandro Troccoli, and Ben Smith (CU Robotics Lab) Peter Allen, Alejandro Troccoli, and Ben Smith (CU Robotics Lab) Ian Morris and Trinity Jackman (Stanford Archaeology Center) Ian Morris and Trinity Jackman (Stanford Archaeology Center) Lynn Meskell and James Conlon (CU Dept. of Archaeology) Lynn Meskell and James Conlon (CU Dept. of Archaeology) Sajid Sadi and Avinanindra Utukuri (P5 glove) Sajid Sadi and Avinanindra Utukuri (P5 glove) Shezan Baig Shezan Baig Mitsubishi Electric Research Labs (DiamondTouch table) Mitsubishi Electric Research Labs (DiamondTouch table) Alias Systems Alias Systems Microsoft Research Microsoft Research Funded in part by NSF ITR Grants IIS and IIS , and Office of Naval Research Contracts N , N , N , and N Funded in part by NSF ITR Grants IIS and IIS , and Office of Naval Research Contracts N , N , N , and N

Questions?

34