Electronic Visualization Laboratory University of Illinois at Chicago Programming the Personal Augmented Reality Immersive System (PARIS) Chris Scharver Thursday September 11, 2003

Electronic Visualization Laboratory University of Illinois at Chicago PARIS MMVR Video Video shot at Medicine Meets Virtual Reality January 2003, Newport Beach, California

Electronic Visualization Laboratory University of Illinois at Chicago What is PARIS? Augmented Reality systems combine real and virtual objects, aligning them so that the user views both. Immersive systems provide head and hand tracking, viewer centered perspective, stereo vision, and absolute scale.

Electronic Visualization Laboratory University of Illinois at Chicago PARIS Features Large field of view immerses the user within the virtual environment Users sees both his hands and the environment Hands do not obscure virtual images Less eye strain due to accommodation

Electronic Visualization Laboratory University of Illinois at Chicago Immersive Displays 1992 CAVE 1995 ImmersaDesk 1996 ImmersaDesk PARIS simulation 2001 PARIS prototype 2002 Manufactured PARIS

Electronic Visualization Laboratory University of Illinois at Chicago Designing PARIS Simulated within the CAVE Users sat at a table Evaluation –Vertical view problematic, users could only look up 10 degrees, but 20 degrees is more comfortable –Screen height and size were adjusted so that reflections would not interfere with the image

Electronic Visualization Laboratory University of Illinois at Chicago Haptics Robotic devices generate force feedback, simulating a sense of touch SensAble’s PHANToM is a popular commercial device –Single point of contact –6DOF input, 3DOF feedback Varying workspace sizes –PHANToM Desktop, 1.5 –PHANToM 3.0

Electronic Visualization Laboratory University of Illinois at Chicago Cranial Implant Research

Electronic Visualization Laboratory University of Illinois at Chicago PARIS™ Hardware and Software Setup

Electronic Visualization Laboratory University of Illinois at Chicago PARIS Display Configuration System setup is similar to other CAVELib- based displays. –Determine world origin –Measure three projection corners –Configure tracking systems if available Screen should remain in a fixed position, and careless bumps can nullify alignment Moving large metal objects can interfere

Electronic Visualization Laboratory University of Illinois at Chicago PARIS Measurements CAVELib origin Projection corners Tracking system, if present, is measured from the CAVELib origin

Electronic Visualization Laboratory University of Illinois at Chicago Incorporating the PHANToM Determine PHANToM device origin At-rest position is not correct! Measure offset from the world origin

Electronic Visualization Laboratory University of Illinois at Chicago Scene Graph Libraries Worlds composed of an object hierarchy Easy to comprehend graphics structures Usually written in C++ Examples: Open Inventor, Performer, OpenRM, OpenSG, OpenSceneGraph

Electronic Visualization Laboratory University of Illinois at Chicago Integrating Software Libraries Open Inventor CAVELib GHOST Open Inventor created by SGI in 1991 Designed for extensibility and interaction Human-readable model files identical to VRML1 Highly portable across operating systems TGS Open Inventor is a commercial implementation General Haptic Open Software Toolkit Device communication for PHANToM Scene graph contains shape and behavior nodes Only includes the most basic graphic rendering! Controls viewer-centered perspective Synchronizes OpenGL buffer swapping among multiple windows Manages user navigation Allows running on multiple display configurations including a simulator

Electronic Visualization Laboratory University of Illinois at Chicago Open Inventor vs. GHOST Traversal is top-down and left-right Primitives, triangle strips, splines Ultimately rendering triangles for every shape Transformations independent of shapes Traversal is top-down only Mathematical primitives Meshes composed of individual triangles Transformations specific to shapes No material or lighting support

Electronic Visualization Laboratory University of Illinois at Chicago Coordinating Haptics and Graphics

Electronic Visualization Laboratory University of Illinois at Chicago RICDemo: a Simple PARIS Application

Electronic Visualization Laboratory University of Illinois at Chicago Simple RICDemo Application: Touch a Ball Requirements –Present 3D colored spheres –Allow touching sphere surfaces –Move spheres while pressing PHANToM button –Provide visual cue to surface contact –Indicate the stylus position Head tracking is not enabled since there is no tracking hardware in place.

Electronic Visualization Laboratory University of Illinois at Chicago Basic CAVELib C Program Structure /* Initialize the library. */ CAVEConfigure(&argc, argv, NULL); /* Give the library the initialization function. */ CAVEInitApplication(app_init, 0); /* Give the library the drawing function. */ CAVEDisplay(app_draw, 0); /* Initialize the CAVE and start the displays. */ CAVEInit(); /* Perform application updates, usually a loop. */ app_update(); /* Clean up & exit */ CAVEExit();

Electronic Visualization Laboratory University of Illinois at Chicago Process Overview main haptics trackingdisplay

Electronic Visualization Laboratory University of Illinois at Chicago Responding to Haptic Events touch ball trigger touch event change ball color PHANToM moves obtain transform update Open Inventor drag ball obtain new position update Open Inventor

Electronic Visualization Laboratory University of Illinois at Chicago RICDemo Classes OIVAppShell: Open Inventor and CAVELib RICDemoApp: Application Behavior RICBall: Open Inventor and GHOST OIVAppShell RICDemoAppRICBall

Electronic Visualization Laboratory University of Illinois at Chicago OIVAppShell Class Stores a root scene graph node Retrieves screen information from the CAVELib for each window Renders the Open Inventor scene Provides functions to CAVELib –Initialization –Once per frame update –Frame draw function

Electronic Visualization Laboratory University of Illinois at Chicago RICBall Contains both Open Inventor and GHOST objects Shares common attributes –Position –Radius Implements behaviors –Touch event changes ball colors –Transform event changes ball position

Electronic Visualization Laboratory University of Illinois at Chicago RICDemoApp CAVELib units must be set to meters –Open Inventor assumes meters as the default –GHOST uses millimeters Hooks triggered by GHOST respond by copying transformations into Open Inventor SoUnits to mm PHANToM transform matrix Ball object transform matrix

Electronic Visualization Laboratory University of Illinois at Chicago Application in Action

Electronic Visualization Laboratory University of Illinois at Chicago Issues Portability with different graphics libraries Data duplication due to multiple representations Dependent upon libraries’ capabilities Must understand intricacies of how the libraries operate and where to place the plugs for them to interact

Electronic Visualization Laboratory University of Illinois at Chicago Questions

Electronic Visualization Laboratory University of Illinois at Chicago Acknowledgements The virtual reality and advanced networking research, collaborations, and outreach programs at the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago are made possible by major funding from the National Science Foundation (NSF), awards EIA , EIA and ANI , as well as the NSF Partnerships for Advanced Computational Infrastructure (PACI) cooperative agreement (ACI ) to the National Computational Science Alliance. EVL also received funding from the US Department of Energy (DOE) ASCI VIEWS program. PARIS is a trademark of the Board of Trustees of the University of Illinois.

Electronic Visualization Laboratory University of Illinois at Chicago Putting RICDemo Together /* Initialize the library. */ CAVEConfigure(&argc, argv, NULL); /* Create the application object. */ RICDemoApp* app = new RICDemoApp(argc, argv); /* Give the library the initialization function. */ CAVEInitApplication(OIVAppShell::DisplayInitCB, 1, app); /* Give the library the drawing function. */ CAVEDisplay(OIVAppShell::FrameDrawCB, 1, app); CAVEFrameFunction(OIVAppShell::FrameUpdateCB, 1, app); /* Initialize the CAVE and start the displays. */ CAVEInit(); /* Perform application updates, usually a loop. */ app->update(); /* Clean up & exit */ CAVEExit();