Graphical User Interfaces in Virtual Reality Patrick Hartling Virtual Reality Applications Center IEEE VR 2002

VR Juggler — Overview Motivation Motivation Issues to consider Issues to consider Previous work Previous work Current user interface work at VRAC Current user interface work at VRAC

VR Juggler — Motivation Why have GUIs in virtual reality? GUIs are everywhere in the computing world GUIs are everywhere in the computing world Most computer users understand how to use a GUI Most computer users understand how to use a GUI Why not have GUIs in VR? Why not have GUIs in VR?

VR Juggler — Issues with GUIs in Virtual Reality Positives Typically more flexible and more extensible than other input devices Typically more flexible and more extensible than other input devices –No overloading of button presses, gestures, etc. More portable between VR systems than hardware input devices More portable between VR systems than hardware input devices Familiarity from desktop experience Familiarity from desktop experience

VR Juggler — Issues with GUIs in Virtual Reality (cont’d.) Negatives No direct physical correspondence No direct physical correspondence –In VR, a user could grab an object with his/her hand or use a GUI –No GUI in real world for moving the object Remote users may not see GUI [Park00] Remote users may not see GUI [Park00] Use of the interface may be difficult Use of the interface may be difficult –Collision detection –Placement Immersive interfaces have to re-invent GUI technology Immersive interfaces have to re-invent GUI technology

VR Juggler — Previous Work Virtual User Interface (VUI) Originally developed at Iowa State University [Heath98] Originally developed at Iowa State University [Heath98] 3D immersive user interface 3D immersive user interface May be an open source release from Fuel Tech May be an open source release from Fuel Tech

VR Juggler — Previous Work Java Interface to Virtual Environment (JAIVE) Used Java interface on handheld computer [Hill00] Used Java interface on handheld computer [Hill00] Communicated with C++ applications Communicated with C++ applications Interface could be extended from C++ using JAIVE protocol Interface could be extended from C++ using JAIVE protocol

VR Juggler — Previous Work (cont’d.) 3DI Group Doug Bowman, Virginia Tech Doug Bowman, Virginia Tech Investigating differences in system interaction techniques between VE displays Investigating differences in system interaction techniques between VE displays Active on 3DUI mailing list Active on 3DUI mailing list

VR Juggler — Tweek Features GUI panels are dynamically loaded components GUI panels are dynamically loaded components –Panels use standard 2D GUI “widgets” Dynamic interface update and extension Dynamic interface update and extension Multi-language capabilities Multi-language capabilities

VR Juggler — Use of 2D Interface Reasoning Some interaction methods are best suited to 2D interfaces Some interaction methods are best suited to 2D interfaces –Database entry, annotations, precision input 2D interface technology is very mature and familiar 2D interface technology is very mature and familiar

VR Juggler — Hybrid Approach Reuse of GUI Desktop simulation Desktop simulation PDA in a projection-based VR system PDA in a projection-based VR system Immersive 3D space Immersive 3D space

VR Juggler — Same GUI, Multiple Uses PDA in Projection System Offers force feedback Offers force feedback High-resolution display High-resolution display –Back-lit display is needed Widely available Widely available –A person with a PDA can walk into the VR system and get the GUI dynamically

VR Juggler — Same GUI, Multiple Uses (cont’.d) Desktop simulation Standard GUI interface used to control VR software in simulator mode Standard GUI interface used to control VR software in simulator mode Immersive 3D space Incorporates 2D GUI into 3D visualization Incorporates 2D GUI into 3D visualization –3Dwm ( –VR VNC viewers –Part of original design, still a future goal

VR Juggler — Tweek Design Language tools C++ VR applications C++ VR applications Java-based GUI composed of JavaBeans Java-based GUI composed of JavaBeans –Beans could be “pushed” to the GUI while the user is in the virtual space –Dynamic extension of interface CORBA communication between C++ and Java CORBA communication between C++ and Java

VR Juggler — Example Tweek Interface Navigation tool

VR Juggler — Future Work Full implementation of JavaBean “pushing” to Tweek GUI Full implementation of JavaBean “pushing” to Tweek GUI Use of 3Dwm or other tool for bringing 2D GUI into 3D space Use of 3Dwm or other tool for bringing 2D GUI into 3D space

VR Juggler — Online Information 3DUI Under “Projects” block Under “Projects” block –Tweek  Documentation Contains guides and references Contains guides and references

VR Juggler — Conclusion Questions?

References [Park00] Park, K. et al. “Lessons Learned from Employing Multiple Perspectives in a Collaborative Virtual Environment for Visualizing Scientific Data”. In Proceedings of ACM Conference on Collaborative Virtual Environments, San Francisco, California, September 10–12, [Park00] Park, K. et al. “Lessons Learned from Employing Multiple Perspectives in a Collaborative Virtual Environment for Visualizing Scientific Data”. In Proceedings of ACM Conference on Collaborative Virtual Environments, San Francisco, California, September 10–12, [Heath98] Heath, D. “Virtual User Interface (VUI): A Windowing System for VR”. In Proceedings of 2 nd Immersive Projection Technology Workshop, Ames, Iowa, May 11–12, [Heath98] Heath, D. “Virtual User Interface (VUI): A Windowing System for VR”. In Proceedings of 2 nd Immersive Projection Technology Workshop, Ames, Iowa, May 11–12, [Hill00] Hill, L., Usability of 2D Palmtop Interaction Device in Immersive Virtual Environment. Master’s thesis, Iowa State University, Ames, IA, [Hill00] Hill, L., Usability of 2D Palmtop Interaction Device in Immersive Virtual Environment. Master’s thesis, Iowa State University, Ames, IA, [Watsen99] Watsen, K. “A Handheld Computer as an Interaction Device to a Virtual Environment”, In Proceedings of 3 rd Immersive Projection Technology Workshop, Stuttgart, Germany, May 10–11, [Watsen99] Watsen, K. “A Handheld Computer as an Interaction Device to a Virtual Environment”, In Proceedings of 3 rd Immersive Projection Technology Workshop, Stuttgart, Germany, May 10–11, 1999.