Interactive and immersive technologies: research and experience of application Saint-Petersburg State University of Aerospace Instrumentation Dept. of Computing Systems and Networks Laboratory of computer graphics and virtual reality A.A.Nikitin, A.V.Nikitin, A.A.Nikitina, N.N.Reshetnikova

Fields of application Culture Cultural tourism Art Education Science Industry Health and Medicine Fitness Entertainment Advertising Smart City and other Directions  Interactive 3D models of objects and events with contextual information.  Interactive and immersive 3/4/5/6 D representations of objects and events.  Modern guides (navigators) presenting historical cultural information about surrounding objects.  Serious massively multiplayer online games with elements of geopositioning and augmented (mixed) reality.  Interactive electronic technical manuals, 3/4D passports. Electronic educational resources. and other Technology Interactive and Immersive 3D Virtual Reality/Worlds Augmented reality Augmented virtuality Multiuser VW/MR Collaborative VW/MR Multimodal interfaces Tangible interface Internet of Things Adaptive interface Simulators Desktop, mobile, wearable and surface projection systems and other Research and custom software development projects

Areas of research and experiments Interactive and immersive 3/4/5/6 D representation of events and objects Augmented reality o Addition and combination of different sensations - visual, auditory, tactile, etc o Use markers, images, geo-location, in- and outdoor, single- and multi-user Augmented virtuality Collaborative VW/MR o Visual, voice, text and nonverbal communications in collaborative work o Autonomous characters with intelligent behavior Multimodal interfaces o Combination of different modalities o Use of human physiological parameters - pulse, blood pressure and others. Tangible interface - person interacts with the VW/ MR through real-world objects Internet of Things Adaptive interface o Changes in the properties of the VW/ MR (presentation, navigation), depending on the user's behavior o Personalized interface Simulators - bicycle, car, plane, etc. Serious games, massively multiplayer online games with elements of geopositioning and augmented reality Desktop, mobile, wearable and surface projection systems Devices - 3D Vision, HMD, trackers, gloves, multiscreen, bicycle, etc.

Basic Interactive and Immersive Technologies Virtual reality/worlds (VR/VW) o immersion into artificial environment o movement in a world with 6DOF and interaction with its objects in real time o modeling of human behavior based on real-world physics and artificial intelligence o usage of various characteristics of a person - physical, sensorimotor, etc. Mixed reality (MR) - augmented reality/virtuality (AR/AV) o addition of human perception of real or virtual world respectively by virtual or real objects Multiuser VW/MR (Collaborative VW/MR) o each visitor is represented by special network object-avatar and see other users o actions performed by users in virtual world are synchronized and visible to everyone Multimodal interfaces o support of communication with VW/ MR using more than one modality such as voice, gesture, touch, movement of the head and body, etc.

Interactive and Immersive Theatre Purpose Creation of models’ database for theatrical stages and Russian spectacles to support preservation and promotion of Theatre Arts Team Alexandrinsky Theater, State Theatre Arts Academy, ITMO, SUAI Solutions i3D model of stage and auditorium i3/4D model of "Masquerade“ spectacle Interaction with the model based on gestures Augmented reality component shows information about recognizable points of interest (single/multi-image) Usage Previsualization of stage and spectacle Scientific historical reconstruction Performance and / or games in mixed reality Teaching and the creative process Printed materials based on augmented reality, an appearance of artists and an audience.

Historical reconstructions of Yam Fortress Purpose Preservation and promotion of historical information about the lost object of cultural heritage Resource for economic and social development of the region (cultural tourism) Team Museum Agency of the Leningrad Region, Historical museum of Kingisepp, SUAI, Institute of History of Material Culture RAS, IBRD + FIBP Solutions i3D models of fortresses and warriors (XIY- XYIII centuries) Interaction with the model based on gestures Augmented reality component shows information about recognizable points of interest (single/multi-image) 2D multimedia information system Usage Virtual audio-visual excursions Extension of museum exhibitions Historical game in mixed reality Historical research, educational and artistic reconstruction i3D Theatre for 20 persons

Personality and History of Yam Fortress

State Museum of the History of Religion Purpose Demonstration of religious buildings of cultural heritage Cultural tourism Solutions i3D model of the Temple of Apollo i3D Church of the Holy Sepulcher Multimedia information system Usage Extension of museum exhibitions o Religion of Ancient Greece (Delphi) o Christian relics on the Holy Land Virtual audio-visual excursions Historical research and educational reconstruction Free walk Quiz Versions - electronic kiosk and CD ROM

Russian Ilizarov Scientific Center for Restorative Traumatology and Orthopaedics Purpose Advertisement of Center Innovation in medicine Conservation of Ilizarovov‘s heritage Entertainment for hospital patients Guide for the Center (Medical tourism) Solutions i3D model of buildings (exterior and interior) and surrounding area i3D Ilizarov apparatus, human organs, etc. Augmented reality component shows information about recognizable points of interest (single/multi-marker) Multimedia information systems Usage Virtual audio-visual excursions and free walk Extension of Center’s museum exhibitions “Augmented Reality in Medicine” application o Image-Guided surgery interventions o Books based on augmented reality Promotional and educational materials

Industrial i3D Vizualisation, Interactive Electronic Technical Manuals Purpose Virtual prototyping and simulation Information resources for training and maintenance Solutions i3D model of the buildings (exterior and interior) and the surrounding area I3/4D model of complex technical objects Augmented reality component shows information about recognizable points of interest (single/multi-image) Multimedia information systems Possibilities of usage Advertising campaigns After-sales service - cars, ships, planes, buildings, etc o Maintenance o Repair o Assembly Work Instructions o Operating Procedures Training Extension of corporate museums exhibitions Promotional and educational materials Corporate museums

Russian as foreign language

Cybernetic Bicycle for training, entertainment, medicine and sport

i3D Panorama Battle for Leningrad

Games