Interactive and immersive technologies: research and experience of application Saint-Petersburg State University of Aerospace Instrumentation Institute of Computing Systems and Programming Department of computing systems and networks Laboratory of computer graphics, virtual and augmented reality

Directions of activities Scientific and experimental research and development of innovative methods and tools for human-computer interface. Development and maintenance of various high-tech products for education, culture, industry, transport, energy, health, security, sport and others. Educational programs in the direction “Informatics and computer techniques“ Bachelor of technique and technology. Master of technique and technology (Multimedia systems and computer graphics).

Areas of research and experiments Interactive and immersive technologies Virtual reality Mixed reality (Augmented reality, Augmented virtuality) Augmented reality Addition and combination of different sensations - visual, auditory, tactile, etc Use markers, images, geo-location, in- and outdoor, single- and multi-user Augmented virtuality Collaborative VR/MR Visual, voice, text and nonverbal communications in collaborative work Autonomous characters with intelligent behavior Multimodal interfaces Combination of different modalities Use of human physiological parameters - pulse, blood pressure and others. Adaptive interface Changes in the properties of the VW/ MR (presentation, navigation), depending on the user's behavior Personalized interface

Examples of technological solutions Applications Scope Science Industry Health and Medicine Culture Museum Archaeology Art Tourism Education Fitness Games Advertising and other Examples of technological solutions Interactive and immersive 3/4 D representations of objects and events with contextual information Serious games with elements of geopositioning Guides (navigators) presenting information about surrounding objects 3/4D passport of the objects and events Electronic educational resources Interactive map and other App Desktop Projection Surface Mobile Wearablе Simulators Devices 3D Vision VR HMD AR Glasses trackers gloves multiscreen 3/4/5 D etc.

Devices and simulators

Network of Motion Simulators Purpose Training Education Entertainment Ergonomics Science and Technology Center Culture Solutions i3D models of terrain and buildings i3D models car, plane/drone, trolley i3D virtual tours (Project Yam Fortress) MultiUser

Cybernetic Bicycle Purpose Entertainment Cultural tourism Health and Medicine Sport, Fitness and Wellness Solutions i3D model SUAI i3D model of bike with character Cyberbike Single- or multiuser modes

VR and AR devices Purpose Training Education Entertainment Ergonomics Industry Science and Technology Center Health and Medicine Solutions i3D models of terrain and buildings i3D models car, plane/drone, trolley i3D model of human

Industrial i3D Applications Purpose Virtual prototyping and simulation After-sales service - cars, ships, planes, buildings, etc Maintenance Repair Assembly Work Instructions Training Operator Training Simulator Normal Conditions Abnormal Conditions Transient Conditions Emergencies Solutions i3D models of terrain, buildings, plant i3/4D models of complex objects, equipment, failures and emergency situations Augmented reality IETM for ship and other systems

Russian Ilizarov Scientific Center for Restorative Traumatology and Orthopaedics Purpose Advertisement of Center Innovation in medicine Conservation of Ilizarovov‘s heritage Entertainment for hospital patients Guide for the Center Education Solutions i3D model of buildings (exterior and interior) and surrounding area i3D Ilizarov apparatus, human organs, etc. Augmented reality (single/multi-marker) Virtual audio-visual excursions and free walk

Managing psychological States of Human Training of attention Sequence of tasks for attention training consisting of displaying basic and distracting objects. Registration and authorization of users, different modes of difficulty, processing of current statuses and statistics of users on the server. Changing a state of mind with breathing techniques Audiovisual virtual environment providing assistance in synchronization of breathing in a certain technique. Selection of a desired state - focus, calming, happiness, energizing, core. Selection of different levels of difficult Treatment of phobias in public speaking The virtual class of students reacting neutrally, positively or negatively to the speaker's speech.

Historical reconstructions of Yam Fortress Purposes Preservation and promotion of historical information about the lost object of cultural heritage The creation of the resource for economic and social development of the region (cultural tourism) The program “Preservation and Use of Cultural Heritage in Russia” Participants Museum Agency of the Leningrad Region Historical museum of Kingisepp Institute of History of Material Culture RAS IBRD + FIBP SUAI VR Lab

Historical reconstructions of Yam Fortress i3D models of fortresses and warriors (XIV + XVI-XVII + XVIII centuries) interactive excursions and free walks

Historical reconstructions of Yam Fortress Historical excursions Virtual audio-visual excursions Gestural Interaction models Historical games Defense and explosion of the fortress Users move freely in the game space 1 or 2 players Augmented reality Museum (20) Territory (5) Mobile, Android, iOS

Historical reconstructions of Yam Fortress Personalities (37) Fortresses (14) of the North-West of Russia Monuments (19)

Historical reconstructions of Yam Fortress

Historical reconstructions of Yam Fortress

Alexandrinsky Theater Purpose Previsualization of stage and spectacle Historical research Teaching and the creative process Publishing with augmented reality Solutions i3D model of stage and auditorium i3/4D model of "Masquerade“ spectacle Gestural Interaction models Augmented reality (single/multi-image)

State Museum of the History of Religion Purpose Extension of museum exhibitions Religion of Ancient Greece (Delphi) Christian relics on the Holy Land Cultural tourism Historical research Education Solutions i3D model of the Temple of Apollo i3D Church of the Holy Sepulcher Multimedia information system Virtual audio-visual excursions Free walk Quiz Versions - electronic kiosk and CD

Russian as foreign language Purpose Education Edutainment  Cultural tourism Solutions i3D model of the Building of the Twelve Collegia (SPbSU) 3D Character models Model educational situations Single- or multiuser modes Information and test system

Games

Thank you for your attention http://guap.ru/labvr guap.cgvr@gmail.com