© 2006 WMG © 2007 WMG Levels of Realism: From Virtual Reality to Real Virtuality - Perception and Virtual Time Alan Chalmers Andrej Ferko Alan Chalmers † Andrej Ferko ‡ Warwick Digital Laboratory, UK † Warwick Digital Laboratory, UK Comenius University, Slovakia ‡ Comenius University, Slovakia

2 Warwick Digital Lab © 2007 WMG “Holy Grail” Realism in Real-Time  Real-Time ≥ 25 fps  Realism …? Courtesy of recomCGI and Steffen Schraegle, produced via SpheronVR Technology

3 Warwick Digital Lab © 2007 WMG Introduction Virtual reality  1960’s Morton Heilig’s Sensorama Ivan Sutherland’s head mounted display  Modern multi-modal VR systems CAVE’s, data-gloves, etc.  Needs real-time  Trade off realism to achieve this  GPUs make things faster Not physically based

4 Warwick Digital Lab © 2007 WMG Types of realism  1998: Lengyel’s rendering spectrum  Images which are appearance based  Geometry which is physically based  2001: Ferwerda  Physical realism Visual simulation  Photorealism Visual response  Functional realism Visual information

5 Warwick Digital Lab © 2007 WMG Physical realism Exact match  Spectral irradiance values  Match at all points within the real and virtual scene  Scene modelled precisely  Geometry  Materials  Light transport BUT  All include approximations  Especially displays (even HDR is only 3,000 cd/m 2 )

6 Warwick Digital Lab © 2007 WMG Photorealism Resultant rendered image  Indistinguishable from photograph of real scene  Other forms  Lumigraph [Gortler et al. 1996]  Image based lighting [Debevec et al. 1996] But  Make rendered images more like a photograph  Blurring  Increasing Gaussian noise

7 Warwick Digital Lab © 2007 WMG Being there

8 Warwick Digital Lab © 2007 WMG Functional realism Image provides  Sufficient information to enable viewer to complete task  Correct relative size and shape  But not material properties

9 Warwick Digital Lab © 2007 WMG Levels of realism One-to-one mapping  Virtual world to Real world experience  Adopt same learning strategy  Real and virtual worlds  High confidence in results

10 Warwick Digital Lab © 2007 WMG Believable realism Believable Realism!

11 Warwick Digital Lab © 2007 WMG Scruffy textures

12 Warwick Digital Lab © 2007 WMG Classification Is Physics IP Not Physics NP Not Believable NB Is Believable IB Real virtuality Photorealism IBNPIBIP NBNPNBIP Avatars Uncanny valley Cartoons NPR Photographs Games Virtual Reality Functional realism Abstract art Vermeer

13 Warwick Digital Lab © 2007 WMG Comparing real and virtual scenes  Physical test environment with diffuse materials  [Meyer et al. 1986]  Comparing photograph with real scene  [Rushmeier et al. 1995]  Visual Difference Predictor (VDP)  [Daly 1993], [Myszkowski 1998], [Mantiuk et al. 2005]  Karol’s new method  Myszkowski et SIGGRAPH 2008

14 Warwick Digital Lab © 2007 WMG Holistic approach Three different human visions  Monocular, binocular, and ambient Four parallel processes  Head, eye, visual perception, cognitive processes  Visual equivalence  [Ramanarayanan et al. 2007]  Judgement of lightness  [McNamara et al. 2000]

15 Warwick Digital Lab © 2007 WMG Multi-modalities Human perceives environment with all senses  Sight  Sound  Smell  Touch  Taste  Crossmodal effects can be considerable

16 Warwick Digital Lab © 2007 WMG Presence Typically a measure of technical immersion  Higher quality  more immersion But  Immersion in books and films Can quantify user’s engagement with VE  Not a measure of realism

17 Warwick Digital Lab © 2007 WMG Preconditioning Human imagination  Dungeons & Dragons  Virtual Vietnam Habituation  Familiarity may make user perceive less

18 Warwick Digital Lab © 2007 WMG Nature of task Top down process  Attention under volitional control  What we perceive is strongly dependent on the task  Looking for a street sign  Finding a coffee shop  On patrol in (hostile) environment

19 Warwick Digital Lab © 2007 WMG There-reality Same perceptual response as if “there” in real world  Exploits limits of human perception Only render in high quality what user is attending to  Selective rendering  Benefits from crossmodal interactions  Can save significant computational effort Experience  Active – infinitive  limited by technology  Passive

20 Warwick Digital Lab © 2007 WMG The perception equation Function of task (t) and preconditioning (ρ): P(t, ρ) = ω v V + ω a A + ω s S + ω t T + ω f F V=Visuals, A=Audio, S=Smell, T=Taste, F=Feel Σω i = 1  ω i are threshold values  Above threshold there is no perceptual difference

21 Warwick Digital Lab © 2007 WMG Example Preconditioning  Experienced sailor  Sound of water across hull  Feel of wind on skin

22 Warwick Digital Lab © 2007 WMG Other approaches Eg. Neurolinguistic programming (NLP)  Sensory input channels  Visual  Audio  Kinesthetic  Olfactory  Gustatory  Can be used to represent eg. desire, emotions …  Initially: Interpersonal communication model  Now: Business and management training

23 Warwick Digital Lab © 2007 WMG Virtual time issue Perceptual equation re-read…  Sensory input channels in real and virtual time (Borges, 2 past)  Visual  Audio  Kinesthetic  Olfactory  Gustatory  Higher level of perception… VR as a semiotic system  How to define and measure interestingness?  Up to now: data mining, Koestler, engagement in virtual museums

24 Warwick Digital Lab © 2007 WMG Conclusions  In virtual reality striving to achieve  More realism  While maintaining real-time performance  Many applications don’t need physical realism  For those that do  May not be possible to compute total physical realism  Perceptual realism Same user response as if there in real scene being portrayed Can save significant computation by exploiting human perception

25 Warwick Digital Lab © 2007 WMG Challenges Delivery system  Virtual cocoon (light weight, portable system) Determine the “weights” ω i in the perception equation  Empirical studies  fMRI brain imaging  Weights are individual  Some generic similarity  Do NOT have to be determined precisely  As long as we are ABOVE the threshold  Perceptually high-fidelity virtual environments  Real virtuality

26 Warwick Digital Lab © 2007 WMG Acknowledgements  Pavel Zemcik and MEMICS team for inviting  SCCG for enabling long-term discussions  EPSRC for funding “…towards Real Virtuality”  Alan’s group for 23 years of exciting research David Howard + Christopher Moir + red wine  Real Virtuality Further details:

27 Warwick Digital Lab © 2007 WMG Levels of Realism: From Virtual Reality to Real Virtuality - Perception and Virtual Time Alan Chalmers Andrej Ferko Alan Chalmers † Andrej Ferko ‡ Warwick Digital Laboratory, UK † Warwick Digital Laboratory, UK Comenius University, Slovakia ‡ Comenius University, Slovakia