Immersive Rendering

General Idea ► Head pose determines eye position  Why not track the eyes? ► Eye position determines perspective point ► Eye properties determine what part of the real world can be seen ► Screen determines how the virtual world can be seen  Screen is a window into the virtual world

Virtual World Head Pose Eye Position Eye Properties Display Pose Head-Eye-Screen-World Relationships

Depth effects ► Most displays are 2D (pixels) ► Yet we want to show a 3D world! ► How can we do this?  We can include ‘cues’ in the image that give our brain 3D information about the scene  These cues are visual depth cues

Visual Depth Cues ► Cues about the 3 rd dimension – total of 10 ► Monoscopic Depth Cues (single 2D image) [6] ► Stereoscopic Depth Cues (two 2D images) [1] ► Motion Depth Cues (series of 2D images) [1] ► Physiological Depth Cues (body cues) [2]

Monoscopic Depth Cues ► Interposition  An occluding object is closer ► Shading  Shape and shadows ► Size  The larger object is closer ► Linear Perspective  Parallel lines converge at a single point  Higher the object is (vertically), the further it is ► Surface Texture Gradient  More detail for closer objects ► Atmospheric effects  Further away objects are blurrier and dimmer ► Images from

Physiological Depth Cues ► Accommodation – focusing adjustment made by the eye to change the shape of the lens. (up to 3 m) ► Convergence – movement of the eyes to bring in the an object into the same location on the retina of each eye.

Stereoscopic Depth Cue ► Stereopsis ► Stereoscopic Display Technology ► Computing Stereoscopic Images ► Stereoscopic Display and HTDs. ► Works for objects < 5m. Why?

Stereopsis The result of the two slightly different views of the world that our laterally-displaced eyes receive.

Screen Parallax P left – Point P projected screen location as seen by left eye P right – Point P projected screen location as seen by right eye Screen parallax - distance between P left and P right P Left eye position Right eye position P left P right P left P Display Screen Object with positive parallax Object with negative parallax

How to create correct left- and right-eye views ► What do you need to specify for most rendering engines?  Eyepoint  Look-at Point  Field-of-View or location of Projection Plane  View Up Direction P Left eye position Right eye position P left P right P left P Display Screen Object with positive parallax Object with negative parallax

Basic Perspective Projection Set Up from Viewing Paramenters Y Z X Projection Plane is orthogonal to one of the major axes (usually Z). That axis is along the vector defined by the eyepoint and the look-at point.

What doesn’t usuallywork Each view has a different projection plane Each view will be presented (usually) on the same plane

What Does Work ii

What if you don’t have 2 displays? Look at point Eye Locations Look at point Eye Locations No Yes

Asymmetric Camera Frustum Images from: us/stereographics/stereorender/

Cue Mismatch: Accommodation/ Convergence Display Screen

Position Dependence (without head-tracking)

Interocular Dependance F Modeled Point Perceived Point Projection Plane True Eyes Modeled Eyes

Two View Points with Head-Tracking Projection Plane Modeled Point Perceived Points Modeled Eyes True Eyes

Stereoscopic Display ► Stereoscopic images are easy to do badly, hard to do well, and impossible to do correctly.

Stereoscopic Displays ► Stereoscopic display systems presents each eye with a slightly different view of a scene.  Time-parallel – 2 images same time  Time-multiplexed – 2 images one right after another

Time Parallel Stereoscopic Display Two Screens ► Each eye sees a different screen ► Optical system directs correct view ► HMD stereo Single Screen ► Two different images projected ► Images are colored or polarized “differently” ► User wears glasses to filter out L image for L eye and R image for R eye

Passive Polarized Projection ► Linear Polarization  Ghosting increases when you tilt head  Reduces brightness of image by about ½  Potential Problems with Multiple Screens ► Circular Polarization  Reduces ghosting  Reduces brightness  Reduces crispness

Problem with Polarization Technology for Multiple Screens ► With linear polarization, the separation of the left and right eye images is dependent on the orientation of the glasses with respect to the projected image. ► The floor image cannot be aligned with both the side screens and the front screens at the same time. ► Solution?

Time Multiplexed Display ► Left and right-eye views of an image are computed ► Alternately displayed on the screen ► A shuttering system occludes the right eye when the left-eye image is being displayed

Shutter Glasses

Ghosting ► Some of L image is visible to R eye and vice versa ► Not a problem for HMDs, why? ► About equal problem for polarized and shuttered glasses ► Pixel persistence ► Vertical screen position of the image.

Other stereo limiting factors ► Right and left-eye images do not match in color, size, vertical alignment. ► Distortion caused by the optical system ► Resolution ► HMDs interocular settings ► Computational model does not match viewing geometry.

Summary ► Monoscopic – Interposition is strongest. ► Stereopsis is very strong. ► Relative Motion is also very strong (or stronger). ► Physiological is weakest (we don’t even use them in VR!)

Stereo Capable HMDs ► Each has some way to send two independent signals  Usually dependent on both graphics card and API ► Lab has 4 different HMDs each with different requirements to produce stereo ► Head Tracking is a separate concept

Emagin Z800 ► Must use machine with Geforce 7900 GPU (on back wall) to get quality stereo ► NVIDIA API (old version) to set convergence and separation ► Must be 800x600x60hz ► Controlled Internally LRLRLRLRLRLRLR ► Best resolution, best color, best head-tracking, best comfort, worst hardware requirement

Vuzix VR920 ► Use their API ► Must hack Ogre or use OpenGL or DirectX directly ► 640x480, frame interleaved LRLRLRLRLRLRLRLR…. ► Very uncomfortable, bad FOV, but internal head tracking

Vuzix Wrap 920 ► Use Internal API (like VR920) ► Use side-by-side stereo (two viewports) settings in control box ► 640x480, two 320x480 viewports ► Coolest looking, worst FOV

VR Research V6 ► Dual Input (L and R connections)  Must have two outputs on graphics card ► 1280x480 in horizontal span window mode  Two 640x480 viewports side by side ► 2 640x480 fullscreen windows ► Best FOV (60 degrees)

Other stereo options ► 2 Viewsonic projectors and Asus 3D monitor  1024x768x120hz frame interleaved  Use new NVIDIA Api (convergence and separation) or Use QuadBuffered OpenGL (or hacked ogre)  Must use newer NVIDIA (desktop) GPU or hacked laptop driver  Use NVIDIA 3D Vision Shutter glasses

Anaglyph Stereo ► Red Cyan glasses ► Render scene twice  First pass – Left - Convert to grayscale then to Red (R channel)  Second pass – Right - Convert to grayscale then to Cyan (BG channels) ► Worst effect ► Easiest to deploy