NANJING UNIVERSITY 18 MARCH D Display: Current and future technologies in Europe Part 2: 3D Display Research at DMU Phil Surman Wing Kai Lee Imaging and Displays Research Group De Montfort University Leicester UK

NANJING UNIVERSITY 18 MARCH 2007 Presentation 1)Principle of operation of DMU display 2)ATTEST multi-user 3D prototype 3)MUTED 3D display project 4)Future work

NANJING UNIVERSITY 18 MARCH 2007 PRINCIPLE OF OPERATION of DMU Display

NANJING UNIVERSITY 18 MARCH 2007 DMU Display Side mirror Exit pupils Viewers Screen Steering optics Head tracker

NANJING UNIVERSITY 18 MARCH 2007 Exit Pupils VIEWER EXIT PUPIL PAIR L R SCREEN A B C MULTIPLE EXIT PUPILS PLAN VIEWS

NANJING UNIVERSITY 18 MARCH 2007 Exit Pupil Formation Illumination source Lens Exit pupil Screen LENS

NANJING UNIVERSITY 18 MARCH 2007 Multiple Exit Pupil Formation with a Lens Illumination source A Exit pupils Illumination source C Illumination source B Viewer A Viewer C Viewer B Lens and vertical diffuser VIEWING FIELD

NANJING UNIVERSITY 18 MARCH 2007 Exit Pupil Formation with Array Illumination sources Exit pupil

NANJING UNIVERSITY 18 MARCH 2007 Exit Pupil Steering Exit pupil Illumination sources Steering array lenses

NANJING UNIVERSITY 18 MARCH 2007 Exit Pupil Steering Illumination sources Steering array lenses Exit pupil

NANJING UNIVERSITY 18 MARCH 2007 Exit Pupil Steering Illumination sources Steering array lenses Exit pupil

NANJING UNIVERSITY 18 MARCH 2007 Coaxial Optical Element Illumination and refracting surfaces both cylindrical with common vertical axis Aperture centred at axis No off-axis aberrations Light contained in element by total internal reflection Aperture Illumınatıon surface Refractıng surface To screen Lıght contaıned wıthın element by total ınternal reflectıon

NANJING UNIVERSITY 18 MARCH 2007 Collimated Beam Formation

NANJING UNIVERSITY 18 MARCH 2007 Spatial Multiplexing MUX screen From right source To exit pupils R L To right exit pupil To left exit pupil LCD From left source

NANJING UNIVERSITY 18 MARCH 2007 Spatial MUX with Parallax Barrier R R L L Illumination sources Parallax barrier SIDE VIEW LCD

NANJING UNIVERSITY 18 MARCH 2007 Spatial MUX with Lenticular Screen R L Lenticular screen R L Illumination sources SIDE VIEW

NANJING UNIVERSITY 18 MARCH 2007 First Prototype R L Screen assembly Light sources Lower mirror Upper mirror L R Exit pupils This prototype has fixed pupils – its purpose is to demonstrate spatial multiplexing

NANJING UNIVERSITY 18 MARCH 2007 First Prototype UPPER MIRROR LOWER MIRROR SCREEN LAMPS

NANJING UNIVERSITY 18 MARCH 2007 Early Work: Schematic Diagram from PhD FIG.1.1 SCHEMATIC DIAGRAM OF PROTOTYPE 3D DISPLAY LCD (Ch.6) Vertical diffuser (Ch.7) IR camera (Ch.12) Illumination source (Chs.9&10) Head tracking processor (Ch.12) Fresnel lens (Ch.3) Retro- reflector (Ch.12) Viewing field (Ch.8) Folding mirrors (Ch.4) Viewer (Ch.8) L R Multiplexing screen (Ch.5) Exit pupils (Ch.3)

NANJING UNIVERSITY 18 MARCH 2007 Early Work: Head Tracker FIG.12.1 HEAD TRACKING SET-UP IR diodes and camera lens LED array showing head position Retro-reflector Head

NANJING UNIVERSITY 18 MARCH 2007 Early Work: Head Tracker Retroreflector Region imaged by IR array Head (a) View from Camera Lens (b) Red LED Array FIG.12.2 HEAD ‘SHADOW’

NANJING UNIVERSITY 18 MARCH 2007 Early Work: Moving Illumination Source Track Pinion Wheel Stepper motor Left halogen aperture Right halogen aperture Rack FIG.10.2 HALOGEN LAMP ILLUMINATION ASSEMBLY Magnet Reed switch

NANJING UNIVERSITY 18 MARCH 2007 ATTEST PROTOTYPE CONSTRUCTION AND RESULTS

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Array Element and Illumination/Driver Board Aperture printed on strip of film (RH figure) 2 aperture components cemented together with aperture in between To viewer Aperture Driver board LED array Soft Aperture

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Array Element and Illumination/Driver Board This shows first version with 90 x 3mm white LEDs. Exit pupils move in large increments (~30mm)

ATTEST:LCD Diffraction 90 x 3mm WHITE LEDs LED DRIVERS LIGHT ATTEST: Illumination/driver Board Version 1 NANJING UNIVERSITY 18 MARCH 2007

ATTEST: Illumination/driver Board Version x 1 mm surface-mount white LEDs Comprises 16 x 16-element modules

NANJING UNIVERSITY 18 MARCH 2007 WHITE LED & LENS ARRAY DRIVER CHIP HEAT SINK DRIVER CHIP LIGHT MICROLENS ARRAY 16 x 1 mm surface-mount white LEDs Integral driver and heat sink ATTEST: LED Module

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Illumination Sources This shows collimated beams formed in different directions Beam width can be increased by lighting more LEDs

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Multiple Exit Beams Multiple beams formed by lighting several sets of adjacent LEDs

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: DemonstratorArray Constructed for demonstration of multiple exit pupil formation but without use of LCD

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Demonstrator Exit Pupils Beams formed on targets. Polhemus electromagnetic tracker pickups located at targets

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Array Configuration Lower layer Illumination surfaces Refracting surfaces Upper layer Upper layer PLAN VIEW Aperture One ten-element array is used for each of the left and right sets of exit pupils comprises two sets of five staggered elements

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Appearance of Front of Array WSWS WIWI Continuous illumination over this width Aperture images are effective LCD backlight Vertical diffuser required to enable aperture images to illuminate full LCD height

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Soft Apertures BB&CA&B SOFT APERTURES (VIEWED FROM FRONT) B A C Soft apertures allow for constructional errors and aperture image width variation Fading width determined from trials on perception of brightness variation

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Aperture Intensity Variation (a) Appearance of aperture images (b) Intensity variation Distance across array Relative intensity

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Folding Mirrors Mirror Steering optics Virtual image Virtual image Steering optics (a) Without Folding(b) With Folding PLAN VIEWS Virtual arrays formed either side of actual array Reduces housing size

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Folding Folding mirrors Multiplexing screen, LCD and vertical diffuser Steering optics Folding mirror Light path 5 Mirror folding enables same housing size as current rear projected displays (side mirrors not shown)

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Prototype Side mirror Optical array Screen Side mirror

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Plan view of Prototype Optical array Side mirror Screen Side mirror TOP VIEW

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Prototype Side Elevation Side mirror Screen Optical array SIDE VIEW

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Prototype STEERING ARRAY FOLDING MIRROR SCREEN ASSY. Incorporates same large optical elements as used in demonstrator Large cylindrical convex in front of LCD to increase brightness

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Display Sub-pixels 100 milliradians 15µM structure within RGB sub-pixels RGB Sub-pixels Diffraction Very high first-order component

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: LCD Diffraction DISTANCE (mm) RELATIVE INTENSITY (%) Vertical diffraction << horizontal diffraction large first order gives ~ 15% crosstalk

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Exit Pupil Profile LR DISTANCE (mm) RELATIVE INTENSITY (%) Maxima produced by use of discrete components Left eye located at position L Right eye located at position R Profile is convolution of aperture function with diffraction function (PSF)

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: White LED Colour Variation 0 0 X 0.5 Typical white LED variation Y Blue region shows total variation from manufacturer This region divided into four Even with LEDs from one batch, variation still large

NANJING UNIVERSITY 18 MARCH 2007 ATTEST: Further Work Identified Use LCD with suitable sub-pixel structure to minimise diffraction Select appropriate material and manufacturing process to minimise scattering Use single illumination source to illuminate colour and brightness variation Use low etendue illumination source to reduce light loss Reduce housing size - consumer preference is for ‘hang-on-wall’ Develop multi-user non-intrusive head tracker

NANJING UNIVERSITY 18 MARCH 2007 MUTED Multi-user Three- dimensional Television Display

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Brief Summary EU-funded Kicked-off July months duration 30 person years of effort 7 partners including SLE and Fraunhofer HHI

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Technical Summary RBG laser illumination source Provides wide colour gamut Holographic projector-controlled exit pupils Developing multi-user non-intrusive head tracker Human factors issues examined Investigation into low-diffraction LCD Investigation into temporal MUX Exploitation of display in medical applications

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Semi-coaxial Array Bottom layer Top layer Apertures Refracting surfaces Light to LCD Illumination Plane TOP VIEW Array elements have flat back surface – hence semi-coaxial Enables other means of illumination, for example projection

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Optical Array 10CM Illumination in this plane Light from projector Light to screen assy. SECTION OF ARRAY

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Illumination Plane LATERAL POSITION DISTANCE EXIT PUPIL POSITION VIEWER A VIEWER B Each exit pupil position can be mapped to a diagonal series of small sources Slope of diagonal determines exit pupil distance and lateral position the x-co- ordinate

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Optical Array X Y X Y I LLUMINATION P LANE Conventional projection blocks ~ of 95% of light Use of CGH projector utilises complete wavefront on LCOS SLM Binary phase hologram gives around 40% efficiency Investigating use of conjugate image to double efficiency H OLOGRAPHIC P ROJECTION

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Schematic Diagram RGB LASER LCOS LCD OPTICAL ARRAY MOBILE VIEWERS HEAD TRACKER S IMPLIFIED S CHEMATIC D IAGRAM OF D ISPLAY

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Current Status Investigation into aperture-less optical elements for simplified construction Refining LCOS algorithms Measurement of suitable LCD panels wrt to speed and diffraction Deciding multi-user tracker route Low power monochromatic version under construction

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Enabling Technologies MUTED MUTED completed

NANJING UNIVERSITY 18 MARCH 2007 MUTED: Display Performance Display type No. of viewers Viewer movement Motion parallax Acc./con. rivalry Image transparency Binocular Fixed – non HT SingleVery limited No Yes No Single user HT* SingleAdequatePossibleYesNo Multi-user HT*MultipleLargePossibleYesNo Multi-viewMultipleReasonableYes No HoloformMultipleLargeYesNo VolumetricMultiple Large YesNoYes HolographicMultipleLargeYesNo MUTED

NANJING UNIVERSITY 18 MARCH 2007 FUTURE WORK

NANJING UNIVERSITY 18 MARCH 2007 Future Work European Union Framework 7 round of funding started in December 2006 First Call closes 8 th May 2007 Multi-user 3D displays included in call Also high colour gamut Interactivity supported

NANJING UNIVERSITY 18 MARCH 2007 Extract from EU Workplan Advanced visualisation systems and novel display technologies. Visualisation systems extending colour gamut and dynamic range beyond current state-of-the- art, taking into account human vision and perceptual models. They should support multi- viewer, unaided and unrestricted 3D viewing, as well as natural interaction modalities. This includes signal acquisition, processing and representation technologies for 3D-systems.

NANJING UNIVERSITY 18 MARCH 2007 High Efficiency Laser-based Multi- user Multi-modal 3D Display (HELIUM3D) Direct-view laser-based 3D display to be developed Does not require LCD Image information supplied by light valve Illumination source is RGB lasers. High colour gamut Direct-view Does not have light attenuation of LCD - energy efficient Frees reliance on LCD fabrication plants

NANJING UNIVERSITY 18 MARCH 2007 HELIUM 3D: Schematic Diagram Mobile viewers MEMS scanner Head tracker RGB Laser Light valve SLM Screen

NANJING UNIVERSITY 18 MARCH 2007 HELIUM 3D: Colour Gamut

NANJING UNIVERSITY 18 MARCH 2007 HELIUM 3D: Display Functionality Display is functionally scalable Fast light valve speed could enable a different image to be seen by each eye in viewing field Enables motion parallax Each viewer could choose their desired viewpoint if scene captured by a camera array Each viewer could see completely different images to other viewers Display will work in near field and far field modes

NANJING UNIVERSITY 18 MARCH 2007 HELIUM 3D: Near Field Operation Screen around 1 – 1.5 metre from viewer Immerse hands into image – therefore image ~ 0.5 m from user 1 or 2 users, single or collaborative working Large disparities – up to I/O distance Large convergence/accommodation rivalry (human factors work necessary)

NANJING UNIVERSITY 18 MARCH 2007 HELIUM 3D: Near Field Tracking Requires low tracker latency – high latency will affect task performance and could cause nausea Requires high tracker accuracy (more than for just locating exit pupils) Head tracking in x, y and z directions Images rendered in accordance with head co- ordinates

NANJING UNIVERSITY 18 MARCH 2007 HELIUM 3D: Near Field Example ‘ Virtual Clay concept’ - ‘clay’ shaped with naked hands A virtual chunk of clay floats in front of screen Touching and shaping the ‘clay’ with the naked hands enables user to directly manipulate object Approach completely differs from existing techniques - perceptual space matches interaction space This technique potentially useful in medical task applications

NANJING UNIVERSITY 18 MARCH 2007 HELIUM 3D: Far Field Operation Viewing distance around 2 – 4 metres Gaming Television Videoconferencing ………