2. Effects of Surface Characteristics on Alignment of Real and Graphic Objects in Stereoscopic Augmented Reality Environments Ming Hou Ergonomics in Teleoperation and Control (ETC) Laboratory Department of Mechanical and Industrial Engineering University of Toronto January 6th, 2003

3. 3D Measurement in Unmodelled World Virtual Tape Measure

4. Problem: Unknown relationship between real and virtual objects in Augmented Reality displays

5. Research on Real World Targets Video Image Virtual Pointer LINE AREA VOLUME Hemisphere Cylinder

6. Pseudo-Transparency Phenomenon Real Object Real Object (in Video) Virtual pointer in front of surface Virtual pointer behind real surface Virtual pointer behind real surface “Transparency Effect” “Transparency Effect” Breakdown of Fusion + Fusion Conflict

7. Fuse Real SurfaceFuse Virtual Pointer Virtual pointer in front of surface Virtual pointer behind real surface Virtual pointer behind real surface Conflict between Binocular Disparity and Occlusion Cues

8. Theory of Surface Interaction: Fusion Breakdown depends on Texture Density Low Texture DensityHigh Texture Density Virtual pointer in front of surface Virtual pointer behind real surface Virtual pointer behind real surface

9. Research Motivation Is this conflict really significant? If yes, can it be used as an extra cue for detecting interactions between real and virtual objects, and thus locating the real objects in AR environment more easily and accurately?

10. Main Hypotheses #1 More accurate to indicate position on (curved) surface with high (HTD) than with low (LTD) More accurate to indicate position on (curved) surface with high texture density (HTD) than with low texture density (LTD) HTD better than LTD

11. Main Hypotheses #2 Orientation of observer relative to target surface will have influence Orientation of observer relative to target surface will have influence Centre different from Off-centre Top View Real Surface Virtual Pointer Stereo Camera Top View Real Surface Virtual Pointer Stereo Camera

12. Main Hypotheses #3 Form of Virtual Pointer (VP) will have impact on alignment performance Form of Virtual Pointer (VP) will have impact on alignment performance LINE AREAVOLUME

13. Main Hypotheses #4 B inocular disparity (i.e. crossed vs 0 vs uncrossed) will affect alignment performance B inocular disparity (i.e. crossed vs 0 vs uncrossed) will affect alignment performance No disparity (ND) > Crossed (C) or Uncrossed (UC) NDCUC

14. Experimental Investigation of AR “ Surface Effects ” Stereo Camera Indigo 2 Cylinder Barrier Virtual Pointer Stereoscopic AR Display Spaceball

15. Methodology Expt. # Factorial Design Independent Variables Dependent Variables (Measurement) #12x2x3x3 Texture Density Surface Orientation Binocular Disparity VP Orientation Placement Error between perceived target and its actual position on real surface #22x2x3x3 Texture Density Surface Orientation Binocular Disparity VP Form Placement Error + Confidence Rating + Preference Rating #35x5 Texture Density Surface Orientation Placement Error + Angular Error between estimated normal and real surface normal

16. 6 Images (15 paired comparisons) Virtual Pointer Form LINEAREAVOLUME Texture Density High Low Subjective Comparisons Ease of Use Transparency Ease of Fusion

17. Methodology Expt. # Factorial Design Independent Variables Dependent Variables (Measurement) Expt. #1 2x2x3x3 Texture Density Surface Orientation Binocular Disparity VP Orientation Placement Error between perceived target and its actual position on real surface Expt. #2 2x2x3x3 Texture Density Surface Orientation Binocular Disparity VP Form Placement Error + Confidence Rating + Preference Rating Expt. #3 5x5 Texture Density Surface Orientation Placement Error + Angular Error between estimated normal and real surface normal

18. Placement Error vs Texture Density - 1 Main Experimental Results - 1 Farther Closer \ \ Cylinder Surface \ \

19. Placement Error vs Texture Density - 2 Main Experimental Results - 2

20. Placement Error vs Surface Orientation - 1 Main Experimental Results - 3 Closer Farther

21. Placement Error vs Surface Orientation - 2 Main Experimental Results - 4 1 2 3 4 5

22. Angular Error vs Surface Orientation Main Experimental Results - 5 1 2 3 4 5

23. Subjective Comparison Results w.r.t. Ease of Use and Ease of Fusion Volume VP better than other VPs, regardless of texture density w.r.t. Transparency Volume VP + Highly textured least transparent combination Main Experimental Results - 6

24. Texture Density High better than Low Surface Orientation Centredifferent fromOff-Centre Summary of Results - 1

25. VP Form VOLUME > AREA > LINE (subjective comparisons) Binocular Disparity Summary of Results - 2NDCUC

26. Conclusions - 1  Perceptual conflict does exist when real and virtual objects interact in 3D AR environments (a model proposed) extra  Perceptual conflict can be used as extra depth cue to indicate interaction between real and virtual objects  Optimal density value for Random Dot texture pattern was found as Engineering Solution for accurate 3D measurement

27. R-V Interaction Process Model behind Localisation Achieved Fusion difficulty No fusion difficulty No Yes (Transparency) at behind (Breakdown/Conflict) Perception (Stereo Matching) AR Display User External World Display of virtual pointer (VP) superimposed upon real object surface VP Controller Cognition / Decision Making (Stereo Matching + Cue Conflict Resolution) VP behind, or at, real surface? Stereo Matching : VP in front of real surface? Fused Image? Adjustment

28. Conclusions - 1  Perceptual conflict does exist when real and virtual objects interact in 3D AR environments (a model proposed) extra  Perceptual conflict can be used as extra depth cue to indicate interaction between real and virtual objects  Optimal density value for Random Dot texture pattern found as Engineering Solution for accurate 3D measurement

29. Conclusions - 2  Target position does affect alignment task: centrally located targets benefit performance, but have disadvantage when along the line of sight  Volumetric stereo graphic cursor (more fusable features along three dimensions) is subjectively the most favoured VP  “Pseudo-transparency” contributes literature of depth perception cues (shape-from-texture and stereo)

30. Implications for AR Interface Design  Random Dot Stereogram enhances 3D alignment performance extra  Perceptual conflicts can be used as extra depth cue to detect real object position  3D VP better than other VPs  Perceptual errors always exist

31. Limitations  Implementation  Display Mode : stationary display without motion parallax and motion perspective  Binocular Disparity : confounded with size cue and resolution  Scope  VP Design : line thickness of wire-frame VP  Texture Pattern : square Random Dot pattern

32. Future Work/Impact - 1  Near Term Research  Projected lighting : more practical  Motion parallax with Video-HMD  Computational vision may alleviate some error (being investigated) Simulated Projector Stereo Cameras Projected lighting with random dot texture pattern

33. Future Work/Impact - 2  Long Term Interests  Integration of Computer Assisted Object Detection in AR Displays  See-through HMD AR for Dismounted Soldiers in the Battlefield (e.g., Perceptual Conflicts, Navigational Aids, etc.)  Other Human Computer Interaction (HCI) Topics (e.g., Integration of Electronic Information with Human-Machine Systems, etc.)

34. Acknowledgement Dr. Julius Grodski at Defence Research & Development Canada (DRDC) – Toronto, Prof. Allison B. Sekuler and Prof. Paul Milgram at University of Toronto Dr. Stephen Ellis at NASA and Prof. Stanley Hamstra at University of Toronto Natural Sciences and Engineering Research Council (NSERC) Doctoral Scholarship Institute of Robotics and Intelligent Systems (IRIS), Canada Ontario Graduate Scholarship (OGS)

35. Experimental #1 Main Result Farther Closer \ \ Cylinder Surface \ \

36. VP Form and Orientation in 1 st Experiment : DiagonalHorizontal Vertical Observer

37. Interaction in Experiment # 1 Interaction between Surface Texture (High vs Low) and Target Position (Center vs Right) VP Placement vs Texture and Target Position (Error Bar = +/- 1SD, F(1,9) = 246.33, P< 0.001) -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 0 1 CentreRight Angular Displacement of Target Normal Placement Error (cm) High Low

38. Paired Comparison Result Ease of Use Transparency Ease of Fusion Image # 1 2 3 45 6 Mean Z score 01.151.71 2.263.16 Image # 2 1 3 6 5 4 0 0.250.89 2.973.824.37 Image # 1 2 3 5 4 6 0 0.690.732.08 2.14 2.26

39. Measurement of Placement Error along Surface Normal Estimated Target Positive Error Real Surface Target Surface Normal Positive error shows the estimated target is inside the sphere along surface normal

40. Placement Error vs Texture Density for Experiments 2 and 3 Experiment 2 Experiment 3

41. Definition of Angular Error X Y P N T Z O Q R S Altitude Error : Angular distance between estimated normal TP and real surface normal TN Azimuth Error : Angular Error between horizontal projection OR and OS

42. Angular Bias in Spherical Coordinate Y Bias Area S E N X Z T Angular bias tilted upwards from real surface normal (TN)

43. Example of Distribution of Altitude and Azimuth Angular bias between estimated and real surface normal

44. Side View Stereo Cameras Ground Truth Measurements in Real Scene Cylinder Stimulus Real Distance in Depth (Z) Cylinder Calibration object Iron plate Top View Real World Origin (0,0,0)

45. Registration Verification: Measurement of a Pin and a Cube Calibration Target Stereo Cameras Calibration Cube

46. VP Resolution for Experiment 2 VP Resolution and Accuracy Tests

47. Retinal Disparity in Stereoscopic Display for One Pixel Separation (exaggerated) bb aa C’ d ZDZD XDXD 2e P = 1 pixel separation Apparent position of point C’, due to 1 pixel horizontal disparity Stereoscopic Display Monitor Viewer’s Eye  b /2

48. Psychophysical Standard for Texture Density Control Spatial scale (size) Homogeneity (spatial regularity, density is approximately constant over the surface) Isotropy (no orientation bias, equally to be oriented in all directions) – compression

49. Practical Augmented Reality Example Distance between point 1 and point 2 is 7mm Coordinate of Point 1 (2.3, 14.7, 96.2) Coordinate of Point 2 (1.8, 14.4, 95.8) Virtual Tape Measure for Minimally Invasive Surgery