1. Computer Vision Aids for the Blind and Low-Vision Patients Itai Segall & Ron Merom Advanced Topics in Computer Vision Seminar April 3 rd, 2005

2. Introduction 180 Million people worldwide, who are visually disabled. 45 Million legally blind. [Vision 2020, 2000] This number is expected to double by the year 2020. [Vision 2020, 2000] Efforts are made in various fields to help people with visual impairments.

3. Types of Visual Impairments Scotomas

4. Types of Visual Impairments Scotomas CFL (Central Field Loss)

5. Types of Visual Impairments Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss)

6. Types of Visual Impairments Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss) Hemianopia

7. Types of Visual Impairments Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss) Hemianopia Total Blindness

8. Lecture Outline Studying the problem Suggested Solutions – Eyewear – Enhancement of TV images – Navigation Aids

9. Studying the Problem Example: How Does the Visual System Deal with Scotomas ? [D. Zur, S. Ullman, 2002]

10. What is a Scotoma? Retinal scotomas can be caused by various diseases such as age-related macular degeneration (AMD) “Visual scientists sometimes pass their time during a boring lecture by staring at a light on the ceiling until it produces a vivid afterimage. The afterimage can be used to blot out the lecturer’s head.” 1 1 Morgan, M. “Making holes in the visual world”, 1999

11. Filling-in of Visual Patterns Patients with small enough scotomas perceive the world as uninterrupted Question: how does the visual system deal with missing information? – eye movements – ignored – filled in

12. Filling-in of Visual Patterns – cont. Why study it? – Better understanding of the visual system – Study can lead to developing visual aids Blind Spot – Extensively studied

13. Experiment Subjects: patients with scotomas Show various visual patterns – Short period of time (400ms) Patients were asked to: – 1. Rate uniformity – 2. When designated as non-uniform, choose: Blur Straightness Contrast

14. Results PatternReport

15. Results

16. Vs. Results

17. Conclusions Missing information is filled-in, not ignored Higher density  Better filling-in Higher regularity of stimulus  Better filling-in

18. Lecture Outline Studying the problem Suggested Solutions – Eyewear – Enhancement of TV images – Navigation Aids

19. Eyewear – classical solutions Hemianopia – Binocular sector prisms PFL-Minifying Devices CFL-Magnifying Devices

20. Eyewear Problem: these solutions correct one problem while creating another one Multiplexing approach: [Peli, 2001] – Combine a few information streams – But make sure they can be separated by the visual system Types of multiplexing: – Temporal – Spatial – Bi-ocular – Composite

21. Temporal Multiplexing Different signals at different times Healthy people use temporal multiplexing Bioptic Telescope (for CFL)

22. Spatial Multiplexing Show different information in different parts of the field of view Micro-Telescope (for CFL)

23. Bi-Ocular Multiplexing Expose each eye to different information May seem too confusing, but experiments show patients adapt Implantable Miniaturized Telescope (for CFL)

24. Composite Multiplexing Devices that implement more than one type of multiplexing Peripheral Monocular Prism (for Hemianopia)

25. Composite Multiplexing - cont

26. Peripheral Monocular Prism combine: – Bi-ocular multiplexing – Spatial multiplexing – Spectral multiplexing

27. Composite Multiplexing 2 Minified Contours Augmented View A computer-aided device for PFL

28. Composite Multiplexing 2 - cont

29. Lecture Outline Studying the problem Suggested Solutions – Eyewear – Enhancement of TV images – Navigation Aids

30. Enhancement of TV Images TV serves as an important medium for retrieving information, entertainment and education Visual impairments make watching TV difficult

31. Enhancement of TV Images – cont. Previous experiments: enhance high frequencies But, studies show that the periphery is more sensitive to wideband enhancements  CFL patients need a different solution Idea: explicitly emphasize edges and bars in the image domain [Peli et al, 2004]

32. Enhancement of TV Images - cont. First – detect edge & bars [Peli, 2002] : Use a visual system-based algorithm Morrone, Burr ’88: edges and bars are where Fourier components come into phase with each other.  In order to find edges and bars, look for phase congruency =

33. Enhancement of TV Images – cont. Simplified feature detection algorithm: – Find congruent polarities instead of congruent phases of Fourier components

34. Algorithm for edge & bar detection = ++ Apply bandpass filters Binarize results

35. Algorithm for edge & bar detection = ++ Apply bandpass filters Binarize results Find congruencies

36. Algorithm for edge & bar detection = Apply bandpass filters Binarize results Find congruencies

37. Algorithm for edge & bar detection = Apply bandpass filters Binarize results Find congruencies

38. Enhancement of TV Images – cont. A more interesting example:

39. Wideband enhancement algorithm Create feature map Substitute/Add map to original image Features can be weighted according to their magnitude

40. Low Enhancement Level

41. Medium Enhancement Level

42. High Enhancement Level

43. Medium Enhancement Level

44. High Enhancement Level

45. Enhancement of TV Images – Experimental Results Most CFL patients selected a slightly enhanced image But… when asked to compare it to the original image, they didn’t find it to be much better  Why? – Any enhancement necessarily distorts the image – High contrast features were enhanced much more than moderate ones

46. Lecture Outline Studying the problem Suggested Solutions – Eyewear – Enhancement of TV images – Navigation Aids

47. Navigation Aids Classics: a cane & a guide dog Will discuss two solutions – Specific – locate & recognize signs – General – first steps towards an “inter-sensory” solution

48. Sign finding “Talking Signs” Obvious problem: should be installed Suggested solution: Signfinder [Yuille et al., 1999] – as an example, we’ll discuss (American) stop signs

49. What does it take to be a stop sign? Being red and white? Being octagonal?

50. Then how to find stop signs? Assumptions: – Two-colored – Stereotypically shaped – There exists a set of typical illuminants Preprocessing – find this set

51. Multiplicative model: Observed color = true color X illuminant Use a database of labeled signs to find typical illuminants How? Preprocessing: find set of typical illuminants 804030 RGB 443 RGB 2010 RGB X =

52. Preprocessing: find set of typical illuminants – cont. Manually mark signs 2-means, for each marked sign (R r 1, G r 1, B r 1 ) ; (R w 1, G w 1, B w 1 ) (R r n, G r n, B r n ) ; (R w n, G w n, B w n )

53. Preprocessing: find set of typical illuminants – cont. Energy function: Minimize using SVD  Get a set of typical illuminants and “true red”, “true white” - Green component of the illuminant in image α - Green component of observed red in image α - Green component of “true” red color Where: - Green component of “true” white color Remember: Observed = True X Illuminant

54. Algorithm Now that we have typical illuminants: Algorithm – Find seed candidates in the image – Find the boundary of the sign – Align it to be fronto-parallel – Recognize it as a “stop sign”

55. Goal: find red and white windows Finding seed candidates

56. (Observed) / (True white) = (Illuminant) Finding seed candidates – cont. Illuminance for red Illuminance for white NNNNNNNN NNNNNNNN NNNNNNNN TTTNNNNN TTTNNNNN TTTNNNNN TTTNNNNN TTTNNNNN TTTTTTTT TTTTTTTT TTTTTTTT TTTTTTTT NNNNTTTT NNNNTTTT NNNNTTTT NNNNTTTT TTT TTT TTT TTT TTT TTTTTTTT TTTTTTTT TTTTTTTT TTTTTTTT TTTT TTTT TTTT TTTT Mean (R 1,G 1,B 1 ) (R 2,G 2,B 2 ) ≈ ? Remember: Observed = True X Illuminant R,G,BR,G,B

57. Finding seed candidates – cont. Seeds - results

58. Boundary detection OK, so we have seeds – now we want to grow them and find the boundaries… New pixel: close to red, white or neither

59. Boundary detection – cont. What if the sign is partly shadowed? Define a standard red as: R>128, G,B<0.8R Start with seeds for which the red is non-standard Then add pixels which are red or white with standard illuminant

60. Boundary detection – cont. Problem: results do not yield straight and exact boundaries. Idea: use a variant of Hough transform to find the edges

61. Boundary detection – cont. 1. Find center of mass of red pixels. Use this as the center of the image.

62. 2.At each pixel, vote for s which split the neighborhood 3.Find the most popular edges Boundary detection – cont.

63. Optimization: send rays from the center out, and look only at locations where these rays last contain red pixels

64. Aligning the sign Usually the sign takes up a small part of the image  a narrow field of view  affine transformation relates the sign and the fronto-parallel prototype.

65. Aligning the sign – cont. Define unknowns: – A, – The affine transformation – V i,a – Does data corner i relate to model corner a? And an energy function: And minimize using EM algorithm  Get affine transformation params and corner matchings Matched corners transformed closely to the model corners Unmatched corners pay a penalty

66. Results

67. Recognizing the sign Now, that’s trivial

68. Signfinder Handles partly occluded and partly shadowed signs What about different signs? Currently manufactured by Blindsight corp.

69. Navigation Aids 2 – An Inter-sensory Solution An idea: when you can’t use your eyes, use your ears instead… How would one transform an image to sound? Grayscale Image Left-Right Up-Down Brightness Sound Time Pitch Volume [Stoerig et al., 2004]

70. Seeing through the ears Example 1: Example 2: What is this? Answer: The last one: Answer: But this is cacophony, can one really learn this?

71. “vOICe” experiment Blindfolded Practices Geometric Natural

72. “vOICe” experiment – cont. Results: – Geometric Images – no improvement – Natural images – big difference Blindfolded Blindfolded, Practices Practices

73. “vOICe” experiment – cont. fMRI results: Differences between blindfolded subjects

74. “vOICe” experiment – cont. And one very interesting result: Day 8Day 15Day 21 Could be a natural image, no idea what. Ominous, planes intermingle Could be anything. Very heterogeneous; reminds me most of a plant PlantA plant, no doubt. And a bar at the bottom

75. “I shall never forget the shock and joy of first glimpsing down my hallway and seeing blinds hanging on the window.” Pat Fletcher vOICe

76. Summary Example of vision impairment research Solutions – Eyewear devices that use multiplexing – Electronic image enhancement – Sign-finding and recognizing – Turning images to sounds