Computer Vision Aids for the Blind and Low-Vision Patients Itai Segall & Ron Merom Advanced Topics in Computer Vision Seminar April 3 rd, 2005

Introduction 180 Million people worldwide, who are visually disabled. 45 Million legally blind. [Vision 2020, 2000] This number is expected to double by the year [Vision 2020, 2000] Efforts are made in various fields to help people with visual impairments.

Types of Visual Impairments Scotomas

Types of Visual Impairments Scotomas CFL (Central Field Loss)

Types of Visual Impairments Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss)

Types of Visual Impairments Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss) Hemianopia

Types of Visual Impairments Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss) Hemianopia Total Blindness

Lecture Outline Studying the problem Suggested Solutions – Eyewear – Enhancement of TV images – Navigation Aids

Studying the Problem Example: How Does the Visual System Deal with Scotomas ? [D. Zur, S. Ullman, 2002]

What is a Scotoma? Retinal scotomas can be caused by various diseases such as age-related macular degeneration (AMD) “Visual scientists sometimes pass their time during a boring lecture by staring at a light on the ceiling until it produces a vivid afterimage. The afterimage can be used to blot out the lecturer’s head.” 1 1 Morgan, M. “Making holes in the visual world”, 1999

Filling-in of Visual Patterns Patients with small enough scotomas perceive the world as uninterrupted Question: how does the visual system deal with missing information? – eye movements – ignored – filled in

Filling-in of Visual Patterns – cont. Why study it? – Better understanding of the visual system – Study can lead to developing visual aids Blind Spot – Extensively studied

Experiment Subjects: patients with scotomas Show various visual patterns – Short period of time (400ms) Patients were asked to: – 1. Rate uniformity – 2. When designated as non-uniform, choose: Blur Straightness Contrast

Results PatternReport

Results

Vs. Results

Conclusions Missing information is filled-in, not ignored Higher density  Better filling-in Higher regularity of stimulus  Better filling-in

Lecture Outline Studying the problem Suggested Solutions – Eyewear – Enhancement of TV images – Navigation Aids

Eyewear – classical solutions Hemianopia – Binocular sector prisms PFL-Minifying Devices CFL-Magnifying Devices

Eyewear Problem: these solutions correct one problem while creating another one Multiplexing approach: [Peli, 2001] – Combine a few information streams – But make sure they can be separated by the visual system Types of multiplexing: – Temporal – Spatial – Bi-ocular – Composite

Temporal Multiplexing Different signals at different times Healthy people use temporal multiplexing Bioptic Telescope (for CFL)

Spatial Multiplexing Show different information in different parts of the field of view Micro-Telescope (for CFL)

Bi-Ocular Multiplexing Expose each eye to different information May seem too confusing, but experiments show patients adapt Implantable Miniaturized Telescope (for CFL)

Composite Multiplexing Devices that implement more than one type of multiplexing Peripheral Monocular Prism (for Hemianopia)

Composite Multiplexing - cont

Peripheral Monocular Prism combine: – Bi-ocular multiplexing – Spatial multiplexing – Spectral multiplexing

Composite Multiplexing 2 Minified Contours Augmented View A computer-aided device for PFL

Composite Multiplexing 2 - cont

Lecture Outline Studying the problem Suggested Solutions – Eyewear – Enhancement of TV images – Navigation Aids

Enhancement of TV Images TV serves as an important medium for retrieving information, entertainment and education Visual impairments make watching TV difficult

Enhancement of TV Images – cont. Previous experiments: enhance high frequencies But, studies show that the periphery is more sensitive to wideband enhancements  CFL patients need a different solution Idea: explicitly emphasize edges and bars in the image domain [Peli et al, 2004]

Enhancement of TV Images - cont. First – detect edge & bars [Peli, 2002] : Use a visual system-based algorithm Morrone, Burr ’88: edges and bars are where Fourier components come into phase with each other.  In order to find edges and bars, look for phase congruency =

Enhancement of TV Images – cont. Simplified feature detection algorithm: – Find congruent polarities instead of congruent phases of Fourier components

Algorithm for edge & bar detection = ++ Apply bandpass filters Binarize results

Algorithm for edge & bar detection = ++ Apply bandpass filters Binarize results Find congruencies

Algorithm for edge & bar detection = Apply bandpass filters Binarize results Find congruencies

Algorithm for edge & bar detection = Apply bandpass filters Binarize results Find congruencies

Enhancement of TV Images – cont. A more interesting example:

Wideband enhancement algorithm Create feature map Substitute/Add map to original image Features can be weighted according to their magnitude

Low Enhancement Level

Medium Enhancement Level

High Enhancement Level

Medium Enhancement Level

High Enhancement Level

Enhancement of TV Images – Experimental Results Most CFL patients selected a slightly enhanced image But… when asked to compare it to the original image, they didn’t find it to be much better  Why? – Any enhancement necessarily distorts the image – High contrast features were enhanced much more than moderate ones

Lecture Outline Studying the problem Suggested Solutions – Eyewear – Enhancement of TV images – Navigation Aids

Navigation Aids Classics: a cane & a guide dog Will discuss two solutions – Specific – locate & recognize signs – General – first steps towards an “inter-sensory” solution

Sign finding “Talking Signs” Obvious problem: should be installed Suggested solution: Signfinder [Yuille et al., 1999] – as an example, we’ll discuss (American) stop signs

What does it take to be a stop sign? Being red and white? Being octagonal?

Then how to find stop signs? Assumptions: – Two-colored – Stereotypically shaped – There exists a set of typical illuminants Preprocessing – find this set

Multiplicative model: Observed color = true color X illuminant Use a database of labeled signs to find typical illuminants How? Preprocessing: find set of typical illuminants RGB 443 RGB 2010 RGB X =

Preprocessing: find set of typical illuminants – cont. Manually mark signs 2-means, for each marked sign (R r 1, G r 1, B r 1 ) ; (R w 1, G w 1, B w 1 ) (R r n, G r n, B r n ) ; (R w n, G w n, B w n )

Preprocessing: find set of typical illuminants – cont. Energy function: Minimize using SVD  Get a set of typical illuminants and “true red”, “true white” - Green component of the illuminant in image α - Green component of observed red in image α - Green component of “true” red color Where: - Green component of “true” white color Remember: Observed = True X Illuminant

Algorithm Now that we have typical illuminants: Algorithm – Find seed candidates in the image – Find the boundary of the sign – Align it to be fronto-parallel – Recognize it as a “stop sign”

Goal: find red and white windows Finding seed candidates

(Observed) / (True white) = (Illuminant) Finding seed candidates – cont. Illuminance for red Illuminance for white NNNNNNNN NNNNNNNN NNNNNNNN TTTNNNNN TTTNNNNN TTTNNNNN TTTNNNNN TTTNNNNN TTTTTTTT TTTTTTTT TTTTTTTT TTTTTTTT NNNNTTTT NNNNTTTT NNNNTTTT NNNNTTTT TTT TTT TTT TTT TTT TTTTTTTT TTTTTTTT TTTTTTTT TTTTTTTT TTTT TTTT TTTT TTTT Mean (R 1,G 1,B 1 ) (R 2,G 2,B 2 ) ≈ ? Remember: Observed = True X Illuminant R,G,BR,G,B

Finding seed candidates – cont. Seeds - results

Boundary detection OK, so we have seeds – now we want to grow them and find the boundaries… New pixel: close to red, white or neither

Boundary detection – cont. What if the sign is partly shadowed? Define a standard red as: R>128, G,B<0.8R Start with seeds for which the red is non-standard Then add pixels which are red or white with standard illuminant

Boundary detection – cont. Problem: results do not yield straight and exact boundaries. Idea: use a variant of Hough transform to find the edges

Boundary detection – cont. 1. Find center of mass of red pixels. Use this as the center of the image.

2.At each pixel, vote for s which split the neighborhood 3.Find the most popular edges Boundary detection – cont.

Optimization: send rays from the center out, and look only at locations where these rays last contain red pixels

Aligning the sign Usually the sign takes up a small part of the image  a narrow field of view  affine transformation relates the sign and the fronto-parallel prototype.

Aligning the sign – cont. Define unknowns: – A, – The affine transformation – V i,a – Does data corner i relate to model corner a? And an energy function: And minimize using EM algorithm  Get affine transformation params and corner matchings Matched corners transformed closely to the model corners Unmatched corners pay a penalty

Results

Recognizing the sign Now, that’s trivial

Signfinder Handles partly occluded and partly shadowed signs What about different signs? Currently manufactured by Blindsight corp.

Navigation Aids 2 – An Inter-sensory Solution An idea: when you can’t use your eyes, use your ears instead… How would one transform an image to sound? Grayscale Image Left-Right Up-Down Brightness Sound Time Pitch Volume [Stoerig et al., 2004]

Seeing through the ears Example 1: Example 2: What is this? Answer: The last one: Answer: But this is cacophony, can one really learn this?

“vOICe” experiment Blindfolded Practices Geometric Natural

“vOICe” experiment – cont. Results: – Geometric Images – no improvement – Natural images – big difference Blindfolded Blindfolded, Practices Practices

“vOICe” experiment – cont. fMRI results: Differences between blindfolded subjects

“vOICe” experiment – cont. And one very interesting result: Day 8Day 15Day 21 Could be a natural image, no idea what. Ominous, planes intermingle Could be anything. Very heterogeneous; reminds me most of a plant PlantA plant, no doubt. And a bar at the bottom

“I shall never forget the shock and joy of first glimpsing down my hallway and seeing blinds hanging on the window.” Pat Fletcher vOICe

Summary Example of vision impairment research Solutions – Eyewear devices that use multiplexing – Electronic image enhancement – Sign-finding and recognizing – Turning images to sounds