Artificial vision Tehisnägemine Mihhail Šubin Tallinna Tehnikaülikool, 2010

Contents Historical facts Different approaches Examples, technical details Conclusions Used materials

Some historical facts Cortical implants –William Dobelle Retinal implants –Mark Humayun

Dobelle’s cortical implants Inspired by Giles Brindley’s research (1968) Cortical stimulation of 37 sighted volunteers Stimulation of visual cortex of three blind volunteers who were temporarily implanted for a few days Four blind volunteers implanted (one retained implant for 3 months, one for 14 years, two for over 20 years) (continued)

2002. Eight people received implants on commercial basis. Total of 16 people implanted on commercial basis, last in y W. Dobelle died in y. 2004, soon after this Dobelle Institute in Lisbon was closed. Development of Dobelle brain implant is continued by Stony Brook University and Avery Biomedical Devices Inc.

Retinal implants Proof of principle demonstrated in early 90-s by Mark Humayun Second Sight Medical Products, Inc. Was founded people implanted with first generation implant (16 electrodes). 5 of them still use the device in their homes today Second Sight announced that the U.S. Food and Drug Administration (FDA) has granted approval for up to 20 people who are blind or who have severely impaired vision to participate in the ArgusTM II Retinal Implant feasibility study in the U.S.

Different approaches Cortical (brain) implant Retinal implant Other options?

Brain implant approach William Dobelle –Surface implant –Electrodes mapped after implantation –Relatively high current needed –Results introduced (several working systems) (continued)

Richard Normann –Utah electrode array (intra-cortical) –Lower stimulation currents –More precision (small groups of neurons can be stimulated) –Still no working prototype for human

Retinal implant approach Chip with array of electrodes on the surface of retina –Stimulates retinal nerve cells –Receives data by wired or wireless (radio/optical) channel –Electrode mapping should be 1:1 (unlike cortical implants where electrodes should be remapped to produce logically ordered “picture”) Chip mounted outside the eyeball with only electrodes connected to retina –No optical data channel option –Less problems with excess heat

Other options Optic nerve stimulation Non-intrusive approaches –Sonic vision –Tactile vision

Examples, technical details Dobelle’s Artificial Vision system Utah intra-cortical electrode array vOICe Tactile systems

Technical details Dobelle’s artificial vision system 108websites/group03retinalimplants/multimedia/article.pdf

Array of electrodes: drawing and x-ray image

Phosphene map in visual space (electrodes matrix implanted to the right occipital lobe produces phosphenes in left visual field)

Frame rate 1 to 10 fps (best 4 fps) System needs to be calibrated before use (stimulation current levels may vary) Image from camera is processed –To map electrodes in more-or less logical order –To provide more contrast image (e.g. edge detection) Phosphene map is stable (stimulation of certain brain area produces a phosphene in certain spot in the visual field) –Phosphene map moves in the visual field following view direction (eyes tracking system is needed to stabilize its position) Active phosphenes flicker. No color. Some of the patients experienced seisures while using the system. One of the latest patients could even drive a car using 144 electrode implant (72+72 at each hemisphere)

Utah intra-cortical electrode array

Electrode array used to find out how nail length influences stimulation results

Biocompatibility research shows good results –2-5 uM thick capsule forms around each electrode Implantation technology is important –Best results achieved when array is rapidly inserted into cortical tissues (in 200 uS) At first electrode arrays are used on animals to record electrical activity of neurons while visual (or auditory*) stimulation is applied. Next stage is behavioral experiments *auditory cortex is chosen for behavioral experiments because of the ease of providing auditory stimuli

vOICe Auditory Display Original camera image (left) and spectrographic reconstruction from vOICe “visual sounds” (right)

Brightness corresponds to amplitude, position to frequency Technically proven effective resolution up to about 4000 pixels (voicels) –limited to some 1000 to 4000 pixels maximum due to limitations in human hearing (32 by 32 up to a 64 by 64). 32 by 32 pixel resolution is more than enough to get recognizable images

Frame rate from 1 to 8 fps (depends on resolution) It is proven that auditory stimulation can cause excitation of visual cortex, especially for blind people, but vOICe usability is still highly individual –Some patients claim that after long term training, adaptation and practice they actually see the image, and the sound produced by the system turns to barely noticeable background noise Some sound masking occurs when using the system Image enhancement algorythms (edge detection, high contrast, negative image) can be used to improve perception 1st publication in 1992, system available since 1998

Other options, such as tactile vision substitution systems (TVSS) are also being developed Veresk ( –Electrotactile –1000 electrodes fixed around the patients torso –Electrode spacing is currently 8mm Numerous electrotactile, mechanotactile and thermotactile systems were developed and tested over time

Conclusions Dobelle’s system is proven to work, but looks frightening. Normann’s research is not yet resulted in any practical application, but still seems more solid and promising. Retinal implants seem to have become more popular (than cortical ones) over last 5-10 years. Non-invasive solutions, like sonic vision systems, look perfect (if they really work as described). They also look like the best choice for people who were born blind. Pixelated vision simulators demonstrate how human brain can adapt to lower resolution. Other tests and experiments confirm great flexibility and adaptability of human brain (which is an advantage to be used in implementation of artificial vision system).

List of used materials (vOICe and Dobelle's brain implant comparison) (vOICe demo) (Article in Wired - Dobelle/Normann/Humayun) (Utah Electrode Array - Richard Normann)

(Second Sight retinal implant approval by FDA) sion_Team/Vision.htm (cortical implants in general) websites/group03retinalimplants/multimedia/article.pdf (Dobelle’s article)