Retrieval of blink- based lexicon for persons with brain-stem injury using video cameras CVPR Workshop on Face Processing in Video  June 28, 2004, Washington, DC, USA Dr. Dmitry Gorodnichy Computational Video Group Institute for Information Technology National Research Council Canada www.cv.iit.nrc.ca/Nouse www.perceptual-vision.com

This work summarizes the problems and solutions Some people cannot use hands to control computer. For some, blinking is the only way to communicate… Types of reply: (to “Do you need anything?”) “YES” – blink “NO” – no blink Non-binary replies are possible using various types of blinks Main solution so far – EMG (Electromyography) readings - Electrodes are attached to skin around oblicularis oculi muscle used in eye closure. - Not effective, because of electrical noise due to skin conductance changes and slight positioning changes of the electrodes. What computer vision can offer? – Perceptual User Interfaces 1. Using high-resolution video [fpiv04-ey1] – expensive, slow 2. Using structured infrared [fpiv04-ft4] – intrusive, problems with sun-light, eye-glasses 3. Using motion component of video This is how it is done by biological vision systems: people can easily detect a blink in video with faces being as small as 20 pixels between the eyes! Not simple for computer vision systems & not much work seems to be done. This work summarizes the problems and solutions “YES” – two blinks “NO” – three blinks “YES” – long blink “NO” – short blink

Perceptual User Interfaces Internet, tendencias & tecnología La nariz utilizada como mouse En el Instituto de Tecnología de la Información, en Canadá, se desarrolló un sistema llamado Nouse que permite manejar softwares con movimientos del rostro. El creador de este programa, Dmitry Gorodnichy, explicó vía e-mail a LA NACION LINE cómo funciona y cuáles son sus utilidades Si desea acceder a más información, contenidos relacionados, material audiovisual y opiniones de nuestros lectores ingrese en : http://www.lanacion.com.ar/03/05/21/dg_497588.asp Copyright S. A. LA NACION 2003. Todos los derechos reservados. x y , z a , b g PUI monitor binary event ON OFF recognition / memorization Unknown User! Goal: to detect, track and recognize face and facial movements of the user.

Evolution of Perceptual User Interfaces 1998. Proof-of-concept colour-based skin tracking [Bradski’98] – not precise though 2001. Motion-based segmentation & localization – not precise though 1999-2002. Several skin colour models developed - unlikely to get much better than that 2001. Viola-Jones face detection using Haar-like wavelets of intensities [fg02] 2002. Subpixel-accuracy convex-shape nose tracking [Nouse, fg02, ivc04] 2002. Stereo face tracking using projective vision [w.Roth, ivc04] 2003. Second-order change detection [Double-blink, ivc04] 2004. Associative visual memory for localized faces [fpiv04-fa2] Figure: Typical result for face detection using colour, motion and intensity components of video using six different webcams.

Nouse™ perceptual vision technology Precision & convenience of tracking the convex-shape nose feature is such that it allows one to use nose as mouse (or joystick handle) [fg02] Next step taken: for people with normal eye lid motion for both eyes - Use eye blinking to replace mouse click Next step to be taken: for people with disabilities who are able to eye blink (wink) only - Reading eye-blink based lexicon

Detection of “Blinked” commands Overview (using motion component of video): stream of video data: I(t) three video frames: I1, I2, I3 change maps: binary image(s) time-stamped “blink” event “blink”-ed command 0: 1: 2: 3: Stages: 0: Detecting a face (eyes) - optional 1: Detecting changes cause by motion 2. Analyzing the detected changes 3. Combining results over time Constraint: All has to be done in real-time (<100ms) Setup: Video: 160x120 resolution suffices!

How to detect change Difficulty with: 1. moving heads (change around other facial features occurs)  second-order change detection can help 2. changing light conditions  non-linear change detection can help [Durucan’02] 3. due to eye closing vs due eye opening examine both negative and positive change imagess 4. one-eye blinks (winks): lack of spatial constraints, other parts move too describe the change caused by a wink using moments compute moments in overlapping regions: around eye & around face Example: changes caused by right eye blink (right image) can be very similar to those caused by slight head motion (left image)

Second-order change detection Key idea: to separate long lasting change from the most recent change (i.e. to detect change in change) dI’ V It-2 It It-1 dI dI’(shifted by V) - * - changed pixels (1) - unchanged pixels(0) ddI 1: 2: Examples: 1. Closing the protector, while moving a diskette 2. Blinking, while moving a head

How to analyze change images Using second-order change to compute eye closings and openings I I1 I2 _ dIon dIoff + dI dI1 = dIlong ddI dI1(V) & Long-term change dIlong = dI – ddI (not shown) can be used to refine positive and negative change images dIon, dIoff Using moments to recognize blobs of predefined shapes: Computing moments is not robust to outliers:  Outliers can be removed by second order change detection  Computing moments in overlapping regions helps too.

Implementation example using PVS.dll For two-eye blinks: '''initialize camera Sub FormLoad() pvsInitializeCamera(160, 120, 1) nIOD = 20 End Sub ''' define change caused by blink Function EyeLidMoved() As Boolean ... End Function ''' for each captured frame Sub Timer_Timer() pvsInitializeNextFrame() nTimestamp = pvsGetFrameNumber() pvsComputeChanges() ' detect eye closings by computing N,X,Y,dX,dY,tilt in negative change If EyeLidMoved( pvsAnalyzeChange(roi1,dIoff), pvsAnalyzeChange(roi2,dIoff) ) Then addToList(timeline,bClosing,nTimestamp) ' detect eye openings by computing N,X,Y,dX,dY,tilt in positive change If EyeLidMoved( pvsAnalyzeChange(roi1,dIon), pvsAnalyzeChange(roi2,dIon) ) Then add2List(timeline,bOpening,nTimestamp) ' detect blink by analyzing time-line signatures in the last K time-stamped eye lid motions If isThereBlink(timeline, K) Then ... 'Blink detected! EyeLidMoved() { if (r<N<r*iod & r<dX<2*iod & dY<r & tilt<1/4) return true; } // iod is intra-ocular distance, r=iod/4. For one-eye blinks (winks): - additional constraints are needed: on eye / face location (Xe,Ye) - this can be obtained using Viola-Jones face detector.

Associative memory for face analysis Potential approach: - store facial expressions as attractors in the attractor-based neural network. - such network of N binary neurons Yi, converges in time from a stimulus binary visual pattern to an attractor according to the update rule: Main question: How to compute synaptic weights Cij so that the network exhibits the best associative recall? The answer in [Gorodnichy-fpiv04-fa04]

Conluding remarks Techniques to be used for eye blink detection are defined. However, each case should be approached individually, depending on properties of the individual's eye lid movement, and computer power available. Time vs reliability tradeoff: The amount of processing should permit real-time execution (<100msec) so that blinks are not missed. On the other hand, less processing, worse change detection. Processing time needed for different video processing tasks (for 160x120 video on 1GHz Celeron laptop, in msec) * computes change based on the intensity in the neighbourhood of a pixel rather than on the intensity of the pixel itself . ** performed with an attractor-based neuron network of 24x24 neurons Acknowledgement: this technology is currently under test at the Midlands Rehabilitation Center, UK no processing 10 Colour-based detection 50 Motion-based detection 40 Viola-Jones face detection 140 Associative recall ** 100 maximal expected motion (pixels) 0 1 2 3 4 5 Second-order change detection 40 80 100 140 180 230 Non-linear change detection* 250