Wrapping Snakes For Improved Lip Segmentation Matthew Ramage Dr Euan Lindsay (Supervisor) Department of Mechanical Engineering

Presentation Overview Brief overview of Visual Speech Recognition Requirements of a good Lip Segmenter One approach: Traditional Snakes An improved approach: “Wrapping Snakes”

What is Visual Speech Recognition? Uses video footage to identify spoken words Does not use any audio information Audio-only speech recognition accuracy falls dramatically as audio signal degraded Basic Principle: Mouth shape is based on the underlying structure of the word being spoken

What makes a good Lip Segmenter? Accurately represents lip shape Robust against poor initialisation Can handle multiple enclosed regions Reliably handles noisy images Can handle weak lip features near strong noise Quickly finds the lip shape

One Approach: Traditional Snakes A form of Active Contour Model Series of connected points Internal forces: Tension Rigidity External forces: Constraints Image Forces: Based on image features Using snakes to track the outer lip boundary (Kass et al. 1988) Kass, M, Witkin, A & Terzopoulos, D, 1988, 'Snakes: Active contour models', International Journal of Computer Vision, vol. 1, no. 4, pp

Traditional Snakes

Problems with Traditional Snakes Not robust against poor initialisation Cannot handle multiple enclosed regions Cannot reliably handle noisy images Cannot handle weak target features with strong noise

Multiple Enclosed Regions (Traditional Snake)

Weak Features & Strong Noise (Traditional Snake)

The Problem? Traditional Snakes don’t continue along features

The Problem? Traditional Snakes don’t continue along features

The Solution: A Wrapping Force A substitute for the Image Force Based on the Image Force Modified by the snake’s shape and position When a snake curves away from a feature, apply a force along the feature Pushes snake along the feature

Calculating the Wrapping Force Component of Image Force in the direction of the Snake’s Normal

Wrapping Behaviour

Cutting the Snake Cut the snake when separate sections come into contact Benefits: Distinguishes between multiple fully enclosed regions Ignores areas of noise near the lips Allows the lip shape to be accurately found Increased robustness of lip segmentation

Pinching Forces A subset of points a made “Pinch Points” Non-adjacent pinch points attract when close enough Pulls sections of the snake together when close enough Helps with cutting

Multiple Enclosed Regions (Wrapping Snake)

Weak Features & Strong Noise (Wrapping Snake)

Properties of Wrapping Snakes Accurately represents lip shape Robust against poor initialisation Can handle multiple enclosed regions Reliably handles noisy images Can handle weak lip features near strong noise Quickly finds the lip shape

Wrapping Snakes For Improved Lip Segmentation Matthew Ramage Dr Euan Lindsay (Supervisor) Department of Mechanical Engineering

Handles Very Poor Initial Position (Wrapping Snake)

Multiple Enclosed Regions (Traditional Snake)

Multiple Enclosed Regions (Wrapping Snake)