Advanced Crosswalk Detection for the Bionic Eyeglass Mihály Radványi Balázs Varga Kristóf Karacs Pázmány Péter Catholic University Berkeley, 2010.

Summary Bionic Camera/Eyeglass The task Difficulties Algorithmic description Results Further plans

Bionic Camera/Eyeglass Visual info audio MULTIMODAL Three situations home banknote, color recognition, etc. office pictograms, displays, etc. street escalator’s direction route number-, crosswalk detection,etc. Autonomous crash avoidance

The task Crosswalk recognition / detection Based on road marks dark – bright alternating parallel Mobile device navigation  fast decision Low consumption ( ~1W) High computing power (TOps/s) visual microprocessor

Difficulties shadows light conditions traffic missing marks others tricky disturbing light conditions traffic missing marks others categorize!

Different approaches Matlab simulation On-chip Image Processing and MatCNN toolbox 2nd generation Mean shift color segmentation On-chip Bi-i visual processor 128 x 128 pixel Still image, video 10 fps (1st generation)

Manual design - flow Goal: clearest zebra lines Adaptive threshold Supposed zebra lines AND Contrast mask Color mask CNN templates RECALL Input image Mean shift

Asphalt detection - introduction foreground-background segmentation foreground (ROI): asphalt + stripes mean shift segmentation Similar to K-means arbitrary number of clusters! Iterative, nonparametric YCbCr color space Density gradient map with modes ( ) Input image (Y – Cb – Cr) Calculate position of a mode* (Iterative scheme on a Probability Density Function using a moving Gaussian window) Put pix. with similar color into same class Select i data points No Are all clustered? Pixels Yes Biggest class results asphalt *dense region of the feature space

Asphalt detection - segmentation Steps of segmentation Preprocessing band-pass color filtering Foreground-background segmentation mean shift Post processing morphological operations eliminates false positive/false negative pixels Input Result of preprocessing Classes of mean shift Result of post processing Masked foreground

Manual design - again Goal: clearest zebra lines Adaptive threshold Supposed zebra lines AND Contrast mask Color mask CNN templates RECALL Input image Mean shift

Localization of huge patches Manual design – flow No.2 Space-variant LOGDIF Localization of huge patches EDGE Decision Scalar output Supposed zebra lines A h 600 400 200 h’ A’ where

Results - simulation

Crosswalk non detected Results Enlarged dataset 89.2% performed well Crosswalk detected Crosswalk non detected Crosswalk 33 7 No Crosswalk 2 41 False positive (2.4%)

The prototype Nokia N95 + Bi-i + WiFi Eye-Ris

Further plans Tracking Online test with blind users(done…) Correlations between frames Online test with blind users(done…) Background estimation through image fusion

Thanks for your attention!