1. Sign Classification Boosted Cascade of Classifiers using University of Southern California Thang Dinh thang.dinh@usc.edu Eunyoung Kim eunyoung.kim@usc.edu Li Zhang li.zhang@usc.edu Yuping Lin yuping.lin@usc.edu Computer Vision group:

2. Jan 2007 2. Problem statement Content 1. Sign classification: Applications and Challenges 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results

3. Jan 2007 Sign classification: Applications & Challenges Applications Sign language …….. Virtual Reality Robot Control Sign Classification

4. Jan 2007 Sign classification: Applications & Challenges Challenges 1. Detection Rate 2. Computation Time An interpreting system that interprets wrongly, a robot that always misunderstands commands… can not be employed. Robots that need 30s to understand a command telling them to do something immediately, Systems that need 30s to interpret each word of speakers… will not be employed either. need a Fast and Robust system

5. Jan 2007 2. Problem statement Content 1. Sign classification: Applications and Challenges 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results

6. Jan 2007 Problem statement Human Gesture: Dynamic Gesture: requires motion of hand, body… Static Gesture: static pose of hand, body… Finger Spelling (ASL System)

7. Jan 2007 Problem statement Implemented system: Challenges: Detection rate, Computation time Many similar signs ….

8. Jan 2007 2. Problem statement Content 1. Sign classification: Applications and Challenges 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results

9. Jan 2007 Why Boosted Cascade of Classifiers Cascade of boosted classifiers with Haar wavelet features (Viola and Jones) is currently ‘The state of the art’ in face detection. Cascade was brought to field of hand detection by Eng-Jon with impressive results.

10. Jan 2007 Why Boosted Cascade of Classifiers Face detection system of Viola and Jones is: 15 times faster than Rowley’s (double layer Neural Network) 600 times faster than Schneiderman-Kanade’s (Statistics) Cascade of boosted classifiers seems to be a good approach for the Sign classification problem.

11. Jan 2007 2. Problem statement Content 1. Sign classification: Applications and Challenges 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results

12. Jan 2007 Haar wavelet features Haar wavelet features (Papageorgiou ) Feature set was enlarged by Lien Hart to 45º rotated. We proposed Double-L for Sign detection

13. Jan 2007 Integral Image – Fast calculation of Haar features There are millions of features that need processing and each feature itself needs time to be calculated makes the training process time-consuming Viola and Jones proposed Integral Image in order to reduce the feature calculation complexity, which results in less computation time Definition of intergral image at point (x, y) Sum(D) = 1 + 4 – (2 + 3) Then:

14. Jan 2007 Adaboost Proposed by Freund and Schapire Combine many ‘weak’ hypothesis to form a ‘strong’ one ‘weak’ classifier h t only need to be better than chance

15. Jan 2007 Adaboost with Haar wavelet features Advantages: Adaboost adjusts adaptively the errors of the weak hypotheses Require large amount of training samples Weak Hypothesis learnt from this algorithm can run very fast because of simple calculation of Haar feature, which speeds up the whole system Disadvantages: The training process is rather time-consuming because the algorithm needs to check through millions of features extracted from thousands of samples.

16. Jan 2007 Cascade of classifiers Cascade of boosted classifiers is a tree of classifiers where classifier lying at each stage is better than the last Only those input patterns having passed through all the layers are considered objects Simple backgrounds can be easily rejected by one-feature classifier

17. Jan 2007 Sign Classifiers Each Sign Detectors D i is a cascade of boosted classifier evaluating an input image to give out value v i which is then compared to the threshold of it 24 Sign Detectors are combined to form a Sign Classifier C = sign i | D i = sign i and g(D i )=max{g(D j )|j=1, 2,… 24} Where g(D i ) = |vi –  i |

18. Jan 2007 Sign Classifiers

19. Jan 2007 2. Problem statement Content 1. Sign classification: Applications and Challenges 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results

20. Jan 2007 Experimental Results Samples Collection Collect raw image Find the ‘key’ point (which make it different from others) Cut and Align the image (base on ‘key’ point) Image B - Raw Image B - Focus Image B - Aligned Aligned images are finally made greyscale

21. Jan 2007 Experimental Results Training Process First stage: Positive: Thousands of aligned sign images Negative: Background only – Buildings, paintings, trees, other parts of human body… (also non-sign) Second stage: Positive: also thousands of aligned sign images Negative: aligned images of other signs Fast eliminate simple background Distinguish each sign from others

22. Jan 2007 Experimental Results Results 1. Detectors We have trained 24 detectors for 24 signs Average number of stages: 14 Total features: approx 123 Detection rate: 90% - 100% Test Images were divided into 2 groups Simple background Complex background DR = 100% FA = 0% DR = … FA = …

23. Jan 2007 Experimental Results Results 1. Detectors (cont) Hit rate diagram

24. Jan 2007 Experimental Results Results 1. Detectors (cont) False alarmHigh FA rate due to similarity between signs

25. Jan 2007 Experimental Results Results 2. Classifiers Test Images: 600 Correct rate: 83%

26. Jan 2007 Thank you