1. 1 Incremental Detection of Text on Road Signs from Video Wen Wu Joint work with Xilin Chen and Jie Yang

2. 2 Acquire Text, Process Text Corpus Languag e (Text) Languag e (Text) Web Visual Speech NLP Translation IR/IE Multimedia Speech

3. 3 Text helps to understand images

4. 4 Why interested in text on signs? Signs are everywhere in our daily life, such as shop names, billboard, street names, etc; Like other information device, road signs are placed to convey information to human for different purposes; Text could be the most flexible way to express dynamic information. Why not make computer to understand those text and further assist human?

5. 5 Too many signs cause problems

6. 6 It happened in Pittsburgh too!

7. 7 Task Automatically detect text on road signs from video.

8. 8 Related work

9. 9 What makes us to detect sign?

10. 10 What do you think?

11. 11 Vertical plane property of signs

12. 12 Divide-and-Conquer Strategy Decompose the original task into two sub-tasks, that is, localization of road signs and detection of text; Propose algorithms for two sub-tasks respectively, integrate them by mapping corresponding feature points; Use features from not only individual 2D images but also temporal dependency between them.

13. 13 Incremental Detection Framework

14. 14 Why incremental? Computation requirement –Detection is a computation-expensive step; –In contrast, mapping correspondence points is a cheap step; Video resolution –Detection requires low resolution –OCR requires high resolution LocalizeDetectRecognize Time

15. 15 System Implementation Prototype Built on a PC with Intel Pentium 4 CPU @1.8 GHz and 1GB memory, Windows XP; Data: 1) Captured by a DV camera mounted on a minivan. 2) Video frame size is 640*480. 3) The database included about 3 hours of videos, captured in different conditions, i.e., in the morning, afternoon, and dusk.

16. 16 A Demo Demo

17. 17 Sequences of the Demo

18. 18 Incremental vs. Non-incremental Another demo

19. 19 Summary of Evaluation 22 video sequences with different driving situations; Vehicle ’ s speed varies from 20 to 55 MPH Testing data contain ~90 road signs and > 300 words. # of signsHit rateFalse hits 9292.4%17.9% Hit rateFalse hitsSpeed Non-Incre-80.2%85.6%2-6 fps Incre-88.9%9.2%8-16fps Table 1. Sign localization performanceTable 2. Text detection performance

20. 20 Contributions Proposed a unified framework for automatically detecting text on road signs from video based on the natural characteristics of the task; Exploited features for text detection not only from individual 2D images but also from temporal dependency in video; Made connection between understanding visual information and understanding language (text).

21. 21 Conclusions & Future Work Automatic detection of text on road signs could be very useful in various applications; Experiments have shown that the new framework could significantly improves robustness and efficiency of any existing text detection algorithm; Future work: Apply various language methods to detected texts in video, e.g., translation, IR, etc.

22. 22 Question ? Thank You