1. Improving Chinese handwriting Recognition by Fusing speech recognition Zhang Xi-Wen CSE, CUHK and HCI Lab., ISCAS 2005.4.12

2. Outline 1 Chinese handwriting recognition 2 Chinese speech recognition 3 Information fusion 4 Experimental results

3. 1.Handwriting Recognition Handwriting segmentation Character recognition

4. 1.1 Handwriting segmentation It is more difficult for Chinese handwriting segmentation

5. Character extraction using histogram A histogram of between-stroke gaps. The dimidiate threshold of the histogram is to extract lines of strokes. The dimidiate threshold of the histogram of a line of strokes is to extract characters.

6. Figure 1. Handwriting segmentation

7. Problems remained A Chinese character may be mis-segmented into many characters. Many Chinese characters may be mis- grouped as a character. The segmentation error will inevitably result in handwriting recognition errors.

8. 1.2 Character recognition –Isolated character recognizer from HW –Many candidates

9. Text recognized from the handwriting. Handwriting. The ground-truth text. Figure 2. Handwriting recognition

10. 2 Speech recognition Chinese speech. On-line, microphone. Continuous speech recognizer from MS.

11. Text recognized from the speech corresponding to the handwriting. The ground-truth text. Figure 3. Speech recognition

12. 3 Text fusion An optimization problem Dynamic Programming

13. 3.1 Principles The fused text should contain more semantic information. Construct a text with the least characters and the most semantic information.

14. 3.2 Four ways Text recognized from the handwriting. Text recognized from the speech corresponding to the handwriting. Figure 4. Texts to be fused

15. 3.3 Dynamic Programming A directed graph. Optimal paths.

16. Figure 5. A directed graph with N levels.

17. (c) The optimal fused text corresponding to the optimal path. (d) The ground-truth text. Figure 6. Text fusion using DP. (b) Text recognized from the speech corresponding to the handwriting. (a) Text recognized from the handwriting.

18. 3.4 A language model Lexicon Syntax Semantic

19. Lexicon

21. 4 Experimental results

24. Thank you very much for your criticism, comments and suggestions! Email: xwzhang@cse.cuhk.edu.hkxwzhang@cse.cuhk.edu.hk Tel: 3163-4260