1. CADAL Digital Library Wu Jiang-Qin,Zhuang Yue-Ting Pan Yun-he College of Computer Science, Zhejiang University,China November 18,2006 November 18,2006 The 2nd International Conference on Universal Digital Library (ICUDL 2006) The 2nd International Conference on Universal Digital Library (ICUDL 2006)

2. Outline 1111 1111 4444 4444 2222 2222 5555 5555 Conclusion and Future Work Introduction Unified Paralleling Search Multimedia Analysis and Retrieval 6666 6666 3333 3333 Bilingual services Chinese Calligraphy Character Retrieval

3. Outline 1111 1111 4444 4444 2222 2222 5555 5555 Conclusion and Future Work Introduction Unified Paralleling Search Multimedia Analysis and Retrieval 6666 6666 3333 3333 Bilingual services Chinese Calligraphy Character Retrieval

4. CADAL  The China-Us Million Book Digital Library(CADAL) is an international cooperation program between China and the US.  The objective of CADAL project, is to create a free-to-read, searchable collection of one million book, available to everyone over the internet.  CADAL is the important part of Universal Digital Library(UDL), universal access to human knowledge.

5. The challenges and services (1)  the amount of the digital resources including digital books and multimedia for research and education can reach 100 terabyte ( The number of digital books is 1,023,425 by October of 2006,including previous Chinese ancient books, Chinese minguo books,Chinese Modern books, Chinese degree dissertation,English books,image,video etc..  active services of unified paralleling search for the different types of digital resources

6. The challenges and services (2)  image, video,3-D model and other types of media resources, various types of media resources are included in the CADAL resources.  the services of quickly retrieving and structurally browsing of multimedia documents including image, video

7. The challenges and services (3)  there are two kinds of language digital books. Chinese and English, in the CADAL resources.  the services of bilingual translation

8. The challenges and services (4)  traditional Chinese culture resources are important part of the CADAL resources.  the services related to Chinese traditional culture resources.

9. Outline 1111 1111 4444 4444 2222 2222 5555 5555 Conclusion and Future Work Introduction Unified Paralleling Search Multimedia Analysis and Retrieval 6666 6666 3333 3333 Bilingual services Chinese Calligraphy Character Retrieval

10. Background  TB volume of various types of digital resources, such as dissertation, ancient minguo book, modern book, minguo journal, English book, drawing, video and illustration are available in the CADAL, which is one of the distinct characteristic of CADAL. So CADAL presents a challenge for the technique of searching resources based on metadata.dissertationancientminguo book modern bookminguo journal

11. Metadata  Dublin core metadata is used to describe the million digital books in the CADAL project. Metadata corresponding to the other types of multimedia resources are used to describe them. Independent data map is designed for each kind of resource metadata.

12. Unified parallel searching  In order to meet the requirements of different users and improve the user’s interactive experience, the service for the different types of digital resources is provided for users’ convenient searching.

15. Outline 1111 1111 4444 4444 2222 2222 5555 5555 Conclusion and Future Work Introduction Unified Paralleling Search Multimedia Analysis and Retrieval 6666 6666 3333 3333 Bilingual services Chinese Calligraphy Character Retrieval

16. Background  As the digital library contains unstructured multimedia resources such as images, videos, audios etc besides digital books, effective and efficient analysis and retrieval of multimedia resources is a challenging problem in the CADAL digital library.  Here we examine the analysis and retrieval issues related to two primary kinds of multimedia, image and video.

17. Contents  Content-based Image Retrieval  Image retrieval by peer indexing  Image annotation  Image search engine  Video analysis system  Video Browser(structure and summary)  Metadata-based Video Retrieval

18. Content based image retrieval   Extracting visual features color feature ： color histogram, color moment, color coherence vector, color correlgram texture ： Tamura textural feature and co- occurrence textural feature   relevance feedback Make image retrieval coincide with user’s requirement

19. Content based image retrieval Positive example Negative example Query example Image searching Relevance feedback

20. Image retrieval by peer index  A new scheme for image indexing, Peer Index, is the method that describe images through semantically relevant peer images.  In particular, each image is associated with a two-level peer index, including global peer index: describing the “data characteristics” of this image personal peer indexes: describing the “user characteristics” of an individual user with respect to this specific image  Both types of peer index are learned interactively and incrementally from user feedback information.

21. Peerindex-based image retrieval Semantic query semantic relevance feedback

22. Image annotation  Automatic semantic annotation for images by machine learning and statistical modeling Classify the training images, and create a semantic skeleton for each class of the training image. Classify new image with Support Vector Machine automatically, and describe it using the semantic skeleton Select the key words for the image by statistical methods

23. images tige r annotation classify............ statistical learning classify segment Images annotate Visual similar Image blobs Semantic skeleton Image annotation

24. Text based image retrieval Query text ： bonsai

25. Image search engine  We implemented an image search engine, Octopus, which provides Peer Index and relevance feedback to avoid the gap between the semantics and low-level features, according to the intuitive and simple idea that the semantic concept is hidden in each image and the semantic concept appears apparently in the relation between the image and the other images.

26. Integrating into CADAL DL images Books scanner Image Manager store imagesmetadata feature CADAL image repository CADAL portal Image retrieval system user images Other images WWW Browse Retrieve Relevance feedback

27. The image retrieval interface

28. Our target for video   Analyze multimodal information, such as the visual, the audial, motion and caption to generate structural information and video summary   Support video browsing and video retrieval based on metadata and structural information efficiently

29. Main idea   Nonlinear browsing ： Generate structural indexing such as key frame, shot and shot group from the original video stream   Content compression ： A nalyze time sequence in video stream, eliminate redundant data, and generate the summary and the highlight scene for the original video.

30. System   Video Fusion Analysis System ( VideoFAS)   VideoBrowser Video data Video fusion analysis system video repository Metadata database Feature database CADAL video repository Video browser user CADAL portal

31. VideoFAS － system interface Original Video Video shot Similar Video shots are Clustered together

32.   Basic operation Importing and Saving Appending Separating the video stream into video and audio data transcoding and compressing VideoFAS － system functions

33.   Feature Extraction Visual feature   color ： color histogram, color moment, color coherence vector, color correlgram   Texture ： Tamura textural feature and co- occurrence textural feature   shape ： contour feature Audial feature   temporal feature ： zero-crossing rate   Frequency feature ： Mel coefficient 、 tone and sub-band statistical feature Target Feature   Integrate OpenCV face detection module into the system Extract the face features VideoFAS － system functions

34.   Video structuring shot detection   Cut shot detection   Transition shot detection key frame extraction Similar shot grouping   group the shots based on Support Vector Machine VideoFAS － system functions

35. Original Video Shot Sequence Video Shot Clustering Video Shot Cluster A Video Shot Cluster B Video Shot Cluster C Video Shot Cluster D Video Shot Cluster E

36.   Video summarization Summarize by Mining Non-Trivial Repeating Patterns   Extract frequent and non-trivial shot sequence to generate video summary VideoFAS － system functions Original Video Shot Sequence A B C D A B C D A C D E A B C D

37.   Metadata annotation Annotate Video clip with metadata conform to Dublin Core Standard Save the metadata and the video structural information in database VideoFAS － system functions

38. VideoBrowser － framework WWW Video repository Original video data Video catalog Video summary user Content service

39. VideoBrowser － system interface

40. metadata Video structural information media player

41. video data annotation structuring Online storage Disk array switcher Archive server taper （ offline storage ） Web server firewall Internet Retrieval service Web Movies System architecture summarization

42. Outline 1111 1111 4444 4444 2222 2222 5555 5555 Conclusion and Future Work Introduction Unified Paralleling Search Multimedia Analysis and Retrieval 6666 6666 3333 3333 Bilingual services Chinese Calligraphy Character Retrieval

43. Background  As there are both English and Chinese books in CADAL, bilingual services are required for users to access resources in any language.

44. Services  Some technologies and prototypes have been developed by north technical center on how to carry out the multi-layered bilingual machine translation in English and Chinese books, such as the metadata translation between English and Chinese the accurate translation of proper nouns such as names for unique individuals, events,or places the selective translation in a full-text context the translation of Old Chinese text the distributed translation service technique.

45. Services  An online translation service is integrated into the CADAL digital library. Users can be directly conducted semantic-based multi-linguistics retrieval of required information in our CADAL digital library. The translation of contents of a page on line. The translation of metadata of a digital book.

46. Bilingual Search

47. The translation of contents of a page

49. Outline 1111 1111 4444 4444 2222 2222 5555 5555 Conclusion and Future Work Introduction Unified Paralleling Search Multimedia Analysis and Retrieval 6666 6666 3333 3333 Bilingual services Chinese Calligraphy Character Retrieval

50. Background  Since most people are interested in the art of the beautiful styles of calligraphy character rather than the meaning of the character, the service of Chinese calligraphy character retrieval is provided in the CADAL digital library, treating them just as they are images without recognizing them like OCR does.

51. Calligraphy art still alive in:

52. Key issues  Feature extraction: character complexity, stroke density and shape, the three kinds of features of the calligraphy character are proposed  similarity matching cost: retrieve relevant images according to it.

53. Contents  Chinese Calligraphy Page Segmentation  Features Extraction  Character Image Retrieval

54. Chinese Calligraphy Page Segmentation  The page image are binarized with characters in black and the background in white.  Cut the page into columns according to the vertical projecting histogram, and columns continued to be cut into individual characters.  All the characters are normalized in order to keep scale invariant Contour information,

55. Page segmentation

56. Features Extraction  shape  character complexity

57. shape representation  Calligraphic character’s shape is represented by its contour points.  The polar coordinates is used to describe directional relationship of points instead of the Cartesian coordinates.  For direction, we use 8 bins in equal degree size to divide the whole space into 8 directions.  For radius, we use 4 bins  For each point of a given point set composed of sampling points, its approximate shape context is described by its relationship with the remaining points in weighted bins.

58. shape representation Contour point

59. Calligraphy Character Complexity  We use Calligraphy Character Complexity as a filter at the beginning to discard the calligraphy character that has no possibility to be similar to the query. L be the number of sampled contour points from the query and L i be the number of sampled contour points from candidate image.  is the threshold obtained by experience.

60. Character Image Retrieval  Compute the values of the character complexity of the calligraphy character.  Normalize the scale size of the query and sample its contour points.  Filter the candidate images by character complexity  Extract the shape feature and employ the shape matching method introduced in [6] to compute the matching cost for every remaining candidate image and the query.  Rank the results according to the matching cost, and return.

61. The calligraphy character retrieval

62. interface of browsing the original works

63. Outline 1111 1111 4444 4444 2222 2222 5555 5555 Conclusion and Future Work Introduction Unified Paralleling Search Multimedia Analysis and Retrieval 6666 6666 3333 3333 Bilingual services Chinese Calligraphy Character Retrieval

64. All the services have been accessed by the users from over 70 countries 280.000 times per day.

65. Conclusion and Future Work  With the increase of the number of the users and the amount of the resources. future work with CADAL digital library will proceed in several directions: Improving the performance of the current services, to be more complete and be more stable; Continuing exploring the application of multimedia in Digital Library.

66. Thanks ！ Welcome Visiting CADAL Digital Library (WWW.CADAL.ZJU.EDU.CN ) Email: wujq@cs.zju.edu.cn yzhuang@zju.edu.cn wujq@cs.zju.edu.cn