1. Bieber et al., NJIT ©2005 - Slide 1 Lightweight Integration and Recommendation of Documents and Services ------- Digital Library Service Integration, IntegraL and IntLib Projects Michael Bieber *, Il Im *, Vincent Oria ** Richard Sweeney ***, Yi-Fang Wu * * Information Systems Department *** Robert Van Houten Library ** Computer Science Department College of Computing Sciences New Jersey Institute of Technology http://is.njit.edu/integral April 2005

2. Bieber et al., NJIT ©2005 - Slide 2 Outline Motivation Illustrations Structural Relationships 3 Types of Integration Personalizing Links Federated Metasearch Recommendations Contributions and Vision Call for Collaboration Project Details

3. Bieber et al., NJIT ©2005 - Slide 3 Challenges for Library Users Need to know what resources to use before they can access them Finding related information outside current system Need to leave current page to do related tasks Why? Library resources aren’t integrated well ==> Project Goal: –Bring relevant resources directly to the user Library resources: databases (e.g., EBSCOhost, ACM Digital Library), external digital libraries, on-line catalog, special collections, library services (e.g., interlibrary loan)...

4. Bieber et al., NJIT ©2005 - Slide 4 Integration through Linking automatically generate link anchors on elements recognized based on: –structural relationships –lexical relationships automatically generate links –to related information –to relevant services ==> lightweight integration of –documents containing links and –documents/services the links point to

5. Bieber et al., NJIT ©2005 - Slide 5 Prototype Services for a launch-date element: - search by launch date - search by month and year - search by year

6. Bieber et al., NJIT ©2005 - Slide 6 Prototype Services for a document element: - open - summarize in 3 sentences

7. Bieber et al., NJIT ©2005 - Slide 7 Mock-up for a library database Services from multiple systems (customized to user tasks/preferences)

8. Bieber et al., NJIT ©2005 - Slide 8 Benefits of Integration for a system (collection/service) Users: direct access to related systems –enlarges a system’s feature set Links leads users to a system –systems gain wider use Users become aware of other systems –systems gain wider awareness Direct access to a system’s features –streamlined access (bypassing menus)

9. Bieber et al., NJIT ©2005 - Slide 9 structural elements and links lexical elements and links Two Types of Links: (1) structural based on element type * title, author, source (2) lexical (found in a glossary)

10. Bieber et al., NJIT ©2005 - Slide 10 Structural Relationships Links generated based on application structure, not search or lexical analysis –You cannot do a search on the display text “$127,322.12” to find related information… –But you can find relationships for the element Sales[2002] $85,101.99$127,322.12 2002 Expenses2002 Sales

11. Bieber et al., NJIT ©2005 - Slide 11 Outline Motivation Illustrations Structural Relationships 3 Types of Integration Personalizing Links Federated Metasearch Recommendations Contributions and Vision Call for Collaboration Project Details

12. Bieber et al., NJIT ©2005 - Slide 12 Three Types of Integration: (1) for documents to receive anchors and links (2) to provide services (which become links) (3) to provide glossaries for content analysis Require a document schema mapper to recognize structural elements: -wrapper -fixed template -XML markup -etc.

13. Bieber et al., NJIT ©2005 - Slide 13 Three Types of Integration: (1) for documents to receive anchors and links (2) to provide services (which become links) (3) to provide glossaries for content analysis Linking Rules represent * every service * that a system can provide * for each kind of element.

14. Bieber et al., NJIT ©2005 - Slide 14 Three Types of Integration: (1) for documents to receive anchors and links (2) to provide services (which become links) (3) to provide glossaries for content analysis Linking Rules represent * every service * that a system can provide * for each kind of element. Example ==>

15. Bieber et al., NJIT ©2005 - Slide 15 Example Linking Rule from the AskNSDL system –a) element type (“concept”) –b) link display label (“Ask an expert about this”) –c) relationship metadata –d) destination collection or service (“Ask NSDL”) –e) the exact command to send to the destination system (logs the user into AskNSDL, opens question template, fills in the element instance (i.e., “physics teaching”) as the subject, and places the cursor in the question area) –f) any relevant conditions for including this relationship

16. Bieber et al., NJIT ©2005 - Slide 16 Three Types of Integration: (1) for documents to receive anchors and links (2) to provide services (which become links) (3) to provide glossaries for content analysis Lexical analysis by: NJIT Noun Phrase Extractor NJIT Ontology Developer

17. Bieber et al., NJIT ©2005 - Slide 17 Each system is integrated independently: (1) Schema mappers for individual systems (2) Linking rules are plugged in” independently for each service (3) Glossaries and thesauri can be independent of other systems

18. Bieber et al., NJIT ©2005 - Slide 18 Outline Motivation Illustrations Structural Relationships 3 Types of Integration Personalizing Links Federated Metasearch Recommendations Contributions and Vision Call for Collaboration Project Details

19. Bieber et al., NJIT ©2005 - Slide 19 Personalizing the Links Customize the list of links according to: Collaborative Filtering –Matching user’s “click stream” to other users’ time spent at each destination asking users to rate links user task information

20. Bieber et al., NJIT ©2005 - Slide 20 Federated Metasearch Searches, merges & ranks

21. Bieber et al., NJIT ©2005 - Slide 21 Federated Metasearch Searches, merges & ranks Clusters results by concept

22. Bieber et al., NJIT ©2005 - Slide 22 Federated Metasearch: Clustering by Concept concept hierarchy search results are clustered by concept

23. Bieber et al., NJIT ©2005 - Slide 23 Outline Motivation Illustrations Structural Relationships 3 Types of Integration Personalizing Links Federated Metasearch Recommendations: General Recommendation Engine Contributions and Vision Call for Collaboration Project Details

24. Bieber et al., NJIT ©2005 - Slide 24 Integration With Partner Libraries Minimum integration More complex integration

25. Bieber et al., NJIT ©2005 - Slide 25

26. Bieber et al., NJIT ©2005 - Slide 26 General Recommendation Engine Research Goals Integrate three major recommendation technologies – Collaborative filtering (CF), Content-based (CB), and Knowledge-based (KB) recommendation Automatically identify users ’ current task (search mode) Study the impacts of the recommendations on information search

27. Bieber et al., NJIT ©2005 - Slide 27 Collaborative Filtering (CF) Recommendations based on similarities of people Traditional CF requires direct user inputs Clickstream-based CF (CCF) does not require direct user inputs Works well for preference goods Does not work so well for information-intensive items

28. Bieber et al., NJIT ©2005 - Slide 28 Content-based Filtering (CB) Recommendations based on similarities of contents –titles, authors, abstracts, or full texts Information retrieval (IR) techniques are used –e.g., tf.idf value Documents similar in content are recommended Demo: http://highlight.njit.edu/ais/http://highlight.njit.edu/ais/

29. Bieber et al., NJIT ©2005 - Slide 29 Knowledge-based Recommendation (KB) CF and CB lack a holistic view –why a document is relevant for a user KB recommends items based on a certain knowledge structure (ontology) KB requires knowledge engineering Goal: to build a automated (or semi- automated) ontology engine based on the ‘ Self- organizing tree ’ algorithm ( Khan and Luo, 2002)

30. Bieber et al., NJIT ©2005 - Slide 30 Automatic User Profile Extractor Records a user ’ s recent documents The user ’ s profile is represented by a set of keywords from those documents As the user visits more documents, his/her profile is updated

31. Bieber et al., NJIT ©2005 - Slide 31 Outline Motivation Illustrations Structural Relationships 3 Types of Integration Personalizing Links Federated Metasearch Recommendations Contributions and Vision Call for Collaboration Project Details

32. Bieber et al., NJIT ©2005 - Slide 32 Contributions straightforward, sustainable approach for integrating documents and services –Lightweight integration through linking combining structural links with content-based links next-generation collaborative filtering federated metasearch next-generation recommendations integrating traditional and digital libraries widespread dissemination

33. Bieber et al., NJIT ©2005 - Slide 33 Vision A nationwide virtual library to and from –your local library –other physical libraries –digital libraries incorporating –traditional library resources –digital library resources Bringing relevant resources directly to the user!

34. Bieber et al., NJIT ©2005 - Slide 34 Looking for Collaboration Additional document systems, digital library collections, services and glossaries to integrate Physical library partners Digital library partners Web services to integrate Other suggestions welcome!

35. Bieber et al., NJIT ©2005 - Slide 35 Additional Slides

36. Bieber et al., NJIT ©2005 - Slide 36 Outline Motivation Illustrations Structural Relationships 3 Types of Integration Personalizing Links Federated Metasearch Contributions and Vision Call for Collaboration Project Details

37. Bieber et al., NJIT ©2005 - Slide 37 Digital Library Service Integration NSF National Science Digital Library Award #DUE-0226075; 2002-2005 Tasks Develop Integration Infrastructure Integrate digital library collections and services Collaborative filtering Evaluation Partners NASA GSFC Library AskNSDL Earth Science Picture of the Day System Atmospheric Visualization Collection Metis Workflow (University of Colorado, Boulder) University of Arizona

38. Bieber et al., NJIT ©2005 - Slide 38 IntLib Institute of Museum and Library Services Award #LG-02-04-0002-04; 2004-2007 Tasks - to integrate: EBSCOhost Gale’s Discovery Collection ProQuest On-line Catalog Systems New Jersey Digital Highway The IntLib Project focuses on integrating the resources of public libraries primarily (and university libraries secondarily) with digital libraries. Additional Partner Newark Public Library

39. Bieber et al., NJIT ©2005 - Slide 39 IntegraL NSF National Science Digital Library Award #DUE-0434581; 2004-2007 Tasks - to integrate: ACM Digital Library Elsevier Science Direct (permission pending) NJIT Electronic Thesis collection JerseyClicks StartingPoint Digital Library for Earth Science Education (DLESE) Science@NASA NSDL Core Integration features an on-line bookstore The IntegraL project focuses on integrating specific resources of college libraries with those of the NSDL. Additional Partners Cumberland C.C. Ramapo College Olin College of Engineering

40. Bieber et al., NJIT ©2005 - Slide 40 General Recommendation Engine NSF National Science Digital Library Award; 2004-2007 Tasks - to integrate: Collaborative filtering recommendations Content-based recommendations Knowledge-based recommendations Partners Digital Library for Earth Science Education (DLESE) Eisenhower National Clearinghouse for Mathematics and Science Education Computer Vision Education Digital Library

41. Bieber et al., NJIT ©2005 - Slide 41 Which additional services would you suggest?

42. Bieber et al., NJIT ©2005 - Slide 42 Ph.D. Research Opportunities Integration through linking Customizing services to work in many domains Service chaining: creating new services by chaining existing ones –e.g., translate and summarize documents Collaborative Filtering for digital libraries Lexical analysis –Automatic summarization of returned documents –including maintaining multiple glossaries/thesauri Full Virtual Community support –analysis, tools, processes, evaluation

43. Bieber et al., NJIT ©2005 - Slide 43 User’s Web Browser AskNSDL Schema Mapper AVC Schema Mapper NSSDC Schema Mapper CI Search Service Schema Mapper Service Schema Mapper (i) AskNSDL AVC N’l Space Science Data Center NSDL CI Search Service Service (i) ME Link Mapping Engine ME Broker ME Desktop Metainformation Engine ME Lexical Analysis existing system or Web service uses Java, XML, XPath, etc. Internal Architecture