Building Digital Libraries Made Easy: Toward Open Digital Libraries ICADL 2002 – Singapore – Dec Edward A. Fox (with Hussein Suleman, Ming Luo) CS DLRL Internet TIC NDLTD CITIDEL NSDL … Virginia Tech, Blacksburg, VA, USA

Acknowledgements (Selected) Sponsors: ACM, Adobe, DLF, IBM, Mellon Foundation, Microsoft, NSF (Grants CDA ; DUE , , ; IIS , , , and ), OCLC, SOLINET, UNESCO, US Dept. Ed. (FIPSE), VTLS, … Faculty/Staff (now): Boots Cassel, Su-Shing Chen, Debra Dudley, Jeremy Frumkin, Joe Futrelle, Lee Giles, Martin Halbert, Rex Hartson, John Impagliazzo, Deborah Knox, JAN Lee, Kurt Maly, Gail McMillan, Eric Morgan, Manuel Perez, Muhammad Zubair, … Students: Fernando Das Neves, Marcos Goncalves, Rohit Kelapure, Aaron Krowne, Paul Mather, Ryan Richardson, Priya Shivakumar, Wensi Xi, Liang Xu, Baoping Zhang, …

Outline Overview, Problem Experience: Case Study Projects Open Archives Initiative Hussein Suleman Dissertation DL in a Box, OCKHAM Summary and Conclusion

Overview We address the problem of how to develop DLs; build on experience in building many DLs; strive for simplicity as per OCKHAM initiative; build upon the Open Archives Initiative; demonstrate our approach in diverse situations; and invite all to use DL-in-a-box and help build Open Digital Libraries.

Problem Why do DL developers continue to “reinvent the wheel”? The top 10 reasons are: 1.The library budget won’t allow purchase of a commercial DL system. 2.Unless the development effort is local, there won’t be any control. 3.DLs are extensions of DBMSs, so they are simple applications to develop. 4.Since DLs operate on the Web, one must adopt the newest W3C proposal.

Problem – cont’d 5.Since technology moves so quickly, it is essential to follow the latest fad. 6.CS students always develop from scratch. 7.This team knows it can do it better. 8.This system must have more capabilities than any other system. 9.This DL has to be more flexible and extensible. 10.This is the right system architecture – at last!

Outline Overview, Problem Experience: Case Study Projects Open Archives Initiative Hussein Suleman Dissertation DL in a Box, OCKHAM Summary and Conclusion

Experience: Case Study Projects AmericanSouth.org NDLTD CSTC JERIC CITIDEL NSDL Digital Library in a Box

AmericanSouth.org Domain: culture and history of the southern region of America (USA) Genre: diverse distributed collections at a dozen universities Submission & Collection: local sites  Emory University (for SOLINET)

Networked Digital Library of Theses and Dissertations (NDLTD) Domain: graduate education and research Genre: electronic theses and dissertations (ETDs) Submission & Collection: local sites 

Computer Science Teaching Center (CSTC) Domain: teaching computer science Genre: courseware Submission & Collection:

CS Teaching Center (CSTC): Lessons Learned Instead of building large, expensive multimedia packages, that become obsolete and are difficult to re-use, concentrate on small knowledge units. Learners benefit from having well-crafted modules that have been reviewed and tested. Use digital libraries to build a powerful base of support for learners, upon which a variety of courses, self-study tutorials & reference resources can be built.

Browsing (2)

ACM Journal of Educational Resources in Computing (JERIC) Domain: teaching computer science Genre: courseware, scholarly articles Submission & Collection: CSTC, ACM Digital Library

JERIC JER iCJournal of Educational Resources in Computing Accessible from and and ACM and SIGCSE support Refereed and interactive Part of ACM Digital Library

Computing and Information Technology Interactive Digital Educational Library (CITIDEL) Domain: computing / information technology Genre: one-stop-shopping for teachers & learners: courseware (CSTC, JERIC), leading DLs (ACM, IEEE-CS, DB&LP, CiteSeer), PlanetMath.org, technical reports, … Submission & Collection: sub/partner collections 

CITIDEL Team An NSDL Collection Track project Led by Virginia Tech, with co-PIs: Fox (director, DL systems) Lee (history) Perez (user interface, Spanish support) Partners College of New Jersey (Knox) Hofstra (Impagliazzo) Villanova (Cassel) Penn State (Giles)

Summary of Spring 2001 Survey of CITIDEL-related Collections and their Sizes Size of Collection 1-5 items items items items Number of Collections Identified

Multi-dimensional Categorization

CITIDEL Collection Sources metadata JERIC fulltext Experts’ finding aids IEEE-CS … include CSTCResearch Index ACM NEC’s data processed w. R.I. SIGCSE proceedings ACM DL include Borner’s info viz software repository NCSTRL

CITIDEL Collection Building thru aided by after using or thru using Submitting VIADUCT GetSmart Searching, Browsing Classifying Nominating Crawling Crawlifier thru Composing include after Creating include after

Overview of CITIDEL architecture

Distributed repository structure

Digital library architecture for local and interoperable CITIDEL services

National Science Digital Library (NSDL) Domain: undergraduate and K-12 education, etc. Genre: educational resources Submission & Collection: sites of 90 projects 

NSDL Information Architecture Developed by the Technical Infrastructure Workgroup referenced items & collections referenced items & collections Special Databases NSDL Services NSDL Services Other NSDL Services CI Services annotation CI Services discussion CI Services personalization CI Services authentication CI Services browsing Core Services: information retrieval Core Collection- Building Services harvesting Core Collection- Building Services protocols Core Services: metadata gathering Portals & Clients Portals & Clients Portals & Clients Usage Enhancement Collection Building User Interfaces NSDL Collections NSDL Collections NSDL Collections Core NSDL “Bus”

Digital Library in a Box Domain: helping DL projects Genre: any domain, but especially those involved in NSDL (since funded in part is through NSDL – with U. FL, NCSA) Software and Documentation:

Outline Overview, Problem Experience: Case Study Projects Open Archives Initiative Hussein Suleman Dissertation DL in a Box, OCKHAM Summary and Conclusion

Open Archives Initiative OAI

Discovery Current Awareness Preservation Service Providers Data Providers Metadata harvesting The World According to OAI

Technical Umbrella for Practical Interoperability… Reference Libraries Publishers E-Print Archives …that can be exploited by different communities Museums

Tiered Model of Interoperability Mediator services Metadata harvesting Document models

OAI – Black Box Perspective OA 1OA 2OA 4OA 3OA 5OA 6OA 7 BrowseSummarizeSearchVisualize DO Services: Docs: Metadata:

Aggregation through OAI Harvesting ArchiveLite SitesNCSTRLEprints IEEE-CS, ACM, … Own: History, ResearchIndex, CSTC, … CITIDELActive

Protocol for Metadata Harvesting Service Requests Identify ListMetadataFormats ListSets GetRecord ListIdentifiers ListRecords Metadata Multiplicity Date/Time Ranges Sets (with semantics depending on local data providers) Resumption Tokens

NDLTD OAI Example

Outline Overview, Problem Experience: Case Study Projects Open Archives Initiative Hussein Suleman Dissertation DL in a Box, OCKHAM Summary and Conclusion

Open Digital Library (ODL) Hypothesis (Hussein Suleman) Can we leverage the successful model of the OAI Protocol for Metadata Harvesting to alleviate our architectural problems ? Maybe … if Digital Libraries can be modeled as networks of extended Open Archives, where each extended Open Archive is a source of data and/or a provider of services.

Example Architecture (NDLTD) Humboldt Duisburg MIT Filter MIT Browse Union Catalog SearchRecent User Interface OAI/ODL archive OAI/ODL protocol legend Virginia Tech PhysNet CalTech Dresden

ODL Demonstration - FrontPage

ODL Demonstration - Search

ODL Demonstration - Browse

Hussein Suleman’s Thesis Summary Open Digital Libraries (DLs) Open Archives Initiative (OAI) Protocol for Metadata Harvesting (PMH) Extending OAI-PMH provides the glue for building componentized DLs. Lightweight protocols connect the components to support modular systems with good efficiency.

Research in a Nutshell We build extensible modular systems with customizable services. This supports interoperability and allows distributed development. This is in use in AmericanSouth.org, … Components include search, browse, annotate, editorial support, union, filter, whats-new, submit, rate, recommend, …

Program Document Document Document Program Program Image Image Image Video Video Video usersdigital objects ?

Program Document Document Document Program Program Image Image Image Video Video Video componentized digital library ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Program Document Document Document Program Program Image Image Image Video Video Video open digital library OA PMH XPMH

ODL Component Requirements Search Retrieve a list of items Index new items Annotate Add annotation to item Retrieve a list of annotations for an item

Open Digital Library Components Running now XML-File (data provider from file system) Union, search, browse, recent, filter E-journal/review, Submit, Edit, Annotation Class projects High performance multilingual search Recommender, Rating; Mirroring (see JCDL’02) Working with NCSA: from DB, unstructured text Others discussed Classification/categorization DL-Viz interconnection (VIDI – Jun Wang ETD)

Harvest from data providers DBUnion Archive Merger Component DBBrowse Browse Engine IRDB-1 Search Engine As Metadata Search Service Provider As Metadata Browse Service Provider XML File Coll. & Data Provider 1 XML File Coll. & Data Provider 2 XML File Coll. & Data Provider 3 Open Digital Library: Extended What’s New Engine As What’s New Service Provider OAI-PMH Data Provider Submit Archive OAIB (NCSA: from RDBMS) Filter Recommend Rate Engine Annotation Engine IRDB-2 Search Engine As Annotation Search Service Provider As Recommend & Rate Service Provider

Program Document Document ETD Program ETD Image Image ETD Video Video ETD-4 Digital Library for the Networked Digital Library of Theses and Dissertations ( Search Filter Union Recent Browse PMH ODLRecent ODLBrowse ODLUnion ODLSearch ODLUnion PMH USER INTERFACE Students and researchers ETD collections Example Open Digital Library

Digital Library for the Computer Science Teaching Center (

CSTC User Interface

Open Digital Library Component Extended OPEN ARCHIVE OPEN ARCHIVE

Layer 1 : OAI PMH Protocol for Metadata Harvesting Transfer stream of metadata from one archive or component to another Service Requests Identify, ListSets, ListMetadataFormats GetRecord, ListIdentifiers, ListRecords

Layer 2 : Extended OAI-PMH OAI-PMH + extensions for general-purpose inter-component communication Added in generic containers in every response for additional information Added “PutRecord” to submit a record Increased granularity to support times as well as dates (same as OAI-PMH v2.0) Ignored DC requirement

Layer 3 : ODL Protocols Specialized protocol semantics for different components, e.g.: Search component uses ODLSearch protocol ListRecords and ListIdentifiers embed query terms in “set” parameter Annotation component uses ODLAnnotate protocol ListRecords and ListIdentifiers specify the item for which annotations are requested in the “set” parameter PutRecord adds an annotation to an item

Performance Optimizations Caching of responses Persistent CGI mechanisms FastCGI SpeedyCGI Request multiple records in a single operation (proposed)

What have we accomplished ? Complete protocol-level separation among components within the DL Seamless integration with little “glue” Simple extensions of OAI-PMH Modular and portable components Efficient in speed – but not as efficient in storage

Outline Overview, Problem Experience: Case Study Projects Open Archives Initiative Hussein Suleman Dissertation DL in a Box, OCKHAM Summary and Conclusion

Digital Library In A Box Part of NSF’s National Science Digital Library ( Offers “Shrink-wrap” Open Digital Library Components – Open Source Software Users install ready-made digital library solutions, or build their own from snap- together components.

OCKHAM Simplicity (a la OCCAM’s razor) Support by Mellon and DLF Next meeting in Atlanta Jan. 8, 2003 Four main ideas: 1.Components 2.Lightweight protocols 3.Open reference models (e.g., 5S, OAIS) 4.Community perspective and involvement

5S Layers Societies Scenarios Spaces Structures Streams

Outline Overview, Problem Experience: Case Study Projects Open Archives Initiative Hussein Suleman Dissertation DL in a Box, OCKHAM Summary and Conclusion

It is possible to build DLs easily. The ODL approach to this has been developed and validated in a number of settings. Everyone is invited to: Use ODL components Refine or add ODL components, protocols Join ODL and OCKHAM For more information see:

(Somewhat) Open Issues Is this scalable? Portable ? Extensible ? Can we define all popular DL services using such a methodology? (completeness problem) Can we define DLs as configurations of ODL components? (composition problem) Is OAI-PMH a good baseline protocol ? Can we design a better baseline protocol upon which to base harvesting and repository access? To what degree is an ODL network equivalent to a monolithic system? (comparison problem)

Ultimate Goal Package different configurations into instant DL systems or subsystems DL building = component configuration All DLs speak the same language(s) Basic services are trivial to provide so more effort is spent on advanced capabilities of DLs

Selected Links CITIDEL – NCSTRL – NDLTD – NSDL – Open Archives Initiative

More Links Hussein Suleman’s Dissertation Repository Explorer DL Courseware – Virginia Tech Digital Library Research Laboratory (DLRL) – Listservs