AOL Search Speaker Series Virginia Tech’s Digital Library Research Laboratory Dec. 20, AOL HQ Edward A. Fox, Virginia Tech, Blacksburg, VA USA

Acknowledgements (Selected) Sponsors: ACM, Adobe, AOL, CAPES, CNI, CONACyT, DFG, IBM, Microsoft, NASA, NDLTD, NLM, NSF (IIS , , , ; ITR ; DUE , , , ), OCLC, SOLINET, SUN, SURA, UNESCO, US Dept. Ed. (FIPSE), VTLS

Acknowledgements: Faculty, Staff Lillian Cassel, Debra Dudley, Roger Ehrich, Joanne Eustis, Weiguo Fan, James Flanagan, C. Lee Giles, Eberhard Hilf, John Impagliazzo, Filip Jagodzinski, Rohit Kelapure, Neill Kipp, Douglas Knight, Deborah Knox, Aaron Krowne, Alberto Laender, Gail McMillan, Claudia Medeiros, Manuel Perez, Naren Ramakrishnan, Layne Watson, …

Acknowledgements: Students Pavel Calado, Yuxin Chen, Fernando Das Neves, Shahrooz Feizabadi, Robert France, Marcos Goncalves, Nithiwat Kampanya, S.H. Kim, Aaron Krowne, Bing Liu, Ming Luo, Paul Mather, Saverio Perugini, Unni. Ravindranathan, Ryan Richardson, Rao Shen, Ohm Sornil, Hussein Suleman, Ricardo Torres, Wensi Xi, Xiaoyan Yu, Baoping Zhang, Qinwei Zhu, …

Rao Shen’s Preliminary Exam: Hypothesis and Research Questions The 5S framework provides effective solutions to DL integration. –Formally define the DL integration problem? –Guide integration of domain focused DLs? How to formally model such domain specific DLs? How to integrate formally defined DL models into a union DL model? How to use the union DL model to help design and implement high quality integrated DLs? –Assess the integration?

Related Work DL interoperability approach Intermediary-basedmapping-based Consists of mediatorwrapperagent use two architectures federationUnion Archiving used in Consists of hybrid mappercomposite mapper use schema mapping use SemInt has an example LSD has an example Interrelated with

DL interoperability approach Intermediary-basedmapping-based Consists of mediatorwrapperagent use two architectures federationUnion Archiving used in Consists of hybrid mappercomposite mapper use schema mapping use Interrelated with GA trained by DL integration formalization based on

Formal Definition of DL Integration DL i =(R i, DM i, Serv i, Soc i ), 1 i n –R i is a network accessible repository –DM i is a set of metadata catalogs for all collections –Serv i is a set of services –Soc i is a society UnionRep UnionCat UnionServices UnionSociety

Formal Definition of DL Integration (Cont.) DL integration problem definition: Given n individual libraries, integrate the n DLs to create a UnionDL. Demonstration: ETANA-DL (NSF ITR w. CWRU) feathers.dlib.vt.edu

Repository1 DL1 Repository2 Union Catalog Union Repository Catalog1Catalog2 Searching Union DLDL2 archaeologists Society General Public Society Archaeologists General Public Union Society Service Browsing Service Union Service Harvesting, Mapping, Searching, Browsing, Clustering, Visualization Architecture of a Union DL

Union Catalog Integration VN Metadata Format Global Metadata Format VN Catalog HD Catalog Union Catalog Mapping Tool Wrapper Mapping Tool Wrapper HD Metadata Format Virtual Nimrin (VN) Halif DigMaster (HD) Union ArchDL

Example of Union Service: CitiViz

CitiViz: A Visual User Interface to the CITIDEL System ECDL 2004, Bath, England, September 2004 Nithiwat Kampanya, Rao Shen, Seonho Kim, Chris North, and Edward A. Fox

Digital Object Repository Collection Minimal DL Metadata Catalog Descriptive Metadata Specification A Minimal DL in the 5S Framework Structural Metadata Specification StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream

StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream Descriptive Metadata specification SpaTemOrg StraDia Arch Descriptive Metadata specification ArchDO ArchObj ArchColl Arch Metadata catalog ArchDColl ArchDR Minimal ArchDL A Minimal ArchDL in the 5S Framework

5SGraph 5S Archaeology MetaModel ArchDL Expert ArchDL Designer Structure Sub-model ETANA-DL Union Services Descriptions Harvesting Mapping Searching Browsing … Scenario Sub-model VN Metadata Format ETANA-DL Metadata Format HD Metadata Format Mapping Tool Wrapper4VNWrapper4HD Inverted Files Services DB Index Browse Service Search Service Browse DB Other ETANA-DL Services Web Interface XOAI VN Catalog HD Catalog Union Catalog 5SGen Component Pool Browsing …

Computing and Information Technology Interactive Digital Educational Library (CITIDEL) Domain: computing / information technology Genre: one-stop-shopping for teachers & learners: courseware (CSTC, JERIC), leading DLs (ACM, IEEE-CS, DB&LP, CiteSeer), PlanetMath.org, NCSTRL (technical reports), … Submission & Collection: sub/partner collections 

Led by Virginia Tech, with co-PIs: –Fox (director, DL systems) –Lee (history) –Perez (user interface, Spanish support) –Students: Ryan Richardson, Kate McDevitt, Jon Pryor, Baoping Zhang Partners –College of New Jersey (Knox) –Hofstra (Impagliazzo) –Villanova (Cassel) –Penn State (Giles)

Digital library architecture for local and interoperable CITIDEL services

CITIDEL Technology Features Component architecture (Open Digital Library) Re-use and compose re-deployable digital library components. Built Using Open Standards & Technologies OAI: Used to collect DL Resources and DL Interoperability XSL and XML: Interface rendering with multi-lingual community based translation of screens and content (Spanish, …) Perl: Component Integration ESSEX: Search Engine Functionality Very fast, utilizing in-memory processing Includes snap-shots for persistence Multi-scheming (Aaron Krowne, now at Emory U. Library) Integrates multiple classifications / views through maps, closure Extensions: clustering, visualization, personalization, …

Cluster Search Results from CITIDEL

Cluster NDLTD-Computing

CITIDEL + PIPE Adds Interaction Personalization to CITIDEL Automatically handles multi-modal conversion to Cell phone, PDA, Etc. Can be adopted to any digital data set, only requires XML file of content with hierarchy maintained. Naren Ramakrishnan and Saverio Perugini (U. Dayton)

CITIDEL -> NSDL A collection project in the National STEM (science, technolgy, engineering, and mathematics) education Digital Library – NSDL National Science Digital Library (Next slides courtesy Lee Zia, NSF)

NSDL ProgramTracks Core Integration: coordinate a distributed alliance of resource collection and service providers; and ensure reliable and extensible access to and usability of the resulting network of learning environments and resources Collections: aggregate and actively manage a subset of the digital library’s content within a coherent theme / specialty Services: increase the impact, reach, efficiency, and value of the digital library in its fully operational form Targeted (Applied) Research: have immediate impact on one or more of the other three tracks Pathways: large efforts across broad ranges of areas or approaches or users

NSDL Information Architecture Essentially as developed by the Technical Infrastructure Workgroup referenced items & collections referenced items & collections Special Databases NSDL Services NSDL Services Other NSDL Services CI Services annotation CI Services discussion CI Services personalization CI Services authentication CI Services browsing Core Services: information retrieval Core Collection- Building Services harvesting Core Collection- Building Services protocols Core Services: metadata gathering Portals & Clients Portals & Clients Portals & Clients Usage Enhancement Collection Building User Interfaces NSDL Collections NSDL Collections NSDL Collections Core NSDL “Bus”

OCKHAM Library Network (NSDL)

OCKHAM (Ming Luo) Simplicity (a la OCCAM’s razor) Support by Mellon and DLF Four main ideas: 1.Components 2.Lightweight protocols 3.Open reference models (e.g., 5S, OAIS) 4.Community perspective and involvement Funded by NSF in NSDL, with P2P, with Emory, Notre Dame, Oregon State, …

OCKHAM Proposed Services Alerting Browsing Cataloging Conversion OAI – Z39.50 Pathfinding Registry (plus others such as from adapted ODL)

A Digital Library Case Study Domain: graduate education, research Genre:ETDs=electronic theses & dissertations Submission: Collection: Project: Networked Digital Library of Theses & Dissertations (NDLTD) (supported by Ming Luo)

OCLC SRU Interface => Dr. A.K. Tyagi

ETD Union Search Mirror Site in China (CALIS) ( – popular site!)

LOCKSS Extensions: Bing Liu, Xiaoyu Zhang, Ji-Sun Kim Lots of copies keep stuff safe Stanford (Vicky Reich) Initial focus on lower levels, journals Shift to OAI, esp. for ETDs Collab with Emory (Martin Halbert) –NDIIP: AmericanSouth, MetaArchive –Help deploy and adapt, apply in other contexts Another registry Set of publisher manifests (information providers) Set of storage systems (archival storage)

Program Document Document Document Program Program Image Image Image Video Video Video open digital library OA PMH XPMH Hussein Suleman (Capetown, S. Africa)

Open Digital Library Protocol Extended OAI-PMH Protocol for Metadata Harvesting

Open Digital Library Component Extended OPEN ARCHIVE OPEN ARCHIVE

Open Digital Library Components Running now –XML-File (data provider from file system) –Search: simple or in-memory (Essex) or generalized –Union, browse, recent, filter –E-journal/review, Submit, Edit, Annotation –Recommender, Rating; Mirroring (see JCDL’02) –Working with NCSA: from DB, unstructured text Others in process –Classification/categorization –Registry (and other connections with web services)

Program Document Document ETD Program ETD Image Image ETD Video Video ETD-4 ETD DL for the Networked Digital Library of Theses and Dissertations ( Search Filter Union Recent Browse PMH ODLRecent ODLBrowse ODLUnion ODLSearch ODLUnion PMH USER INTERFACE Students and researchers ETD collections Example Open Digital Library

Open Digital Library Deployments NDLTD ( Computer Science Teaching Center ( Computing and Information Technology Interactive Digital Educational Library ( Open Archives Distributed (NSF, DFG) – enhancements to PhysNet OCKHAM Open to others through DL-in-a-box

Interest-based User Grouping Model for Collaborative Filtering in Digital Libraries 7 th ICADL 2004 Shanghai, P.R. China Dec. 15, 2004 Edward A. Fox, Seonho Kim Virginia Tech, Blacksburg, VA USA

Some Other Students/Projects Wensi Xi: Matrices, reinforcement, clusters (Microsoft) Paul Mather: mod/sim of large DLs on clusters; characterization: uses, files (NASA) Ming Luo: personalization aided by demographics Ryan Richarson: CLIR with concept maps Xiaoyan Yu: Stepping Stones and Pathways (NSF, Fernando Das Neves completed & returned to Argentina) Baoping Zhang: Physics and classification (NSF, DFG) Several: TREC with GP New projects: –Superimposed information w. PSU (NSF NSDL) –Quality and metasearch and structure w. Emory (IMLS) …

Conclusion Many DL/IR: areas, projects, students Theory Architecture Modeling and simulation Systems development and testing to: validate above, demonstrate innovations Users, interfaces, visualization, usability Special thanks to AOL for 4 years of Fellowships!