1 From the WWW and Minimal Digital Libraries, to Powerful Digital Libraries: Why and How Edward A. Fox ICADL 2005 Bangkok, Thailand – December 13, 2005

Acknowledgements (selected) 5S Helpers: Weiguo Fan, Marcos Gonçalves, Doug Gorton, Rohit Kelapure, Neill Kipp, Uma Murthy, Ananth Raghavan, Rao Shen, Hussein Suleman, Srinivas, Vemuri, Layne Watson, … Sponsors: ACM, Adobe, AOL, CAPES, CNI, CONACyT, DFG, IBM, Microsoft, NASA, NDLTD, NLM, NSF (IIS , , , , , ; ITR ; DUE , , , ), OCLC, SOLINET, SUN, SURA, UNESCO, US Dept. Ed. (FIPSE), VTLS

3 Outline WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs –Learning Object Repository Requirements –NSDL, OCKHAM, User Interfaces, Services Why –DL education, Practical systems –General requirements, Domain specific requirements –Personal DLs, Global DLs How –Components, Metamodels, Models –Graphical aids, Generators –Integration, Quality

4 WWW and DLs Both emerged in early 1990s. Convergence began around Example: Google spun off from Stanford DL. Crawling WWW is one way to build DLs. WWW support many portals to DLs. Parts of WWW that have catalogs (e.g., Yahoo categories) are close to DLs. Web Services help move WWW toward DLs, as the Semantic Web emerges.

5 Degree of Structure Chaotic OrganizedStructured WebDLsDBs

6 NSDL Information Architecture Essentially as developed by the Technical Infrastructure Workgroup referenced items & collections referenced items & collections Special Databases NSDL Services NSDL Services Other NSDL Services CI Services annotation CI Services discussion CI Services personalization CI Services authentication CI Services browsing Core Services: information retrieval Core Collection- Building Services harvesting Core Collection- Building Services protocols Core Services: metadata gathering Portals & Clients Portals & Clients Portals & Clients Usage Enhancement Collection Building User Interfaces NSDL Collections NSDL Collections NSDL Collections Core NSDL “Bus”

7

8

9

10 Outline WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs –Learning Object Repository Requirements –NSDL, OCKHAM, User Interfaces, Services Why –DL education, Practical systems –General requirements, Domain specific requirements –Personal DLs, Global DLs How –Components, Metamodels, Models –Graphical aids, Generators –Integration, Quality

11 Minimal Digital Libraries Key concepts, core ideas Minimalist perspective Underlying concepts: 5S (ETANA example) Higher DL constructs Bases: –Literature –Informal explanations –Formal definitions

12 Informal 5S & DL Definitions DLs are complex systems that help satisfy info needs of users (societies) provide info services (scenarios) organize info in usable ways (structures) present info in usable ways (spaces) communicate info with users (streams)

13 5Ss SsExamplesObjectives Streams Text; video; audio; image Describes properties of the DL content such as encoding and language for textual material or particular forms of multimedia data Structures Collection; catalog; hypertext; document; metadata Specifies organizational aspects of the DL content Spaces Measure; measurable, topological, vector, probabilistic Defines logical and presentational views of several DL components Scenarios Searching, browsing, recommending Details the behavior of DL services Societies Service managers, learners, teachers, etc. Defines managers, responsible for running DL services; actors, that use those services; and relationships among them

14 Example of 5Ss: ETANA-DL Archaeological DL (Electronic Tools for Ancient Near Eastern Archaeology Digital Library) Integrated DL –Heterogeneous data handling Applies and extends the OAI-PMH –Open Archives Initiative Protocol for Metadata Handling Design considerations –Componentized –Extensible –Portable –Work based on 5S framework

15

16 ETANA Societies 1.Historic and pre-historic societies (being studied) 2.Archaeologists (in academic institutes, fieldwork settings, or local and national governmental bodies) 3.Project directors 4.Technical staff (consisting of photographers, technical illustrators, and their assistants) 5.Field staff (responsible for the actual work of excavation) 6.Camp staff (e.g., camp managers, registrars, tool stewards) 7.General public (e.g., educators, learners, citizens)

17 ETANA Societies – cont’d Social issues 1.Who owns the finds? 2.Where should they be preserved? 3.What nationality and ethnicity do they represent? 4.Who has publication rights? 5.What interactions took place between those at the site studied, and others? What theories are proposed by whom about this?

18 ETANA Scenarios 1.Life in the site in former times 2.Digital recording: the planning stage and the excavation stage 3.Planning stage: remote sensing, fieldwalking, field surveys, building surveys, consulting historical and other documentary sources, and managing the sites and monuments 4.Excavation 1.Detailed information is recorded, including for each layer of soil, and for features such as pole holes, pits, and ditches. 2.Data about each artifact is recorded together with information about its exact find spot. 3.Numerous environmental and other samples are taken for laboratory analysis, and the location and purpose of each is carefully recorded. 4.Large numbers of photographs are taken, both general views of the progress of excavation and detailed shots showing the contexts of finds. 5.Organization and storage of material 6.Analysis and hypotheses generation and testing 7.Publications, museum displays 8.Information services for the general public

19 ETANA Spaces 1.Geographic distribution of found artifacts 2.Temporal dimension (as inferred by archaeologists) 3.Metric or vector spaces 1.used to support retrieval operations, and to calculate distance (and similarity) 2.used to browse / constrain searches spatially 4.3D models of the past, used to reconstruct and visualize archaeological ruins 5.2D interfaces for human-computer interaction

20 ETANA Structures 1.Site Organization 1.Region, site, partition, sub-partition, locus, … 2.Temporal orderings (ages, periods) 3.Taxonomies 1.for bones, seeds, building materials, … 4.Stratigraphic relationships 1.above, beneath, coexistent

21 ETANA Streams 1.successive photos and drawings of excavation sites, loci, unearthed artifacts 2.audio and video recordings of excavation activities and discussions 3.textual reports 4.3D models used to reconstruct and visualize archaeological ruins.

22 5S and DL formal definitions and compositions (April 2004 TOIS)

23 Digital Object Repository Collection Minimal DL Metadata Catalog Descriptive Metadata Specification A Minimal DL in the 5S Framework Structural Metadata Specification StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream

24

25 Outline WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs –Learning Object Repository Requirements –NSDL, OCKHAM, User Interfaces, Services Why –DL education, Practical systems –General requirements, Domain specific requirements –Personal DLs, Global DLs How –Components, Metamodels, Models –Graphical aids, Generators –Integration, Quality

26 Powerful Digital Libraries Features, functions (LOR report) Services User interfaces Services to support interactions –tasks, activities

27 Learning Object Repository S/W WCET EduTools LOR Comparative Research Report of Nov. 2004, at – –For 4 project partners, giving requirements: –University of Georgia System –Utah Education Network –Virginia Community College System –Virginia Tech Review criteria 10 categories 44 features Reviews of six products

28 LOR S/W – Feature Categories 1.Discovery tools 2.Aggregation tools 3.Community & evaluation 4.Meta-tagging 5.Content management 6.Digital rights management 7.Presentation and consortia issues 8.Integration and interoperability 9.Technical considerations 10.Pricing/licensing/other

29 LOR S/W – Feature Category 1 Discovery tools –Searching Indexable? –Browsing Schema support Vocabulary and thesaurus support –Syndication and notification

30 LOR S/W – Feature Category 2 Aggregation tools –Personal collections Bookmarked objects Order? Organization? Shareable? –Content aggregator and packaging tools Objects made of elements Create/package/use aggregate objects

31 LOR S/W – Feature Category 3 Community & evaluation –Evaluation system annotations –Context usage illustrators reuse –Wish lists requests

32 LOR S/W – Feature Category 4 Meta-tagging –Metadata markup tool (form, batch) –Schema support (IEEE LOM, DC) –Indexing workflow support (route records through people and processes) –Import and export tools –Unique identifier support

33 LOR S/W – Feature Category 5 Content management –Authoring and publishing workflow support –Version control & archiving functions –Authoring tools (direct, or through packages like Word)

34 LOR S/W – Feature Category 6 Digital rights management –DRM (enforce rights, create rights policies, associate specific content with specific rights policies) –Payment and fulfillment (permit users to pay for access, or to get non-digital copies)

35 LOR S/W – Feature Category 7 Presentation and consortia issues –Accessibility (W3C guidelines: WCAG) –Multiple output formats (HTML, WAP) –Customized look and feel –Internationalization –Multiple collections –Media transformation and display (on upload or on demand, thumbnails too)

36 LOR S/W – Feature Category 8 Integration and interoperability –Federation and harvesting –Course management integration (CMS repository links, searches) –API and Web Service support (separate repository backend, or one integrated with authoring/aggregation)

37 Harvesting: Black Box Perspective OA 1OA 2OA 4OA 3OA 5OA 6OA 7

38 LOR S/W – Feature Category 9 Technical considerations –Authentication –Authorization & Personalization –Usage reporting –Unix/Linux/Apple server support –Windows server support –Application server requirements –Database requirements –Scalability –Tech model (P2P?) –Support (help?) –Staffing requirements –Client browser requirements (version?)

39 LOR S/W – Feature Category 10 Pricing/licensing/other –Company profile –Number of installations –Costs / licensing model

40

41

42 NSDL: Collection Services Discovery of content Classification and cataloguing Acquisition and/or linking; referencing Disciplinary-based themes define a natural body of content, but other possibilities are also encouraged Access to massive real-time or archived datasets Software tool suites for analysis, modeling, simulation, or visualization Reviewed commentary on learning materials and pedagogy

43 NSDL: User/Other Services Help services, frequently asked questions, etc. Synchronous/asynchronous collaborative learning environments using shared resources Mechanisms for building personal annotated digital information spaces Reliability testing for applets or other digital learning objects Audio, image, and video search capability Metadata system translation Community feedback mechanisms

44 OCKHAM Proposed Services NSDL project where University Libraries constitute a P2P network, supporting: Alerting Browsing Cataloging Conversion OAI – Z39.50 Pathfinding Registry

45 OCLC SRU Interface

46

47 ETD Union Search Mirror Site in China (CALIS) ( – popular site!)

48

49 Ontology: Applications

50 Ontology: Applications Expand definition of minimal DL by characterizing –typical DL services –in the context of “employs” and “produces” relationships Use characterization to: –Reason about how DL services can be built from other DL components –As well as be composed with other services through extension or reuse

51 Composition of key fundamental / infrastructure services

52

53 Outline WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs –Learning Object Repository Requirements –NSDL, OCKHAM, User Interfaces, Services Why –DL education, Practical systems –General requirements, Domain specific requirements –Personal DLs, Global DLs How –Components, Metamodels, Models –Graphical aids, Generators –Integration, Quality

54 Why? Support DL education Practical systems General requirements: US-Korea diagram Domain specific requirements: DLEs Example 1: Personal DLs Example 2: Global DLs

55 DL Curriculum Framework

56 Foundations for Information Systems: Digital Libraries and the 5S Framework Ch. 1. Introduction (Motivation, Synopsis) Part 1 – The “Ss” Part 2 – Higher DL Constructs Part 3 – Advanced Topics Appendix

57 Book Parts and Chapters - 1 Ch. 1. Introduction (Motivation, Synopsis) Part 1 – The “Ss” –Ch. 2: Streams –Ch. 3: Structures –Ch. 4: Spaces –Ch. 5: Scenarios –Ch. 6: Societies

58 Book Parts and Chapters - 2 Part 2 – Higher DL Constructs –Ch. 7: Collections –Ch. 8: Catalogs –Ch. 9: Repositories and Archives –Ch. 10: Services –Ch. 11: Systems –Ch. 12: Case Studies

59 Book Parts and Chapters - 3 Part 3 – Advanced Topics –Ch. 13: Quality –Ch. 14: Integration –Ch. 15: How to build a digital library –Ch. 16: Research Challenges, Future Perspectives Appendix –A: Mathematical preliminaries –B: Formal Definitions: Ss –C: Formal Definitions: DL terms, Minimal DL –D: Formal Definitions: Archeological DL –E: Glossary of terms, mappings

60 Practical Systems Commercial: IBM, VTLS, … Open Source –Greenstone –CWIS (for NSDL) –Institutional repositories DSpace Fedora

61 Institutional Repositories “A university-based institutional repository is a set of services that a university offers to the members of its community for the management and dissemination of digital materials created by the institution and its community members. It is most essentially an organizational commitment to the stewardship of these digital materials, including long-term preservation where appropriate, as well as organization and access or distribution.” Lynch, C.A. In ARL Bimonthly Report 226, pp. 1-7, Feb. 2003,

62 Application Domain Related InstitutionsExamples Technical ChallengesBenefit / Impact Publishing Publishers, Eprint archives OAI Quality control, opennessAggregation, organization Education Schools, colleges, universities NSDL, NCSTRL Knowledge management, reuseability Access to data Art, CultureMuseumAMICO, PRDLA Digitization, describing, catalogingGlobal understanding Science Government, Academia, Commerce NVO, PDG, SwissProt, UK eScience,European Union Commission Data models reproducibility, faster reuse, faster advance (e) Government Government Agencies (all levels) Census Intellectual property rights, privacy, multi-national Accountability, homeland security (e) Commerce, (e) Industry Legal institutionsCourt cases, patents Developing standardsStandardization, economic development History, Heritage FoundationsAmerican Memory Content, context, interpretation Long term view, perspective, documentation, recording, facilitating, interpretation, understanding Cross- cutting Library, Archive Web, personal collections Multi-language, preservation, scalability, interoperability, dynamic behavior, workflow, sustainability, ontologies, distributed data, infrastructure Reduced cost, increased access, pereservation, democratization, leveling, peace, competitiveness Reagan MooreEd FoxReagan MooreEd Fox June 2002for NSFJune 2002for NSF

63 Digital Libraries in Education Analytical Survey, ed. Leonid Kalinichenko © 2003, Transforming the Way to Learn DLs of Educational Resources & Services Integrated/Virtual Learning Environment Educational Metadata Current DLEs: US (NSDL, DLESE, CITIDEL, NDLTD), Europe (Scholnet, Cyclades), UK (Distributed National Electronic Resource)

64 Digital Libraries in Education - 2 Advanced Frameworks & Methodologies –Instructional course development with learning module repositories, Learning Object reuse –Community organization around DLEs –Other content for science and research –Cyberinfrastructure, data grids –Curriculum-based interfaces (see Krowne et al.) –Concept-based organization of learning materials and courses (CMs, ontologies)

65 DLEs: Future Vision (p. 6) Global learning environment of the future: Student-centered Interactive and dynamic Enabling group work on real world problems Enabling students to determine their own learning routes (styles, personalization) Supporting lifelong learning

66 DLEs: Objectives (p. 11) Long-range: lifelong/distance/anytime- anywhere Intermediate goals –Support for students, teachers, parents –Enhanced student performance –More students excited about science –More Internet-based science educ. resources with increased quality and comprehensiveness, easy to discover and retrieve, preserved and universally available

67 DLEs: Guiding Principles (p. 12) Driven by educational and science needs Facilitating educational innovation Stable, reliable, permanent Accessible to all Leverage prior research: DL, courseware, … Adaptable to new technologies Supporting decentralized services Resource integration thru tools/organization

68 Personal DLs Old Dominion University’s Kepler Microsoft and Google support to move search and other services to the PC to include local data Microsoft MyLifeBits project with SenseCam

69

70 ETANA-DL Global Architecture DigBase and DigKit Lahav Nimrin Umayri Hisban Megiddo Jalul New Sites DATABASEWRAPPERSDATABASEWRAPPERS ETANA-DL UNION CATALOG Search USERINTERFACEUSERINTERFACE Browse Recommend Note Personalize Review Visualizations Archaeology Specific Work in progress …

71 Megiddo Opening Screen

72 Locus Screen: Pictures View all

73 Area Screen

74 Repository1 DL1 Repository2 Union Catalog Union Repository Catalog1Catalog2 Searching Union DLDL2 archaeologists Society General Public Society Archaeologists General Public Union Society Service Browsing Service Union Service Harvesting, Mapping, Searching, Browsing, Clustering, Visualization Global DL: Architecture of a Union DL

75 Outline WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs –Learning Object Repository Requirements –NSDL, OCKHAM, User Interfaces, Services Why –DL education, Practical systems –General requirements, Domain specific requirements –Personal DLs, Global DLs How –Components, Metamodels, Models –Graphical aids, Generators –Integration, Quality

76 How? Components Metamodels Models Graphical model building aids DL generators Integration Quality

Program Document Document Document Program Program Image Image Image Video Video Video componentized digital library ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

78 Discovery Current Awareness Preservation Service Providers Data Providers Metadata harvesting The World According to OAI: Open Archives Initiative – Protocol for Metadata Harvesting

79

80

81 Metamodels Completed –Minimal –Archaeological Planned –Practical –System oriented Doug Gorton’s thesis, so people can build models for their systems, and have them generated to work with a particular DL system

82 Digital Object Repository Collection Minimal DL Metadata Catalog Descriptive Metadata Specification A Minimal DL in the 5S Framework Structural Metadata Specification StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream

83 5SL – The Minimal DL Metamodel

84 StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream Descriptive Metadata specification SpaTemOrg StraDia Arch Descriptive Metadata specification ArchDO ArchObj ArchColl Arch Metadata catalog ArchDColl ArchDR Minimal ArchDL A Minimal ArchDL in the 5S Framework

85

86 Overview of 5SGraph Workspace (instance model) Structured toolbox (metamodel)

87 Tools/Applications

88 5SGen – Version 2: ODL, Services, Scenarios

89 XML-based DL Log Standard Log analysis –is a source of information on: How patrons really use DL services How systems behave while supporting user information seeking activities Used to: –Evaluate and enhance services –Guide allocation of resources Common practice in the web setting –Supported by web servers, proxy caches DL Logging can be more detailed

90 The XML Log Format Log SessionIdMachineInfo StatementTransactionTimestamp SessionInfoRegisterInfo StatementEventTimestamp Action SearchBrowse StoreSysInfoUpdate SearchBy QueryString CatalogCollection PresentationInfo StatusInfo Timeout

91 DL Integration What is “DL Integration” –Hide distribution –Hide heterogeneity –Enable autonomy of individual component Why Integration –island-DLs –inability to seamlessly and transparently access knowledge across DLs Utilize various autonomous DLs in concert

92 Formal Definition of DL Integration DL i =(R i, DM i, Serv i, Soc i ), 1 i n –R i is a network accessible repository –DM i is a set of metadata catalogs for all collections –Serv i is a set of services –Soc i is a society UnionRep UnionCat UnionServices UnionSociety

93 Formal Definition of DL Integration (Cont.) DL integration problem definition: Given n individual libraries, integrate the n DLs to create a UnionDL.

94 ETANA-DL Approach Applying and extending Digital Library (DL) techniques to solve key problems: making primary data available, data preservation, and interoperability Modeling archaeological information systems using 5S to better understand the domain and design the system and the supporting services Rapidly prototyping DLs that handle heterogeneous archaeological data using componentized frameworks: –eliciting requirements –refining metamodel and union schema –modeling sites –mapping –harvesting –providing useful services

95 Example of Union Service: CitiViz

96 Union Catalog Integration VN Metadata Format Global Metadata Format VN Catalog HD Catalog Union Catalog Mapping Tool Wrapper Mapping Tool Wrapper HD Metadata Format Virtual Nimrin (VN) Halif DigMaster (HD) Union ArchDL

97 local schemaglobal schema

98 Describing Quality in Digital Libraries What’s a “good” digital Library? –Central Concept: Quality! –Hypotheses of this work: Formal theory can help to define “what’s a good digital library” by: New formalizations of quality indicators for DLs within our 5S framework Contextualizing these measures within the Information Life Cycle

99 Quality Dimensions

100 Quality and the Information Life Cycle

101 Summary WWW and Digital Libraries (DLs) Minimal DLs Powerful DLs –Learning Object Repository Requirements –NSDL, OCKHAM, User Interfaces, Services Why –DL education, Practical systems –General requirements, Domain specific requirements –Personal DLs, Global DLs How –Components, Metamodels, Models –Graphical aids, Generators –Integration, Quality

102 Selected Links - CITIDEL (computing education resources) – NCSTRL (computing technical reports) – NDLTD (electronic theses and dissertations worldwide) – and etdguide.org NSDL (National Science Digital Library) – OAI (Open Archives Initiative) – Virginia Tech Digital Library Research Laboratory (DLRL, –5S, AmericanSouth.Org, CSTC, DL-in-a-box, ENVISION, ETANA, MARIAN, NDLTD, NSDL, OAD, ODL, …)

103 Questions? Discussion? Thank You!