1 Digital Libraries : Archaeology, Automation, ETDs, and Enhancements Edward A. Fox Virginia Tech, USA IADLC 2005 The International Advanced Digital Library Conference in Nagoya August 25-26, 2005

2 Outline Acknowledgements Introduction: Life Cycle, Curric., 5S, Book ETANA-DL, 5S Description Theory and Automation Education: CS, ETDs Quality, Integration, and Automation Selected Links, Discussion

3 Acknowledgements: Students Pavel Calado, Yuxin Chen, Fernando Das Neves, Shahrooz Feizabadi, Robert France, Marcos Gonçalves, Nithiwat Kampanya, S.H. Kim, Aaron Krowne, Bing Liu, Ming Luo, Paul Mather, Fernando Das Neves, Unni. Ravindranathan, Ryan Richardson, Rao Shen, Ohm Sornil, Hussein Suleman, Ricardo Torres, Wensi Xi, Baoping Zhang, Qinwei Zhu, …

4 Acknowledgements: Faculty, Staff Lillian Cassel, Debra Dudley, Roger Ehrich, Joanne Eustis, Weiguo Fan, James Flanagan, C. Lee Giles, Eberhard Hilf, John Impagliazzo, Filip Jagodzinski, Rohit Kelapure, Neill Kipp, Douglas Knight, Deborah Knox, Aaron Krowne, Alberto Laender, Gail McMillan, Claudia Medeiros, Manuel Perez, Naren Ramakrishnan, Layne Watson, …

5 Other Collaborators (Selected) Brazil: FUA, UFMG, UNICAMP Case Western Reserve University Emory, Notre Dame, Oregon State Germany: Univ. Oldenburg Mexico: UDLA (Puebla), Monterrey College of NJ, Hofstra, Penn State, Villanova University of Arizona University of Florida, Univ. of Illinois University of Virginia

6 Acknowledgements - Mentors JCR Licklider – undergrad advisor ( ) –Author in 1965 of “Libraries of the Future” –Before, at ARPA, funded start of Internet Michael Kessler – BS thesis advisor –Project TIP (technical information project) –Defined bibliographic coupling Gerard Salton – graduate advisor ( ) –“Father of Information Retrieval”

Acknowledgements: Support ACM, Adobe, AOL, CAPES, CNI, CONACyT, DFG, IBM, Microsoft, NASA, NDLTD, NLM, NSF (IIS , , , ; ITR ; DUE , , , ), OCLC, SOLINET, SUN, SURA, UNESCO, US Dept. Ed. (FIPSE), VTLS

8 Outline Acknowledgements Introduction: Life Cycle, Curric., 5S, Book ETANA-DL, 5S Description Theory and Automation Education: CS, ETDs Quality, Integration, and Automation Selected Links, Discussion

9 Information Life Cycle Authoring Modifying Organizing Indexing Storing Retrieving Distributing Networking Retention / Mining Accessing Filtering Using Creating

10 DL Curriculum Framework

11 5S Layers Societies Scenarios Spaces Structures Streams

12 5S Layers Societies Scenarios Spaces Structures Streams Fire Wood Earth Metal Water 5 Elements

13 5Ss SsExamplesObjectives Streams Text; video; audio; image Describes properties of the DL content such as encoding and language for textual material or particular forms of multimedia data Structures Collection; catalog; hypertext; document; metadata Specifies organizational aspects of the DL content Spaces Measure; measurable, topological, vector, probabilistic Defines logical and presentational views of several DL components Scenarios Searching, browsing, recommending Details the behavior of DL services Societies Service managers, learners, teachers, etc. Defines managers, responsible for running DL services; actors, that use those services; and relationships among them

14 Informal 5S & DL Definitions DLs are complex systems that help satisfy info needs of users (societies) provide info services (scenarios) organize info in usable ways (structures) present info in usable ways (spaces) communicate info with users (streams)

15 Hypotheses A formal theory for DLs can be built based on 5S. The formalization can serve as a basis for modeling and building high- quality DLs.

16 Research Questions 1. Can we formally elaborate 5S? 2. How can we use 5S to formally describe digital libraries? 3. What are the fundamental relationships among the Ss and high-level DL concepts? 4. How can we allow digital librarians to easily express those relationships? 5. Which are the fundamental quality properties of a DL? Can we use the formalized DL framework to characterize those properties? 6. Where in the life cycle of digital libraries can key aspects of quality be measured and how?

17 Book Parts Ch. 1. Introduction (Motivation, Synopsis) Part 1 – The “Ss” Part 2 – Higher DL Constructs Part 3 – Advanced Topics Appendix

18 Book Parts and Chapters - 1 Ch. 1. Introduction (Motivation, Synopsis) Part 1 – The “Ss” –Ch. 2: Streams –Ch. 3: Structures –Ch. 4: Spaces –Ch. 5: Scenarios –Ch. 6: Societies

19 Book Parts and Chapters - 2 Part 2 – Higher DL Constructs –Ch. 7: Collections –Ch. 8: Catalogs –Ch. 9: Repositories and Archives –Ch. 10: Services –Ch. 11: Systems –Ch. 12: Case Studies

20 Book Parts and Chapters - 3 Part 3 – Advanced Topics –Ch. 13: Quality –Ch. 14: Integration –Ch. 15: How to build a digital library –Ch. 16: Research Challenges, Future Perspectives Appendix –A: Mathematical preliminaries –B: Formal Definitions: Ss –C: Formal Definitions: DL terms, Minimal DL –D: Formal Definitions: Archeological DL –E: Glossary of terms, mappings

21 Outline Acknowledgements Introduction: Life Cycle, Curric., 5S, Book ETANA-DL, 5S Description Theory and Automation Education: CS, ETDs Quality, Integration, and Automation Selected Links, Discussion

22

23 Map courtesy: Initial ETANA-DL Member Locations Virginia Tech Mississippi State University Vanderbilt University Canadian University College Walla Walla College Andrews University CWRU Willamette University

24

25

26 Lahav Website

27 Megiddo Opening Screen

28 Locus Screen: Pictures View all

29 Area Screen

30

31 ETANA-DL Approach Applying and extending Digital Library (DL) techniques to solve key problems: making primary data available, data preservation, and interoperability Modeling archaeological information systems using 5S to better understand the domain and design the system and the supporting services Rapidly prototyping DLs that handle heterogeneous archaeological data using componentized frameworks: –eliciting requirements –refining metamodel and union schema –modeling sites –mapping –harvesting –providing useful services

32 ETANA-DL Website

33 Marking – writing notes for a specific user Marking Items

34 Marked Items Display Sender, Date, Object OAI ID Sender Comments Options: View Record, Add record to Items Of Interest, Re-mark item (Redirect), Unmark item (Remove item from list)

35 Discussions Page Discussions about an object View/Post messages, create new threads

36 Recommendations Items recommended on the basis of similar interests

37 ETANA-DL Multi-dimensional Browsing 3 new sites 2 new types of artifacts

38 ETANA-DL Visual Browsing Service Visual Browse By site

39 Visual Browsing Nimrin: Topographical Drawings Full siteNorth west quadrant Square: N40/W20

40 Visual Browsing Nimrin : Square information Square: N40/W20 Locus: 86 Loci layout

41 Visual Browsing Nimrin : locus sheet

42 Visual Browsing Bab edh-Dhra' Cemetery Pottery # 25

43 Visual Browsing Bab edh-Dhra' Cemetery Pottery # 25

44 ETANA Societies 1.Historic and pre-historic societies (being studied) 2.Archaeologists (in academic institutes, fieldwork settings, or local and national governmental bodies) 3.Project directors 4.Technical staff (consisting of photographers, technical illustrators, and their assistants) 5.Field staff (responsible for the actual work of excavation) 6.Camp staff (e.g., camp managers, registrars, tool stewards) 7.General public (e.g., educators, learners, citizens)

45 ETANA Societies Social issues 1.Who owns the finds? 2.Where should they be preserved? 3.What nationality and ethnicity do they represent? 4.Who has publication rights? 5.What interactions took place between those at the site studied, and others? What theories are proposed by whom about this?

46 ETANA Scenarios 1.Life in the site in former times 2.Digital recording: the planning stage and the excavation stage 3.Planning stage: remote sensing, fieldwalking, field surveys, building surveys, consulting historical and other documentary sources, and managing the sites and monuments 4.Excavation 1.Detailed information is recorded, including for each layer of soil, and for features such as pole holes, pits, and ditches. 2.Data about each artifact is recorded together with information about its exact find spot. 3.Numerous environmental and other samples are taken for laboratory analysis, and the location and purpose of each is carefully recorded. 4.Large numbers of photographs are taken, both general views of the progress of excavation and detailed shots showing the contexts of finds. 5.Organization and storage of material 6.Analysis and hypotheses generation and testing 7.Publications, museum displays 8.Information services for the general public

47 ETANA Spaces 1.Geographic distribution of found artifacts 2.Temporal dimension (as inferred by archaeologists) 3.Metric or vector spaces 1.used to support retrieval operations, and to calculate distance (and similarity) 2.used to browse / constrain searches spatially 4.3D models of the past, used to reconstruct and visualize archaeological ruins 5.2D interfaces for human-computer interaction

48 ETANA Structures 1.Site Organization 1.Region, site, partition, sub-partition, locus, … 2.Temporal orderings (ages, periods) 3.Taxonomies 1.for bones, seeds, building materials, … 4.Stratigraphic relationships 1.above, beneath, coexistent

49 ETANA Streams 1.successive photos and drawings of excavation sites, loci, unearthed artifacts 2.audio and video recordings of excavation activities and discussions 3.textual reports 4.3D models used to reconstruct and visualize archaeological ruins.

50 Outline Acknowledgements Introduction: Life Cycle, Curric., 5S, Book ETANA-DL, 5S Description Theory and Automation Education: CS, ETDs Quality, Integration, and Automation Selected Links, Discussion

51 5S and DL formal definitions and compositions (April 2004 TOIS)

52

53

54

55 The XML Log Format Log SessionIdMachineInfo StatementTransactionTimestamp SessionInfoRegisterInfo StatementEventTimestamp Action SearchBrowse StoreSysInfoUpdate SearchBy QueryString CatalogCollection PresentationInfo StatusInfo Timeout

56 5S Modeling -> Systems

57 Tools/Applications

58 Digital Object Repository Collection Minimal DL Metadata Catalog Descriptive Metadata Specification A Minimal DL in the 5S Framework Structural Metadata Specification StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream

59 StreamsStructuresSpacesScenariosSocieties indexing browsing searching services hypertext Structured Stream Descriptive Metadata specification SpaTemOrg StraDia Arch Descriptive Metadata specification ArchDO ArchObj ArchColl Arch Metadata catalog ArchDColl ArchDR Minimal ArchDL A Minimal ArchDL in the 5S Framework

60 Overview of 5SGraph Workspace (instance model) Structured toolbox (metamodel)

61

62 Outline Acknowledgements Introduction: Life Cycle, Curric., 5S, Book ETANA-DL, 5S Description Theory and Automation Education: CS, ETDs Quality, Integration, and Automation Selected Links, Discussion

63 Computing and Information Technology Interactive Digital Educational Library (CITIDEL) Domain: computing / information technology Genre: one-stop-shopping for teachers & learners: courseware (CSTC, JERIC), leading DLs (ACM, IEEE-CS, DB&LP, CiteSeer), PlanetMath.org, NCSTRL (technical reports), … Submission & Collection: sub/partner collections 

64 Digital library architecture for local and interoperable CITIDEL services

65

CITIDEL -> NSDL A collection project in the National STEM (science, technolgy, engineering, and mathematics) education Digital Library – NSDL National Science Digital Library (Next slides courtesy Lee Zia, NSF)

67 NSDL ProgramTracks Core Integration: coordinate a distributed alliance of resource collection and service providers; and ensure reliable and extensible access to and usability of the resulting network of learning environments and resources Collections: aggregate and actively manage a subset of the digital library’s content within a coherent theme / specialty Services: increase the impact, reach, efficiency, and value of the digital library in its fully operational form Targeted (Applied) Research: have immediate impact on one or more of the other three tracks Pathways: large efforts across broad ranges of areas or approaches or users

68

69

70 NSDL Information Architecture Essentially as developed by the Technical Infrastructure Workgroup referenced items & collections referenced items & collections Special Databases NSDL Services NSDL Services Other NSDL Services CI Services annotation CI Services discussion CI Services personalization CI Services authentication CI Services browsing Core Services: information retrieval Core Collection- Building Services harvesting Core Collection- Building Services protocols Core Services: metadata gathering Portals & Clients Portals & Clients Portals & Clients Usage Enhancement Collection Building User Interfaces NSDL Collections NSDL Collections NSDL Collections Core NSDL “Bus”

71 Digital Libraries in Education Analytical Survey, ed. Leonid Kalinichenko © 2003, Transforming the Way to Learn DLs of Educational Resources & Services Integrated/Virtual Learning Environment Educational Metadata Current DLEs: US (NSDL, DLESE, CITIDEL, NDLTD), Europe (Scholnet, Cyclades), UK (Distributed National Electronic Resource)

A Digital Library Case Study Domain: graduate education, research Genre:ETDs=electronic theses & dissertations Submission: Collection: Project: Networked Digital Library of Theses & Dissertations (NDLTD)

Student Gets Committee Signatures and Submits ETD Signed Grad School

Library Catalogs ETD, Access is Opened to the New Research WWW NDLTD

75

76

77

78 OCLC SRU Interface

79

80 ETD Union Search Mirror Site in China (CALIS) ( – popular site!)

81

82 Board of Directors Suzie Allard (ETD 2004, U. Kentucky) Denise A. D. Bedford (World Bank) Julia C. Blixrud (ARL, SPARC) José Luis Borbinha (Natl Lib Portugal) Alex Byrne (ETD 2005, ADT: Australia) Tony Cargnelutti (ETD 2005, Australia) Vinod Chachra (VTLS) Susan Copeland (RGU, UK) Jude Edminster (Bowling Green St. U.) Scott Eldredge (Treasurer, ETD 2002, BYU) Edward A. Fox (Exec Director,Virginia Tech) John H. Hagen (West Virginia U.) Thomas B. Hickey (OCLC) Christine Jewell (U. Waterloo, Canada) Delphine Lewis (ProQuest) Joan K. Lippincott (CNI) Mike Looney (Adobe) Gail McMillan (Secretary, Virginia Tech) Joseph Moxley (ETD 2000, USF) Eva M ü ller (U. Uppsala, Sweden) Ana Pavani (PUC Rio, Brazil) Axel Plathe (UNESCO, Paris) Sharon Reeves (National Library Canada) Peter Schirmbacher (ETD 2003, Humboldt) Hussein Suleman (U.Cape Town, S. Africa) Shalini R. Urs (U. Mysore, India) Eric F. Van de Velde (ETD 2001, Caltech)

83 Selected Projects / Sponsors Australia (ADT) Brazil (BDT, IBICT) Canada Catalunya Chile (Cybertesis) Germany India (Vidyanidhi) Korea OhioLINK: 79 colleges/univs Portugal (National Library) South Africa UK (British Library, JISC, Edinburgh, …) UNESCO (especially Latin America, Eastern Europe, Africa) Venezuela

84 Why ETD? Short Answer For Students: –Gain knowledge and skills for the Information Age –Richer communication (digital information, multimedia, …) For Universities: –Easy way to enter the digital library field and benefit thereby For the World: –Global digital library – large, useful, many services General: –Save time and money –Increased visibility for all associated with research results

85

86 Outline Acknowledgements Introduction: Life Cycle, Curric., 5S, Book ETANA-DL, 5S Description Theory and Automation Education: CS, ETDs Quality, Integration, and Automation Selected Links, Discussion

87 Describing Quality in Digital Libraries What’s a “good” digital Library? –Central Concept: Quality! –Hypotheses of this work: Formal theory can help to define “what’s a good digital library” by: New formalizations of quality indicators for DLs within our 5S framework Contextualizing these measures within the Information Life Cycle

88 Quality and the Information Life Cycle

89 Formal Definition of DL Integration DL i =(R i, DM i, Serv i, Soc i ), 1 i n –R i is a network accessible repository –DM i is a set of metadata catalogs for all collections –Serv i is a set of services –Soc i is a society UnionRep UnionCat UnionServices UnionSociety

90 Formal Definition of DL Integration (Cont.) DL integration problem definition: Given n individual libraries, integrate the n DLs to create a UnionDL.

91 Repository1 DL1 Repository2 Union Catalog Union Repository Catalog1Catalog2 Searching Union DLDL2 archaeologists Society General Public Society Archaeologists General Public Union Society Service Browsing Service Union Service Harvesting, Mapping, Searching, Browsing, Clustering, Visualization Architecture of a Union DL

92 Example of Union Service: CitiViz

93 Multidimensional Browsing: Percentages of Animal Bones Across Nimrin Cultural Phases

94 local schemaglobal schema

95 Mapping recommendation

96 5S Meta Model 5SGraph DL Expert DL Designer 5SL DL Model 5SLGen Practitioner Researcher Tailored DL Services Teacher c omponent pool ODLSearch, ODLBrowse, ODLRate, ODLReview, ……. Requirements (1) Analysis (2) Implementation (4) Design (3) 5SGraph5SGen Mapping Tool 5SSuite

97 5SGraph 5S Archaeology MetaModel ArchDL Expert ArchDL Designer ETANA-DL Union Services Descriptions Harvesting Mapping Searching Browsing … Scenario Sub-model VN Metadata Format ETANA-DL Metadata Format HD Metadata Format Mapping Tool Wrapper4VNWrapper4HD Inverted Files Services DB Index Browse Service Search Service Browse DB Other ETANA-DL Services Web Interface XOAI VN Catalog HD Catalog Union Catalog 5SGen Component Pool Browsing …

98 Outline Acknowledgements Introduction: Life Cycle, Curric., 5S, Book ETANA-DL, 5S Description Theory and Automation Education: CS, ETDs Quality, Integration, and Automation Selected Links, Discussion

99 Selected Links - CITIDEL (computing education resources) – NCSTRL (computing technical reports) – NDLTD (electronic theses and dissertations worldwide) – and etdguide.org NSDL (National Science Digital Library) – OAI (Open Archives Initiative) – Virginia Tech Digital Library Research Laboratory (DLRL, –5S, AmericanSouth.Org, CSTC, DL-in-a-box, ENVISION, ETANA, MARIAN, NDLTD, NSDL, OAD, ODL, …)

100 Questions? Discussion? Thank You!