1. From NDLTD to Technology for Digital Libraries: Progress and Challenges Ricoh July 27, 1999 Edward A. Fox fox@vt.edu CC CS DLRL Internet TIC Virginia Tech, Blacksburg, VA, USA

2. Acknowledgements (Selected) F Trip Support: Kobe, NAIST, NEC, Ricoh, ULIS F Sponsors: ACM, Adobe, IBM, Microsoft, NSF, OCLC, US Dept. of Education, … F Co-PIs: Marc Abrams, Robert Akscyn, John Eaton, Brian Kleiner, Gail McMillan F Students: Fernando Das Neves, Robert France, Neill Kipp, Paul Mather, Constantinos Phanouriou, James Powell, Ohm Sornil, David Watkins, Chang Zhang, Jianxin Zhao

3. Virginia Tech Background F Largest university in Virginia, land-grant, town population 35K plus 25K students F Blacksburg Electronic Village, since 1992, with 80% of community on Internet F Net.Work.Virginia, largest ATM network, with over 600 sites, for education, research, govt F LMDS, Local Multipoint Distribution Service, gigabit wireless networking - 1/3 of Virginia F Math Emporium, 500 workstations F Faculty Development Initiative, round 2

4. Virginia Tech CS F Department of CS focussed on HCI F $2M labs: usability, group decisions, info access F Faculty (+ Abrams, Kafura, Shaffer, …) –Barfield (ISE - wearable) –Carroll (design, scenarios) –Ehrich (equipment, graphics) –Hartson (theory & methodology, remote evaluation) –Hix (usability, VR/CAVE) –Rosson (object orientation/languages, collaboration) –Williges (ISE - experimentation, meta-evaluation)

5. ACITC F Advanced Communications and Information Technology Center, opening summer 2000 F Connects to the library, with a focus on IT F 1/3 high-tech (multimedia) classrooms F 1/3 digital/electronic library (reading room) F 1/3 research labs: 10, including: –Digital Library Research Laboratory (DLRL) –Center for Applied Technologies in the Humanities –HCI; HPC; Multimedia; Visualization (CAVE), … –Spaces for industry-supported labs, visitors

6. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress F DL as industrial success

7. Digital Libraries --- Objectives F World Lit.: 24hr / 7day / from desktop F Integrated “super” information systems: 5S: streams, structures, spaces, scenarios, societies F Ubiquitous, Higher Quality, Lower Cost F Education, Knowledge Sharing, Discovery F Disintermediation -> Collaboration F Universities Reclaim Property F Interactive Courseware, Student Works F Scalable, Sustainable, Usable, Useful

8. DLs: Why of Global Interest? F National projects can preserve antiquities and heritage: cultural, historical, linguistic, scholarly F Knowledge and information are essential to economic and technological growth, education F DL - a domain for international collaboration –wherein all can contribute and benefit –which leverages investment in networking –which provides useful content on Internet & WWW –which will tie nations and peoples together more strongly and through deeper understanding

9. Why of Interest in Computing? F Next step in fields of DBMS, HT, IR, MM F Efficiency requires advances in, e.g., –algorithms and data structures (ex., MPHF) –networking (ex., HTTP-NG) –OS (ex., support for streams) F Effectiveness requires advances in, e.g., –AI (ex., multilingual texts, user adaptation) –HCI (ex., visualization, DLs embedded in activities) F CS Educ. can benefit; CS can aid Dist. Educ.

10. Digital Libraries --- Virginia Tech F MARIAN (NLM) F CS DL Prototype - ENVISION (NSF, ACM) F TULIP (Elsevier, OCLC) F BEV History Base (NSF, Blacksburg) F DL for CS Education - EI (NSF, ACM) F WATERS, NCSTRL (NSF) F NDLTD (SURA, US Dept. of Education) F CSTC (NSF, ACM), CRIM (NSF, SIGMM) F WCA (Log) Repository (W3C) F VT-PetaPlex-1 (Knowledge Systems)

11. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress F DL as industrial success

12. DLs Shorten the Chain from Editor Publisher A&I Consolidator Library Reviewer

13. DLs Shorten the Chain to Editor A&I Digital Library Reviewer

14. DLs --- Educational Implications F Support distance education F Level the playing field by giving everyone access to high quality resources F Enrich learning by giving access to primary rather than secondary materials/objects F Integrate simulations, visualizations, and multi-modal presentations to enhance learning F Allow learn-at-your-own-pace, ensuring success F Allow specialized interfaces, or embedded DL for rich support of learning activities

15. How do universities and digital libraries relate? F Each U. will have its own digital library. Hence there will be large numbers (i.e., critical mass). F All students will learn how to use and how to “feed” digital libraries (and bring those habits to future work as needs and skills). F All digital library problems (esp. federation, flexibility, personalization) appear at U’s (so they are a good type of testbed, with willing collaborators in-place for developing solutions).

16. SMETE Library (from www.dlib.org) F Context: Global movement toward Digital Libraries (see April 1998 CACM) F NSF effort: Science, Mathematics, Engineering, and Technology Education Digital Library (focussed on undergraduates) –3 workshops, yearly increasing funds / new calls F SMETE Library likely to operate as distributed federation, with separate parts for each key discipline, and to lead to a global effort

17. Enhancing Learning with DLs

20. NSF Education Innovation (EI) F NSF “Interactive Learning with a Digital Library in Computer Science” (1993-98) F 45 online courses (esp. Internet, IR, MM, Professionalism, overall EI project pages): 100+K accesses/wk F Tools: SWAN (visualization), QUIZIT F Evaluation –traditional –network logging and analysis –tools for visualization

21. Digital Library Courseware F http://ei.cs.vt.edu/~dlib/ F WWW pages or large PDF copy files F Online quizzes based on book by Michael Lesk (Morgan Kaufmann Publishers) F Contents based on book, with several other popular topics added (e.g., agents) F Separate pages to supplement: Definitions, Resources (People, Projects), and References

22. OUTLINE F Challenges to CS F Opportunities for education –CSTC, CRIM F 5S framework F NDLTD case study F Technical progress F DL as industrial success

23. CS -> CSTC -> CRIM F NSF and ACM Education Committee are funding a 2 year project “A Computer Science Teaching Center” - CSTC - http://www.cstc.org/ F College of NJ, U. Ill. Springfield, Virginia Tech F Focus initially on labs, visualization, multimedia F Multimedia part is also supported by “Curriculum Resources in Interactive Multimedia” - CRIM - grant to Virginia Tech and George Washington University: http://www.cstc.org/~crim/

24. CS Teaching Center (CSTC) F Instead of building large, expensive multimedia packages, that become obsolete and are difficult to re-use, concentrate on small knowledge units. F Learners benefit from having well-crafted modules that have been reviewed and tested. F Use digital libraries to build a powerful base of support for learners, upon which a variety of courses, self-study tutorials & reference resources can be built. (See NSF SMETE-Lib Study at http://www.dlib.org/smete/public/smete-public.html )

26. CRIM Rationale F MM field needs properly trained personnel F Support this with resources + curricula F Together these help us move toward a DL for Interactive MM -> CS -> SMETE F Benefits will go to teachers (who have more to build upon) and students (who will have a richer environment for learning

27. Concerns, Problems F Motivating educators to create modules that can be used elsewhere is difficult without a suitable reward structure and an infrastructure of testing, packaging, discovery, reuse, and evaluation. F There is a unnecessary disconnect between researchers (e.g., in laboratories) preparing exciting demonstrations for conferences and instructors interesting in helping students grasp underlying concepts and innovations in their area.

28. Solutions, Plans F CSTC will have a variety of focused centers so that different types of resources can be collected, tested, and suitably packaged: –laboratory exercises, activities, assignments –visualizations and visualization tools –interactive multimedia resources (CRIM) F ACM may launch a digital library “Transactions in Courseware and Education in Computing” to provide an ongoing infrastructure for CSTC.

30. CRIM Project Activities F Workshops, other ways to involve community F WWW site including DL in CSTC re MM –Devised cataloging schema, designed interface –Referring to all MM syllabi and curriculum –Inviting learning resources for the CRIM DL, with reviews, reuse certifications F Publish report on MM curriculum through ACM and IEEE, after careful review

31. Dimensions / Categories (matrix+) F Level: K-12, ugrad (low, upper), grad (MS, PhD), prof. F Length: reference, short course, course unit, course, … F Academic orientation: science, engineering, art, communications, multidisciplinary, marketing F Pedagogical orientation: mm use, survey/hands-on, traditional/constructivist (design, develop) F Tool connection: course on, compare animation tools, assume know or can learn, use tool as example F (Target) audience (and background): culture F Learning style: visual/auditory, indiv/group F Relation: closed/complete, part of some structure (e.g., course, program)

32. Virginia Tech Courses F Art: Digital Art and Design course (Photoshop) F CS: 1604 Introduction to the Internet (1 cr.) F CS: 3604 Professionalism in Computing F CS: 4624 Multimedia, Hypertext and Information Access (3 cr.) F CS: 5604 Information Storage & Retrieval (3 cr.) F CS: 6604 Digital Libraries (3 cr.)

33. CS4624 F Units –Applications&Authoring –Capture&Representation –Compression&Models –Presentation&Interaction –Communication&Networking F Pedagogy –field trips –readings & quizzes; exercises in lab; final –“real” term project in groups

34. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress F DL as industrial success

35. How to Build a Digital Library F Understand the problem (using the 5S Framework) F Solve the problem (using the Star Methodology) –design, develop, evaluate, –refine, operate

36. Definition: Digital Libraries are complex systems that F help satisfy info needs of users (societies) F provide info services (scenarios) F organize info in usable ways (structures) F present info in usable ways (spaces) F communicate info with users (streams)

37. 5S Layers Societies Scenarios Spaces Structures Streams

38. Definition: 5S Framework F Societies: interacting people (, computers) F Scenarios: services, functions, operations, methods F Spaces: domains + constraints (e.g., distance, adjacency): 2D, vector, probability F Structures: relations, trees, nodes and arcs F Streams: sequences of items (text, audio, video, network traffic) F (5 Element System: Fire, Wood, Earth, Metal, Water)

39. 5S: Components F Societies: roles, rituals, reasons, relationships, artifacts F Scenarios: acquire, index, consult, administer, preserve F Spaces: physical, temporal, functional, presentational, conceptual F Structures: architectures, taxonomies, schema, grammars, links, objects F Streams: granularities, protocols, paths, flows, turbulences

40. Star Methodology

41. Neill Kipp Dissertation F Training interested groups about 5S and the Star Methodology, refining the Framework to have solid mathematical foundation F Case studies of projects at Virginia Tech or involving VT staff/students: CSTC, NDLTD, NARA (with SAIC), Lexis,... F Open also to study DL projects elsewhere F Focusing too on the design artifacts developed and related issues of efficient description and representation (esp. with markup, hypermedia)

42. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress F DL as industrial success

43. A Digital Library Case Study F Domain: graduate education, research F Genre:ETDs=electronic theses & dissertations F Submission: http://etd.vt.edu F Collection: http://www.theses.org Project: Networked Digital Library of Theses & Dissertations (NDLTD) http:// www.ndltd.org

44. Media ETD Web Site http://www.ndltd.org/ ETDs Got Your Interest? Graduate Students Singapore AM Chronicle of Higher Ed. National Public Radio NY Times... U. Laval

45. Key Ideas: Networked infrastructure Scalability Education is the rationale University collaboration Workflow, automation Authors must submit Maximal access PDF, SGML, MM Standards Federated search 8th graders vs. grads MARC, DC, URNs

46. Status of the Local Project F Approved by university governance Spring 1996; required starting 1/1/97 F Submission & access software in place F Submission workshops for students (and faculty) occur often: beginner/adv. F Faculty training as part of Faculty Development Initiative F Over 2000 ETDs in collection

47. F Aiding universities to enhance grad educ., publishing and IPR efforts F Helping improve the availability and content of theses and dissertations F Educating ALL future scholars so they can publish electronically and effectively use digital libraries (i.e., are Information Literate and can be more expressive) What are we doing?

48. What are the long term goals? F 400K US students / year getting grad degrees are exposed / involved F 200K/yr rich hypermedia ETDs that may turn into electronic portfolios F Dramatic increase in knowledge sharing: lit. reviews, bibliographies, … F Services providing lifelong access for students: browse, search, prior searches, citation links

49. NDLTD Computer Resources Research Literature Student Prepares Thesis or Dissertation

50. Student Defends and Finalizes ETD My Thesis ETD

51. Student Gets Committee Signatures and Submits ETD Signed Grad School

52. Graduate School Approves ETD Student is Graduated Ph.D.

53. Library Catalogs ETD and New Students Have Access to the New Research WWW NDLTD

54. Institutional Members F Coalition for Networked Information (CNI) F Committee on Inst. Coop. (CIC) F Diplomica.com F Dissertation.com F National Library of Portugal F UNESCO

55. US University Members F U. of Iowa F U. of Maine F U. of Oklahoma F U. of South Florida F U. of Tennessee, Knoxville F U. of Tennessee, Memphis F U. of Texas at Austin F U. of Virginia F U. Wisconsin - Madison F Vanderbilt U. F Virginia Tech - required since 1/97 F West Virginia U. - required beginning fall 1998 F Worcester Polytechnic Inst. F Air University (Alabama) F Cal Tech F Clemson University F College of William & Mary F Concordia University (Illinois) F East Tenn. State University F Florida Institute of Tech. F Florida International University F Michigan Tech F Naval Postgraduate School (CA) F North Carolina State U. F Penn. State University F Rochester Institute of Tech. F U. of Florida F U. of Georgia F University of Hawaii, Manoa

56. Australian Project Members F U. New South Wales (lead institution) F U. of Melbourne F U. of Queensland F U. of Sydney F Australian National University F Curtin U. of Technology F Griffith U.

57. German Project Members F Humboldt University (lead institution) F 3 other universities F 5 learned societies F 1 computing center F 2 major libraries

58. Other International Members F Chinese University of Hong Kong F Chungnam National U., Dept of CS (S. Korea) F City University, London (UK) F Darmstadt U. of Tech. (Germany) F Free University of Berlin (Germany - Vet. Med.) F Gyeongsang National U. (Korea) F India Institute of Technology, Bombay (India) F Nanyang Technological U. (Singapore, part) F National U. of Singapore (Singapore, part) F *National Library of Portugal F Polytechnic University of Valencia (Spain) F Rhodes U. (South Africa) F St. Petersburg St. Tech.U (Russia) F Univ. de las Américas Puebla (Mexico) F U. Laval; U. of Guelph; U. Waterloo; Wilfrid Laurier U. (Canada)

59. NUDL F 1/15/99 NUDL proposal to NSF under DLI2 international program –VT: Library, Grad School, Industrial&Systems Eng. –Partners: UK (2), Singapore, Russia, Korea, Greece, Germany, plus Iberoamerican group (Spain, Portugal, Argentina, Brazil, Chile, Mexico) –Problems: Multilingual search, multimedia submissions, requirements/usability, … F Start with ETDs, then expand to other student works, portfolios, data sets, (CS) courseware,...

61. NUDL Partners F Ricardo A. Baeza-Yates, Universidad de Chile, Chile F José Luis Brinquete Borbinha, Biblioteca Nacional, Portugal F José Hilario Canós Cerdá, Universidad Politécnica de Valencia, Spain F Stavros Christodoulakis, Technical University of Crete, Greece F Lautaro Guerra Genskowsky, Universidad Técnica Federico Santa Maria,Chile F Juan José Goldschtein, Univesidad de Belgrano, Argentina F Peter Diepold, Humboldt University, Germany F Francisco Javier Jaén Martinez, Spain F Sung Hyon Myaeng, Chungnam National University, Korea F Ana Maria Beltran Pavani, Prédio Cardeal Leme, Brazil F Lim Ee Peng, Nanyang Technological University, Singapore F Alexander I. Plemnek, St.-Petersburg State Technical University, Russia F J. Alfredo Sánchez, Universidad de las Américas-Puebla, Mexico

62. Access Statistics 1996 19971998 Total successful requests:37,171247,573628,401 Av. successful requests/day: 102 685 1,690 Requests for.PDF files: 4,600 72, 854343,236 Requests for.HTML file28,225129,831215,896 Distinct hosts served 9,015 22,725 36,724 Total data transferred:3,229M 25,953M 74,051M Av. data transferred/day: 9M 73M 222M

63. Popular Works 1996 458 Seevers, Gary L. Identification of Criteria for Delivery of Theological Education Through Distance Education: An International Delphi Study (Ph.D., Educational Research and Evaluation, April 1993; 1353Kb) 432 Hohauser, Robyn Lisa. The Social Construction of Technology: The Case of LSD (MS in Science and Technology Studies, Feb. 1995; 244Kb) 390 Childress, Vincent William. The Effects of Technology Education, Science, and Mathematics Integration Upon Eighth Grader's Technological Problem-Solving Ability (Ph.D. in Vocational and Technical Education, July 1994; 285Kb) 310 Kuhn, William B. Design of Integrated, Low Power, Radio Receivers in BiCMOS Technologies (Ph.D. in Electrical Engineering, Dec. 1995; 2Mb) 287 Sprague, Milo D. A High Performance DSP Based System Architecture for Motor Drive Control ( MS in Electrical Engineering, May 1993; 878Kb) 165 Wallace, Richard A. Regional Differences in the Treatment of Karl Marx by the Founders of American Academic Sociology (MS in Sociology, Nov. 1993; 479Kb) 150 McKeel, Scott Andrew. Numerical Simulation of the Transition Region in Hypersonic Flow (Ph.D. in Aerospace Engineering, Feb. 1996; 3Mb)

64. Popular Works 1997 9920 Liu, Xiangdong. Analysis and Reduction of Moire Patterns in Scanned Halftone Pictures (Ph.D. in Computer Science, May 1996; 6.6Mb) 7656 Petrus, Paul. Novel Adaptive Array Algorithms and Their Impact on Cellular System Capacity (Ph.D. in Electrical Engineering, March 1997; 5Mb) 2781 Agnes, Gregory Stephen. Performance of Nonlinear Mechanical, Resonant-Shunted Piezoelectric, and Electronic Vibration Absorbers for Multi-Degree-of-Freedom Structures (Ph.D. in Engineering Mechanics, Sept. 1997; ? + 7926Kb) 2492 Gonzalez, Reinaldo J. Raman, Infrared, X-ray, and EELS Studies of Nanophase Titania (Ph.D. in Physics, July 1996; 4607Kb) 1877 Shih, Po-Jen. On-Line Consolidation of Thermoplastic Composites (Ph.D. in Engineering Mechanics, Feb. 1997; 3.3Mb) 1791 Saldanha, Kevin J. Performance Evaluation of DECT in Different Radio Environments (MS in Electrical Engineering, Aug. 1996; 3.2Mb) 1431 DeVaux, David. A Tutorial on Authorware (MS in CS, April 1996; 2.3Mb) 1394 Kuhn, William B. Design of Integrated, Low Power, Radio Receivers in BiCMOS Technologies (Ph.D. in Electrical Engineering, Dec. 1995; 2518Kb)

65. Popular Works 1998 F K-accesses Mbytes Degree Year Dept Tables/Figures Author F 75, 12, PhD, 1997, ME, 38/174, Maillard F 56, 6.5, PhD, 1996, CS, 8/93, Liu F 20, 3.9, PhD, 1997, EE, 9/121, Laster F 15, 4.9, PhD, 1997, CpE, 17/127, Tripathi F 12, 6.6, MS, 1997, EE, 7/96, Nicoloso F 6.7, 4.6, PhD, 1996, Physics, 8/62 (32 color), Gonzalez

66. International Use  199619971998  85029928170 United Kingdom  6082,5014223 Australia  34623787373 Germany  71323673970 Canada  38712642201 South Korea  46311614431 France  2507252553 Italy  1918672781 Netherlands  18311301449 Brazil  229671089 Thailand  839581414 Greece

67. Who are sponsors / cooperators? F Funding, Donations of hardware/software –SURA –US Dept. of Education (FIPSE) –Adobe Systems –IBM –Microsoft –OCLC F Others Serving on Steering Committee –National/Regional Projects: Australia, French speaking group, Germany, IberoAmerica (ISTEC), UK (UTOG) –CGS, National Lib. Canada, NSF, OAS, SOLINET, UMI, UNESCO,...

68. How can a university get involved? F Select planning/implementation team – Graduate School – Library – Computing / Information Technology – Institutional Research / Educ. Tech. F Send us letter, give us contact names F Adapt Virginia Tech solution – Build interest and consensus – Start trial / allow optional submission

69. Build Local ETD Site Digital Library Policies Inspection/Approval Workshop/Training ETD

70. Type 1 Members University Requires ETDs F Adobe Acrobat and/or XML/SGML tools F Automated submission & processing F Archive/access through UMI, (OCLC,) Virginia Tech,... F (Local) WWW site, publicity F (Local) Assistance provided as requested: email, phone, listserv(s)

71. Type 2 Members University Agrees to Require ETDs F Like Type 1 but set date not reached F Usually has an option or pilot F May: wait for new AY; start with all who enter after; … F Build grass roots support – Advisory committee: representative? expert? – Champions to spread by word of mouth – Approval: Senates, Commissions, Deans, Students – Publicity to reach community

72. NDLTD Members, Types 3-7 F 3. Part of university requires ETDs F 4. University allows ETDs F 5. University investigating, has pilot F 6. University consortium joins: –CIC (Big 10 coordinating body) F 7. Non-university organization joins –CNI (Coalition for Networked Info.)

73. Everyone Learns F Students become “info literate” F Students learn about discovery, search, categorization/classification, e-pub, preservation, helping others find/reuse F Campus starts to think about IPR –e.g., Virginia Tech symposium F Faculty and students improve quality as reader base expands

74. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress –NDLTD services F DL as industrial success

75. User Search Support (multilingual, XML) Note: All groups shown are connected with NDLTD.

76. www.theses.org F James Powell student project, D-Lib Magazine description in Sept. 1998 F XML description of each site –type of search engine / service –language –coverage (for resource discovery) F Adding Z39.50 gateway capability

77. Interoperability Testing F IBM DL: donated equipment, technical support, powerful IPR (see TOIS, D-Lib) F Z39.50: OCLC SiteSearch / VT tailored s/w –university libraries w. catalogs of freely shared MARC records pointing to archival copies –via URNs: handles & PURLs F Dienst / NCSTRL - www.ncstrl.org: CS depts., DARPA, NSF, CNRI, Cornell - UVA is working on extensions for ETDs - Portugal is studying use for Europe - VT is working on Dienst to Z39.50 gateway

78. Access Approaches F Goal: Maximize access and services, e.g., by encouraging: F UMI centralized services F Distributed service: Dienst, Z39.50 F Regional services (e.g., OhioLink, AZ/NM) F Local servers with browse, search – From local catalogs to local archives F WWW robot indexing and search services

79. Support Services Developed F WWW site with > 300 Mb, CD, videotape F Automated submission system (MySQL, UNIX, WWW scripts - grad school/library) F Student guidelines, style sheets, multimedia training materials, FAQs, press info F SGML and XML DTDs for ETDs F SGML to HTML (web generator) F LaTeX, Word templates, converters

80. Support Offered F Software, documentation, tech support F Email, listservs (etd-l@listserv.vt.edu, - eval, -grad, -library, -technical) F Donations: Adobe, Microsoft F Evaluation: instruments, analysis http://scholar.lib.vt.edu - solutions/statistics F (Temporary storage / archiving; aid - in setting up an int’l service & archive)

81. Enhancements F Dublin Core spec, MARC crosswalk F DTDs for SGML, XML(+ ML) F Annotation system (author, friends, notes) F Routing system (based on Sift) F Multilingual WWW site, training materials (Spanish recently done in Valencia) F Better federated search (w. Z39.50, planned with Dienst and Harvest - maybe MARIAN)

82. Further Services F Adding services currently prototyped –support with IBM DL, OCLC SiteSearch F Adding other services planned –building and using citation database (w. SFX) –implementing plagiarism check (like “SCAM”) F Developing NUDL as a sustainable self governing global institution (w. committees)

83. Other Work F Working with publishers to increase level of access as much as possible F Interoperability tests among universities and with UMI to provide integrated services F Study with testbed that emerges, to improve information retrieval, browsing, interface, and other types of user support F Evaluation, improving learning experience, spread to worldwide initiative, sustainable support and coordination

84. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress –Networking F DL as industrial success

85. Network Research Group F NSF 3 year grant on WWW logging, characterization, and optimization: Abrams, Fox, Pollard (CNS) F Core member of Web Characterization Activity of World-Wide Web Consortium F Providing DL (with OCLC) to support WCA (at http://www.cs.vt.edu/repository/): –logs –tools –publications

86. NRG Tools WebJamma: Artificial HTTP traffic generator WebWatcher: HTTP traffic monitoring and logging system CLFmunge: Anonymizes common log format HTTPdump: Protocol decode for tcpdump Caching proxy simulator Splus programs Log description and validation interface & routines

87. DL Submission Software F Similar software developed for WCA, CSTC, and NDLTD F CSTC version field-tested to manage papers for ACM Digital Libraries ‘99 F May generalize for –conferences –electronic journal –resource description (e.g., courses, Web content)

88. Dissertations F Abdulla (completed) –collected diversity of Web logs –analyzed EI logs re educational use –Fourier analysis, self similarity F Sulleman (starting) –WCA implementation –dynamic documents u patterns, regularities u classes, templates, OIDs, variable data

89. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress –Interfaces F DL as industrial success

90. Accessibility Activities / Plans F Interface design (simple, 3D, VR) F Usability studies F Generic multi-lingual support F Support for those with disabilities F Hybrid collection (paper, MARC, abstracts, full-text, multimedia) F Disciplinary classifications, tools F Visualization of results, collection

91. SPIRE Visualization

92. CAVE Experiments F Use a familiar metaphor –building / floor / room / shelf / book F Rearrange orderings / shelving –use categories, clustering, ranking –use visualization: colors and gaps –study space mappings: physical, logical F Simplify movement for key tasks

94. ENVISION F NSF “A User-Centered Database from the Computer Science Literature” (1991-93) F Collected bib/typesetter data, converted to SGML F Scanned thousands of page images F MARIAN search engine - can be made available (also applied to the Virginia Tech library catalog) used as part of a prototype object-based DL, with tailored visualization interface (L. Nowell dissertation)

95. Envision Results Window

96. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress –MARIAN F DL as industrial success

97. MARIAN F Multiple Access Retrieval of Information with ANnotations F (Musical: Marian the Librarian …) F Evolved from 1980’s CODER system to a distributed Online Public Access Catalog (OPAC), then DL backend, now becoming a full DL system F From C/C++ to Java by Jianxin Zhao F Future uses: NDLTD, NUDL, PetaPlex

101. MARIAN Layers Database Layer Search Engine Layer User Information Layer User Interface Layer User

102. MARIAN Testing Architecture Load Generator Webgate Java Server C/C++ Server

103. MARIAN Parallelism

104. MARIAN Response Time

105. France Dissertation F Key developer since CODER F Applying computational linguistics efforts with machine readable dictionaries F Applying opportunistic handling of term lists for ranking, usable displays (“to be or not to be, that is the”) F Developing and evaluating variety of interfaces

106. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress –PetaPlex F DL as industrial success

107. PetaPlex F Digital Library Machine (“super” object store) F Parallel computer / storage utility for scale of 1000 to 100,000,000 gigabytes (1 Tbyte - 100 Pbyte) F Knowledge Systems Incorporated is supplying VT-PetaPlex-1 for $250,000 with –high speed backbone connection (OC-12) –2.5 terabytes through 100 “Nanoservers”: –Each = Network connection + IBM 25GB disk + 233 MHz Pentium II + Linux

108. PetaPlex Approach F Extend work on KMS from 1970s F Achieve qualitative improvement in quality of hypertext –sub-second response –with terabyte and petabyte scale stores F Do everything with one seek - through hashing over a very large storage space –support URN access as primitive service –support name / repository model for digital library

109. PetaPlex Complex FRONT END MACHINE RS/6000, 1G RAM, 4 Proc. Nanoserver Service Machine 1 Service Machine 2 Service Machine 3 Service Machine 4

110. PetaPlex Service Machines F Small object server F Large object server –video on demand –streaming audio F Information retrieval server F Proxy / cache server (e.g., 1 terabyte server of 1000 worldwide for Comsat/Intelsat)

111. PetaPlex Top View 4 ft. side

112. PetaPlex Side View 4 ft. wide 8 ft. high Roles: * Support * Cooling * Power many shelves / side

113. PetaPlex Cost Goals, Approach F Maximize number of seeks achievable F Maximize % of cost invested in disks F Maximize flexibility and reliability F Minimize cost per unit of storage F Approach “information utility” F Increase throughput and reliability by replicating on other PetaPlex systems F Use robotics, wireless, and commodity production of nanoservers

114. Sornil & Mather Dissertations F Proposing 50 Tbyte wireless Petaplex for $2M F Mather: efficiently handling very large numbers of objects of varying sizes F Sornil: efficiently handling IR for very large collections, large numbers of users, high transaction rates, large inverted files –modeling and simulation –data organization –parallelization of algorithms, alone and in combination for retrieval (related) tasks

115. OUTLINE F Challenges to CS F Opportunities for education F 5S framework F NDLTD case study F Technical progress F DL as industrial success

116. DL Challenges F Preservation - so people with trust DLs F Affordable storage - so DLs will be universally used F DL industry - critical mass by covering libraries, archives, museums, corporate info, govt info, personal info - “quality WWW” integrating IR, HT, MM,...

117. DLs: Broad Impact F DLs should be in companies F DLs should be in government (integrated) F DLs should be built for all data generated - all types of data, information, and knowledge –covering content/knowledge management –covering data mining, IR, discovery, visualization –promoting specialized work on all types of collections for all types of user groups F 5S framework and university-industry collaboration may help move us to these goals!