1 Automated Digital Libraries William Y. Arms Department of Computer Science Cornell University

2 Two Questions

3 Before Digital Libraries Access to scientific, medical, legal information In the United States: -- excellent if you belonged to a rich organization (e.g, a major university) -- very poor otherwise In many countries of the world: -- very poor for everybody

4 Question 1 Must access to scientific and professional information be expensive?

5 Research Libraries are Expensive library materials buildings & facilities staff

6 The Potential of Digital Libraries materials open access computers & networks staff ?

7 Question 2 How effectively can computers be used for the skilled tasks of professional librarianship? -- Time horizon: 5 to 20 years -- All materials in digital form

8 Automated Library Services

9 Skilled Librarianship People are skilled at judgment, understanding, discrimination, etc.: -- selection -- cataloguing, indexing -- seeking for information -- evaluating information -- reference service Can computers provide equivalent services?

10 Equivalent Services Example: Cataloguing rules -- Application of cataloguing rules to monographs is skilled -- It is hard to imagine a computer system with these skills but Catalogs and cataloguing rules are the means not the end

11 Equivalent Services Information discovery Why are web search services the most widely used information discovery tools in universities today?

12 Conventional Criteria Web search services have many weaknesses -- selection is arbitrary -- index records are crude -- no authority control -- duplicate detection is weak -- search precision is deplorable yet they clearly satisfy important requirements...

13 Effectiveness of Web Search Inspec v. Google Google is usually superior for general computing and computer science questions > Broader coverage > Adequate indexing records > Better ranking

14 Simple Algorithms + Immense Computing Power

15 History: Licklider J. C. R. Licklider Libraries of the Future, envisaged digital libraries for scientists at their place of work -- listed desiderata for a digital library -- studied construction of fully automated digital libraries -- put emphasis on artificial intelligence and natural language processing

16 History: Licklider Licklider's predictions for digital libraries were remarkably good, but over optimistic about progress in artificial intelligence -- underestimated what can be done by brute force computing

17 Brute Force Computing Few people can appreciate the power of Moore's Law -- Computing power doubles every 18 months -- Increases 100 times in 10 years -- Increases 10,000 times in 20 years Simple algorithms + immense computing power may outperform human intelligence

18 Brute Force Computing Example Creators of the world champion chess program (Deep Thought later Deep Blue) -- moderate chess players -- simple tree-search algorithm -- very, very fast computer hardware

19 An Anecdote The question (Marvin Minsky) -- How would you design as computer system that can answer questions such as, "Why was the space station a bad idea?"? The answer (Danny Hillis) -- Design much more powerful computers!

20 Examples of Automated Digital Library Services

21 Web Search Brute force indexing and retrieval -- retrieve every page on the web -- index every word -- repeat every month Getting better all the time -- improved algorithms -- faster computers and networks -- analysis of users

22 Web Search Ranking algorithms Closeness of match -- vector space and statistical methods (Salton, et al., c. 1970) Importance of digital object -- Google ranks web pages by how many other pages link to them, gives greater weight to links from higher ranking pages. (NSF/DARPA/NASA Digital Libraries Initiative)

23 Archiving and Preservation Internet Archive -- Monthly, web crawler gathers every open access web page with associated images -- Web pages are preserved for future generations -- Files are available for scholarly research not perfect HTML pages, images; no Java applets, style sheets -- materials are dumped with no organization or indexing -- access for scholars is rudimentary

24 Reference Linking Web of Science (ISI) -- input: combination of automatic means, skilled people -- limited number of journals -- very expensive ResearchIndex (a.k.a. CiteSeer, a.k.a. ScienceIndex) (NEC) -- fully automatic -- all open access material in computer science -- a free service

25 Beyond Text Informedia (Carnegie Mellon) Automatic processing of segments of video, e.g., television news. Algorithms for: -- dividing raw video into discrete items -- generating short summaries -- indexing the sound track using speech recognition -- recognizing faces -- searching using natural language processing (NSF/DARPA/NASA Digital Libraries Initiative)

26 Costs and Benefits

27 Costs of Catalogs and Indexes Catalog, index and abstracting records are very expensive when created by skilled professionals -- only available for certain categories of material (e.g., monographs, scientific journals) -- contain limited fields of information (e.g., no contents page) -- restricted to static information High costs reduce effectiveness and access

28 Costs of Automated Digital Libraries The Google company million searches daily people (half technical, 14 with Ph.D. in computing) -- 2,500 PCs running Linux, with 80 terabytes of disk The Internet Archive -- 7 people with support from Alexa (March 2000)

29 Overall If you are rich Research libraries, using commercial information services, provide excellent service at very high cost to a favored few -- Automated digital libraries are a long way from providing the personal reference service available to a faculty member at a well-endowed university but...

30 The Model T Library The Model T Ford, with mass production, brought car travel to the masses Automated digital libraries, with open access materials, can already provide good service at low cost -- In the future automated digital libraries can bring scientific, scholarly, medical and legal information to everybody at negligible cost

31 A Footnote

32 Library Expertise The future of scientific and professional information is tied to computing, but automated digital libraries need small teams of highly skilled people -- development of automated digital libraries is bypassing libraries (Google, ResearchIndex, Informedia, Internet Archive) The level of computing expertise in U.S. research libraries is depressingly low

33 Further reading William Y. Arms, "Automated digital libraries." To be submitted to D-Lib Magazine, July/August William Y. Arms, "Economic models for open-access publishing." iMP, March

34 Automated Digital Libraries William Y. Arms Department of Computer Science Cornell University