Working out the Future of Library Resource Discovery Marshall Breeding Independent Consultant, Founder and Publisher, Library Technology Guides October 5, 2015 NISO Event: Future of Library Resource Discovery /

Description Marshall Breeding will highlight some of the key findings of the white paper he developed for the NISO Discovery to Delivery topic committee. The presentation will include some updated information on the state of the current arena of commercial and open source discovery services, including trends in adoption and new technical and functional capabilities. Looking forward, Breeding will mention some longer-term possibilities and opportunities for discovery services to move beyond the current models of centralized indexes, including greater reliance on semantic technologies and linked data.

Library Discovery Past

Bound Catalog National Library of Colombia

Card Catalog National Library of Argentina

Online Catalog  Books, Journals, and Media at the Title Level  Not in scope:  Articles  Book Chapters  Digital objects Scope of Search Search: Search Results ILS Data

NOTIS: MDAS  Multiple Database Access System  Released in 1989  Article-level indexing (Mostly Wilson Databases)  Grant supported by Pew Charitable Trusts  Development Partners: NOTIS and Vanderbilt University See: Steffey, RJ. “NOTIS multiple database access system: a look behind the scenes” Online, v14 n5 p46-49 Sep 1990

Next-gen Catalogs or Discovery Interface  Single search box  Query tools  Did you mean  Type-ahead  Relevance ranked results  Faceted navigation  Enhanced visual displays  Cover art  Summaries, reviews,  Recommendation services  Books, Journals, and Media at the Title Level  Other local and open access content  Not in scope:  Articles  Book Chapters  Digital objects  Scope of Search

Discovery Interface search model Search: Digital Collections ProQuest EBSCOhost … MLA Bibliography ABC-CLIO Search Results Real-time query and responses ILS Data Local Index MetaSearch Engine

Web-scale Index-based Discovery Search: Digital Collections Web Site Content Institutional Repositories … E-Journals Reference Sources Search Results Pre-built harvesting and indexing Consolidated Index ILS Data Aggregated Content packages (2009- present) Usage- generated Data Customer Profile Open Access

Evaluating the Performance of Index- based Discovery Services  Intense competition: how well the index covers the body of scholarly content stands as a key differentiator  Difficult to evaluate based on numbers of items indexed alone.  Important to ascertain how your library’s content packages are represented by the discovery service.  Important to know what items are indexed by citation, which are full text, and how A&I content is handled

Open Discovery Initiative LibrariesPublishersService Providers 14 Marshall Breeding, Vanderbilt University Jamene Brooks-Kieffer, Kansas State University Laura Morse, Harvard University Ken Varnum, University of Michigan Sara Brownmiller, University of Oregon Lucy Harrison, College Center for Library Automation (D2D liaison/observer) Michele Newberry Lettie Conrad, SAGE Publications Roger Schonfeld, ITHAKA/JSTOR/Portico Jeff Lang, Thomson Reuters Linda Beebe, American Psychological Assoc Aaron Wood, Alexander Street Press Jenny Walker, Ex Libris Group John Law, Serials Solutions Michael Gorrell, EBSCO Information Services David Lindahl, University of Rochester (XC) Jeff Penka, OCLC (D2D liaison/observer)

The Context for ODI  Based on a meeting at ALA Annual Conference in New Orleans on Sunday, June 26, Recognition of the following trends and issues:  Emergence of Library Discovery Services solutions Based on index of a wide range of content Commercial and open access Primary journal literature, e-books, and more  Adopted by thousands of libraries around the world, and impact millions of users  Agreements between content providers and discovery providers ad-hoc, not representative of all content, and opaque to customers. 15

ODI deliverables  Standard vocabulary  NISO Recommended Practice:  Data format & transfer  Communicating content rights  Levels of indexing, content availability  Linking to content  Usage statistics  Evaluate compliance  Inform and Promote Adoption 16

ODI Recommended Practices  Published June 25, 2014  NISO RP    Metadata elements for content providers to contribute to discovery service providers  Content providers disclose extent to which they participate with each discovery service  Discovery Service providers disclose what content is represented in index  Discovery services disclose any bias in search results or relevancy relative to business relationships  Discovery services provide use statistics

NISO Discovery White Paper  Commissioned by NISO Discovery to Delivery Topic Committee  First Draft Nov 2014  Revised based on feedback from D2D  Published Feb 20, 2015  Launched at ER&L

NISO Discovery Paper Outline  General Background  Integration between Discovery Services and Management Systems  Linked Data  Gap Analysis  Opportunities for Future Enhancements in discovery  Discovery Beyond Library-provided Interfaces  Open Discovery Initiative: recommendations for Phase II  Longer term prospects

Library Discovery Present

Library Perspective  Strategic investments in subscriptions  Strategic investments in Discovery Solutions to provide access to their collections  Expect comprehensive representation of resources in discovery indexes  Problem with access to resources not represented in index  Encourage all publishers to participate and to lower thresholds of technical involvement and clarify the business rules associated with involvement  Need to be able to evaluate the coverage and performance of competing index-based discovery products

Value and Economy  Academic and research libraries spend far more of their budgets on content than resource management or discovery technologies  Discovery represents essential infrastructure to maximize impact of library collections  Resource management represents essential infrastructure to assemble and assess optimal collection to support library mission  Ever increasing costs of content exert pressure on budgets and demand more effective discovery and more efficient management

Role of the library in discovery  Acquisition and Management of resources  Integrate content into campus enterprise infrastructure and information architecture  Provide general and specialized interfaces  Participate in production and publication  Participate more deeply in research process  Manage content on behalf on the institution in ways that optimize access and discovery.

Web-scale Index-based Discovery Search: Digital Collections Web Site Content Institutional Repositories … E-Journals Reference Sources Search Results Pre-built harvesting and indexing Consolidated Index ILS Data Aggregated Content packages (2009- present) Usage- generated Data Customer Profile Open Access

Bento Box Discovery Model Search: Digital Collections Web Site Content Institutional Repositories E-Journals Search Results Pre-built harvesting and indexing Consolidated Index ILS Data Aggregated Content packages Open Access VuFind / Blacklight

State of Discovery indexes  Very strong coverage of primary publishers of scholarly materials  Especially English and other Western Languages  Weaker coverage of scholarly content in other international regions  Asian languages, Arabic, etc.  Mixed coverage of A&I resources  Mixed converge of non-textual resources

Some Key Areas for Publishers 1. Expose content appropriately 2. Trust that access to material will be controlled consistent with subscription terms 3. “Fair” Linking 4. Materials not disadvantaged or underrepresented in library discovery implementations 5. Usage reporting

Representation of A&I  Important to understand how a discovery service incorporates A&I resources  Does it receive content from the A&I provider directly and make use of value-added terminology  If not: citations or full-text indexing of some portion of the titles represented in the A&I product  NOT the same, and possibly misleading 28

A&I Content in Discovery Services  What is the place for A&I services in the discovery ecosystem  Are there technology solutions capable of substituting for A&I content?  Specialized and scoped search methodologies  Clustering, term extraction, etc.?  Specialized vocabulary and other metadata make positive contributions to the discovery process  Researchers value A&I tools

ODI Standing Committee LibrariesPublishersService Providers 30 Marshall Breeding, Independent Consultant Laura Morse, Harvard University Jason Price, SCELC Ken Varnum, University of Michigan Dave Whisenant, Florida Virtual Campus Lettie Conrad, SAGE Publications Michael McFarland, CredoReference Jill O’Neill, NFAIS Elise Sassone, Springer Aaron Wood, Ingram Content Group Julie Zhu, IEEE Scott Bernier, EBSCO Information Services Steven Guttman, Proquest Rachel Kessler, Ex LIbris John McCullough, OCLC

ODI Standing Committee The Open Discovery Initiative Standing Committee was formed following approval of the Recommended Practice published by NISO on June 25, 2014 We are charged with the following tasks: Promotion and education of ODI Recommended Practice for all stakeholders Provide support for content providers and discovery service providers during adoption and completion of conformance checklists Provide a forum for ongoing discussion related to all aspects of discovery platforms for all stakeholders Consider next steps for items deemed out scope from the original ODI Work Group Recommended Practice Identify emerging needs in the open discovery space and determine appropriate courses of action Make recommendations to the D2D topic committee on further work items required to fulfill the goals of the Open Discovery Initiative 31

Current issues and areas of development

Challenge for Relevancy  Technically feasible to index hundreds of millions or billions of records through Lucene or SOLR  Difficult to order records in ways that make sense  Expectation that relevancy be neutral relative to content source or publisher  Many fairly equivalent candidates returned for any given query  Must rely on use-based and social factors to improve relevancy rankings

Relevancy  Ever-improving, yet flaws remain  Increased use of use data and personalize context to identify and order search results  State of the art improving via more sophisticated search and retrieval technology, increased use of aggregated contextualized data, and other factors

Socially-powered discovery  Leverage use data to increase effectiveness of discovery  Usage data can identify important or popular materials to inform relevancy engines  Identify related materials that may not otherwise be uncovered through keyword matching  Be careful to avoid introducing bias loops

Externalizing functionality  Provide tools and widgets in course management platforms  Reading list management  Improving presentation via mobile devices

Open access content  Only a minority of scholarly resources available through open access licenses  Difficult to identify open access versions available  Often presented proprietary content when open access is also available

Interoperability of Discovery Services and Management Platforms  Discovery and Management solutions offered as matched sets  Ex Libris: Primo / Alma  ProQuest: Summon / Intota  OCLC: WorldCat Discovery Service / WorldShare Platform  Independent Discovery and Management  Kuali OLE: no discovery component  EBSCO Discovery Service: Works with any Resource management system  Both product categories depend on an ecosystem of interrelated knowledge bases  API’s exposed to mix and match, but are efficiencies and synergies are lost?  Recommendation to explore expectation regarding interoperability between these two product categories

Discovery Service Installations Product Installed EBSCO EDS Primo AquaBrowser Encore BiblioCommons41~200 Summon Enterprise Infor Iguana 1874 Axiell Arena

Gap Analysis  Many resources still not addressed in central indexes  Especially A&I products  Better coverage of open access materials  Better support for internationalization and multilingual search and retrieval  Improved capabilities for precise search, known items, browsing  Improved and more transparent relevancy rankings  Non-textual content and retrieval mechanisms  Better integration with learning management systems

Opportunities for Enhancements in Discovery  Improved delivery of APIs  More coherent ecosystem of APIs among discovery services and with resource management systems  Social features and scholarly collaboration  Address research data  Special Collections and archival materials: hierarchical discovery and browsing  Expanded Analytics and Altmetrics

Beyond Index-based Discovery Library Discovery Future

The future of Resource Discovery  More comprehensive discovery indexes  Stronger technologies for search and retrieval  Discovery beyond library-provided interfaces  Linked Data to supplement discovery indexes

Universal participation  Barriers to participation soften as mutual interest prevails over competitive conditions  Advantage to content providers to maximize exposure of resources  Discovery providers gain value in functionality as metadata becomes increasingly commoditized  Essential to preserve value of indexing and abstracting services  Content providers see discovery as a essential channel for distribution

More Distributed Discovery  Address the reality that discovery takes place outside of library provided interfaces  Optimized exposure in the ecosystem of search engine and social network  Not Concentrated on the Library web site  Expression of discovery services via other campus tools and portals and beyond

Multi-layered discovery  Native interfaces of specialized content services  Disciplinary aggregations  General library discovery tools  Global Internet-based discovery

Discovery beyond Library Interfaces  Improved performance of library content through Google Scholar  Same expectations for transparency?  Better exposure of library-oriented content  Schema.org or other microdata formats  Better exposure of scholarly resources  Open access & Proprietary  Embedded tools in other campus interfaces

Part of the General Internet Infrastructure  Scholarly content will be promoted via similar mechanisms as commercial content  Additional levels of infrastructure to protect privacy  Resource management and/or discovery tools expose content items as open linked data

Library opts out of Discovery  Utrecht University Library  Decision to not implement a discovery service but to rely entirely on Google Scholar and other general and scholarly search engines  literature/searching-for-articles-books-theses literature/searching-for-articles-books-theses Kortekaas, Simone. “Thinking the unthinkable: a library without a catalogue — Reconsidering the future of discovery tools for Utrecht University library.” LIBER General Annual Conference 2012

Linked Data  Major trend toward information systems based on linked data  Many projects now based on linked data  Area of peak interest for Library of Congress, OCLC, etc  BIBFRAME  Potential to transform how libraries approach discovery  Likely interim hybrid models: central indexes + Linked Data  Current opportunities in making library content more discoverable

Library adoption of Linked data architecture  Not yet a fully operational method for library- oriented content  Increasing representation of bibliographic resources  BIBFRAME stands to make great impact  Universe of scholarly resources not well represented  Will current expectations for content providers to make metadata or full text available for discovery expand to exposure as open linked data?

Hybrid models  Can index-based search tools be improved through Linked Data  Browse to related resources  Add additional hierarchies of structure to search results

Will linked data models prevail?  Possibility that open linked data may eventually supplant index-based products?  Index technology supplements fundamental architecture based on linked data

Possibilities for Open Access discovery index  Open source tools exist for discovery Interfaces:  VuFind  Blacklight  No open access discovery indexes  High threshold of expense and difficulty to build index  Platform costs  Software development  Publisher relations  Billions of content items to index and maintain

Current model requires massive resources  Threshold of resources required currently too high for open access central discovery index  Assessment might change if options narrowed  Opportunities to lower barriers to entry?  More open model more likely to come through linked data discovery model

Commoditization of Central Indexes  Knowledgebases of e-resource coverage commoditized via KBART and other factors  Central index content likewise will eventually become commoditized  Limited number of discovery service platforms?  Value found in the synergies between library resource management and optimized discovery and delivery

Value in open scholarship  Hopefully the future will be based on open access to scholarly research  Mandates from funding organizations will transform scholarly communications  Current discovery models based on preponderance of proprietary content  Future discovery must assume dominance of open access publishing and underlying data sets

Future of discovery service products  Remain one of the essential components of library technology infrastructure  Loosely or tightly tied to resource management  Increased sophistication in direct discovery and delivery functionality  Increased expectation to syndicate content to local and global discovery context  Investments made in creation of discovery service platforms will provide leverage into each next phase of scholarly information infrastructure  Scholarly publishing arena may change dramatically in next decade.

Open Discovery Initiative: recommendations for Phase II  Address A&I concerns to improve participation  Data exchange mechanisms: metadata + content  Lower threshold of participation  Interoperability with resource management systems

Potential Opportunities for NISO  Convene a second phase of the Open Discovery Initiative  Launch research project on open linked data in scholarly publishing sector to facilitate new models of discovery and access  Expand scope of Altmetrics group to address their integration in discovery service ecosystem  Possible new workgroup to explore recommended practices for improving discoverability of resources via open linked data, schema.org, and other mechanisms.

Longer term prospects  Opportunities for discovery directly tied to realities in scholarly publishing  Dominance of proprietary publishing requires index-based discovery  Future to open access and exposure as open linked data will enable additional models of discovery

An ongoing conversation  Now in a critical point for discovery  Current products evolve  Reaching limits of the prevailing architecture?  Current set of products and services an interim step  Important for stakeholders to engage in defining the future of library resource discovery  Future products must address expected changes in scholarly publishing, library priorities, and institutional strategies.

Questions and discussion