Interoperability, Z39.50 Profiles & Testing William E. Moen School of Library and Information Sciences Texas Center for Digital Knowledge University of North Texas Denton, TX Netspeed 2002 Conference, October 25, 2002 Calgary, Alberta

Moen Netspeed Calgary, Alberta -- October Overview Interoperability Profiles The Bath Profile The U.S. National Profile Beyond profiles Indexing and search functionality Interoperability testing

Moen Netspeed Calgary, Alberta -- October Interoperability Systems and organizations will interoperate! One should actively be engaged in the ongoing process of ensuring that the systems, procedures and culture of an organisation are managed in such a way as to maximise opportunities for exchange and re-use of information, whether internally or externally. Paul Miller, 2000

Moen Netspeed Calgary, Alberta -- October Defining interoperability System-oriented definition The ability of two or more systems or components to exchange information and use the exchanged information without special effort on either system User-oriented definition User’s ability to successfully search and retrieve information in a meaningful way and have confidence in the results The condition achieved when two or more technical systems can exchange information directly in a way that is satisfactory to users of the systems (AAP)

Moen Netspeed Calgary, Alberta -- October Assessing interoperability Binary Interoperable Not interoperable Continuum More or less interoperable Acceptable levels of interoperability

Moen Netspeed Calgary, Alberta -- October Factors affecting interoperability Multiple and disparate systems operating systems, information retrieval systems, etc. Multiple protocols Z39.50, HTTP, SOAP, etc. Multiple data formats, syntax, metadata schemes MARC 21, UNIMARC, XML, / ISBD/AACR2-based, Dublin Core Multiple vocabularies, ontologies, disciplines LCSH, MESH, AAT Multiple languages, Multiple character sets Indexing, word normalization, and word extraction policies

Moen Netspeed Calgary, Alberta -- October Mapping the landscape Networked information retrieval occurs within and across communities Information communities Focal community (e.g., libraries) Extended community (e.g., cultural heritage community) Extra community Knowledge Domains Intra domain Extra domain Costs to achieve interoperability vary

Moen Netspeed Calgary, Alberta -- October Information communities Focal Community (e.g., Libraries) Focal Community (e.g., Archives Focal Community (e.g., Museum) Extended Community (e.g., Cultural Heritage) Focal Community (e.g., Geospatial ) Focal Community (e.g., Geospatial) Focal Community (e.g., Natural History Museums) Extended Community Extra Community

Moen Netspeed Calgary, Alberta -- October Focal community Community agreements exist (e.g., standards, rules, etc.) Interoperability factors reduced Interoperability more easily achieved Libraries as Focal Community  Relative homogeneity of data and systems  Z39.50 widely implemented  Standards-based MARC records  Content and structure prescribed by AACR  Commonly understood access points  Use of controlled vocabularies

Moen Netspeed Calgary, Alberta -- October Threats to Z39.50 interoperability Differences in implementation of the standard Differences in local information retrieval systems Search functionality Indexing policies These threats can be addressed by Z39.50 specifications and configuration Enhancing local information retrieval systems Recommendations for local indexing decisions

Moen Netspeed Calgary, Alberta -- October Virtual Catalog Application

Moen Netspeed Calgary, Alberta -- October Z39.50 Model of Resource Discovery

Moen Netspeed Calgary, Alberta -- October Profiles Z39.50 specifications Complete Z39.50 Specifications Z39.50 Profile Represent community consensus on requirements Identify Z39.50 specifications to support those requirements Aid in purchasing decisions Provide specifications for vendors Profiles are a solution path for improving interoperability

Moen Netspeed Calgary, Alberta -- October Profiles Defines a subset of specifications from one or more standards Goal of profiles is to improve interoperability Profiles are useful for: prescribing how Z39.50 should be used in a particular application environment solving interoperability problems with existing Z39.50 implementations within a community or across two or more communities

Moen Netspeed Calgary, Alberta -- October The Bath Profile The Bath Profile: An International Z39.50 Specification for Library Applications and Resource Discovery, Release 2 (Draft 3,Oct. 2002) Enables effective use of Z39.50 in a range of library applications: Search and retrieval from library catalogues Search and retrieval of bibliographic holdings Search and retrieval of authority records Cross-domain searching FOR MORE INFORMATION, VISIT THE BATH MAINTENANCE AGENCY WEBSITE…

Moen Netspeed Calgary, Alberta -- October Structure of the profile Modular for extensibility Related requirements and specifications group in Functional Areas Release 2 defines four Functional Areas Functional Area A: Basic Bibliographic Search and Retrieval, with Primary Focus on Library Catalogues Functional Area B: Bibliographic Holdings Search and Retrieval Functional Area C: Cross-Domain Search and Retrieval Functional Area D: Authority Record Search and Retrieval in Online Library Catalogues Defines Conformance Levels for each area

Moen Netspeed Calgary, Alberta -- October Addressing interoperability The Bath Profile: Identifies searching requirements (tasks) Defines the searches (semantics and behavior) Specifies Z39.50 query to represent the search Standard combination of Z39.50 attribute types and values Clients must send all attribute type values specified for search Servers must be able to process all values No default behavior by client or server Requires support for specific formats for interchanging retrieval records

Moen Netspeed Calgary, Alberta -- October Functional Area A, Level 0 Conformance Level 0 Version 2 required, Version 3 recommended Basic Bibliographic Search (Z39.50 Search Service) Author Search — Keyword Title Search — Keyword Subject Search — Keyword Any Search — Keyword Basic Bibliographic Retrieval (Z39.50 Present Service) Z-clients to support MARC21 and SUTRS Z-servers to support MARC 21

Moen Netspeed Calgary, Alberta -- October Functional Area A, Level 1 Conformance Level 1 Inherits search requirements form Level 0 Requires 15 additional searches, including: Exact Match (author, title, subject) First Words & First Characters in Field (author, title, subject) Keyword with Right Truncation (author, title, subject) Standard ID, Date, Browse Indexes (Z39.50 Scan Service) 3 scans defined Retrieval Z-clients to support MARC21 and SUTRS Z-servers to support MARC 21

Moen Netspeed Calgary, Alberta -- October Functional Areas B, C, D Area B -- Holdings Information Address the challenge of search and retrieval of bibliographic holdings information Locations Only Locations, Summary Information and Count if available Summary Copy Level Holdings Use of XML as Record Syntax Area C -- Cross Domain Search/Retrieval Defines two conformance levels (13 searches) Dublin Core DTD for XML record syntax Area D – Authority Record Search/Retrieval Defines one conformance level Defines 14 searches

Moen Netspeed Calgary, Alberta -- October Level 0: title keyword search Attribute TypeAttribute ValueAttribute Names Use (1)4Title Relation (2)3Equal Position (3)3Any Structure (4)2Word Truncation (5)100Do not truncate Completeness (6)1Incomplete subfield Uses: Searches for complete word in a title of a resource. Example: Title search for “woman” represented in Z query as: (1,4)(2,3)(3,3)(4,2)(5,100)(6,1) woman

Moen Netspeed Calgary, Alberta -- October Level 0: title keyword right truncation Attribute TypeAttribute ValueAttribute Names Use (1)4Title Relation (2)3Equal Position (3)3Any Structure (4)2Word Truncation (5)1Right Truncation Completeness (6)1Incomplete subfield Uses: Searches for complete word beginning with the specified character string in fields that contain a title of a resource. Example: Title search for woman truncated as “wom” represented in Z query as: (1,4)(2,3)(3,3)(4,2) (5,1) (6,1) wom

Moen Netspeed Calgary, Alberta -- October Level 1: title first words in field Attribute TypeAttribute ValueAttribute Names Use (1)4Title Relation (2)3Equal Position (3)1First in field Structure (4)1Phrase Truncation (5)100Do not truncate Completeness (6)1Incomplete subfield Uses: Searches for complete word(s) in the order specified in fields that contain a title of a resource. The field must begin with the specified character string. This search is useful when the beginning words in a title are known to the user. Example: Title search for “Gone with the” represented in Z query as: (1,4)(2,3) (3,1)(4,1) (5,2)(6,1) gone with the

Moen Netspeed Calgary, Alberta -- October Endorsements of Bath Profile Atlantic Scholarly Information Network CENL Working Group on Technical Standards Czech and Slovak Library Information Network (CASLIN) Committee on Institutional Cooperation (CIC) International Coalition of Library Consortia (ICOLC) Istituto Centrale per il Catalogo Unico delle Biblioteche Italiane e per le Informazioni Bibliografiche (ICCU) M25 Consortium of Higher Education Libraries National Library of Canada OCLC ONE2 SmartLibrary Standing Conference of National and University Libraries (SCONUL) Z Texas Project

Moen Netspeed Calgary, Alberta -- October Bath as foundation profile National, regional, and state profiles based on the Bath Profile ONE-2 Profile DanZIG Profile U.S. National Z39.50 Profile Z Texas Profile

Moen Netspeed Calgary, Alberta -- October Library application profiles The Bath Profile: An International Z39.50 Specification for Library Applications and Resource Discovery U.S. National Z39.50 Profile for Library Applications Z Texas Profile: A Z39.50 Profile for Library Systems Applications in Texas Bath Profile Core Specifications For Global Interoperability Relationship among profiles

Moen Netspeed Calgary, Alberta -- October U.S. National Profile National Information Standards Organization (NISO) standards effort National Profile: Addresses cross-catalog searching and holdings information interchange Bath Profile is foundation for U.S. National Profile Responds to national requirements Work initiated in November 2000 Draft standard ready by end of 2002 FOR MORE INFORMATION, VISIT THE PROJECT WEBSITE…

Moen Netspeed Calgary, Alberta -- October U.S. Profile Functional Area A Conformance Level 0 Version 2 required, Version 3 recommended Basic Bibliographic Search (Z39.50 Search Service) Author Search — Keyword (NISO) Title Search — Keyword (Bath) Subject Search — Keyword (Bath) Any Search — Keyword (Bath) Basic Bibliographic Retrieval (Z39.50 Present Service) MARC 21 supported by Z-client and Z-servers

Moen Netspeed Calgary, Alberta -- October U.S. Profile Functional Area A Conformance Level 1 Version 3 required Inherits search requirements form Level 0 Requires 20 additional searches, including: Exact Match (author, title, subject) First Words & First Characters in Field (author, title, subject) Keyword with Right Truncation (author, title, subject) ISBN, ISSN, Standard ID, Format/Type, Date, Language Browse Indexes (Z39.50 Scan Service) Retrieval Z-clients support MARC 21 Z-servers support MARC 21

Moen Netspeed Calgary, Alberta -- October U.S. Profile Functional Area A Conformance Level 2 38 additional searches, including Key Title, Series Title, Uniform Title, Unanchored phrase searches for Title, Subject, Name, Any Personal Author, Corporate Author, Conference Meeting Notes, other standard number (e.g., LCCN) Pattern searches for one or more controlled vocabularies

Moen Netspeed Calgary, Alberta -- October U.S. Profile Functional Area B Bibliographic Holdings Information Retrieval Use of XML as Record Syntax Z39.50 Holdings XML Schema Harmonized with Bath Profile

Moen Netspeed Calgary, Alberta -- October Z39.50 profiles are not enough Profiles can: Identify searching requirements (tasks) Define the searches (semantics and behavior) Specify Z39.50 query to represent the search and formats of retrieval records Also needed are: Agreements on indexing Common search functionality Methods and testbed for interoperability testing Conformance to profiles by vendors and libraries

Moen Netspeed Calgary, Alberta -- October Indexing & search functionality Indexing Access points Populating indexes from which MARC fields/subfields Moving toward community agreements on common indexing policies to support profile-defined searches Indexing guidelines available for use Related issues: word normalization, word extraction Search functionality Phrase searching Truncation Proximity searching, etc.

Moen Netspeed Calgary, Alberta -- October Interoperability testbed project Realizing the Vision of Networked Access to Library Resources: An Applied Research and Demonstration Project to Establish and Operate a Z39.50 Interoperability Testbed A Institute of Museum and Library Services National Leadership Grant Goal: Improve Z39.50 semantic interoperability among libraries for information access and resource sharing FOR MORE INFORMATION, VISIT THE PROJECT WEBSITE…

Moen Netspeed Calgary, Alberta -- October Z-Interop vision Provide a technically and organizationally trusted environment for vendors and consumers to demonstrate and evaluate Z39.50 products Develop rigorous methodologies, test scenarios & procedures to measure and assess the extent of interoperability Demonstrate and operate a Z39.50 interoperability testbed

Moen Netspeed Calgary, Alberta -- October Z-Interop partners Institute of Museum and Library Services UNT ’ s Texas Center for Digital Knowledge University of North Texas School of Library and Information Sciences OCLC Online Computer Library Center Sirsi Corporation Sea Change Corporation, Bookwhere 2000

Moen Netspeed Calgary, Alberta -- October Components of the testbed Test dataset 400,000 MARC 21 records from OCLC’s WorldCat Z39.50 reference implementations Z-client, Z-server, information retrieval system Test scenarios & searches Searches with known result records from dataset Benchmarks Results of test searches against reference implementations

Moen Netspeed Calgary, Alberta -- October Analysis of test dataset Determine frequency of words in dataset Systematically select words for use in test searches Identify records that contain selected word Aggregate Record Group Word appears in any fields and subfields Identify records that contain selected word in specified fields/subfields Candidate Record Group For example, examine records for occurrence of word in title-related fields/subfields

Moen Netspeed Calgary, Alberta -- October Decomposed MARC records OCLC # Tag1 st Ind 2 nd Ind SubFldFld Pos SubFld Pos Word Pos Word Ocm OCoLC 31102a1111 National 31102a1112 Study 31102a1113 Service a1211 Illegitimacy a1212 and a1213 Adoption b1221 Report 36500a1711 Illegitimacy 36500z1721 Maine 400,000 MARC21 records = 33 million decomposed records

Moen Netspeed Calgary, Alberta -- October Analysis logic Test Dataset (decomposed records) 1. Examine for occurrence of word “river” Candidate Record Group 2. Yields Aggregate Record Group for word “river” 3. Examine for occurrence of word “river” in selected fields/subfields 4. Yields Candidate Record Group for word “river” in selected fields/subfields Aggregate Record Group

Moen Netspeed Calgary, Alberta -- October Some critical questions What is a “word” Self-help Self help Normalization Elena Éléna What are the appropriate Author, Title, and Subject fields to look in for the word? Decision related to indexing policies

Moen Netspeed Calgary, Alberta -- October Reference implementations Online Catalog Software Z-Interop testbed uses SIRSI’s UNICORN system Test dataset loaded on the system Indexing policies based on guidelines Z39.50 Server SIRSI Z39.50 Module Configured according to Bath/U.S. Profile Z39.50 Client Bookwhere 2000 Configured according to Bath/ U.S. Profile

Moen Netspeed Calgary, Alberta -- October Establishing benchmarks Reference Z39.50 Client Reference Z39.50 Server Configured to Support Profile Specifications Indexed per guidelines to support Profile searches Test Dataset Test searches Retrieval Results Candidate Record Group Compared to Benchmarks For Test Search Yields

Moen Netspeed Calgary, Alberta -- October Interoperability testing Z-Interop Interoperability Testing Policies and Procedures Test dataset loaded on participant’s system Configured conform with Bath/U.S. Profiles Indexed according to participant’s policies Testing Z-servers Z-Interop will send test searches from reference Z-client Report results compared with benchmarks Analyze results to assist implementor to improve interop Testing Z-clients Test searches sent to reference Z-server

Moen Netspeed Calgary, Alberta -- October Testing & assessment Reference Z39.50 Client Vendor Z39.50 Server Configured to Support Profile Specifications Configured by Vendor for Conformance to Profile Indexed by Vendor According to Vendor’s Specifications Test Dataset Loaded by Vendor or Library Test Searches Retrieval Results Compared to Benchmarks For Test Search

Moen Netspeed Calgary, Alberta -- October Current testing Validate testing methodologies, procedures, policies Bath/U.S. National Profiles Levels 0 & 1 Search & Retrieval Title Search – Keyword Author Search – Keyword Subject Search – Keyword Any Search – Keyword Title, Author, Subject Searches – Keyword Right Truncation Simple Keyword Boolean searches (AND, OR, NOT) Test participants InQuirion OCLC Innovative Interfaces TLC/CARL epixtech Fretwell-Dowing M 25 (UK) Others expressing interest

Moen Netspeed Calgary, Alberta -- October Research questions What are acceptable levels of interoperability? What are appropriate measures of interoperability? What does conformance to a Profile mean? Conformance of vendor’s product Conformance of your implementation of vendor’s product To what extent are organizations willing to support common indexing practices to improve interoperability?

Moen Netspeed Calgary, Alberta -- October Critical success factors Openness and transparency of processes Project documents available on website Culture of nurturing improvement Trustworthiness Confidentiality of participants’ results

Moen Netspeed Calgary, Alberta -- October An opportunity for Z39.50 Z39.50 experience has shown the challenges of interoperability Problems of interoperability are better understood within a focal community Solution paths exist Interoperability testing serves as platform for improvement The pieces are finally falling into place!

Moen Netspeed Calgary, Alberta -- October References The Bath Profile Maintenance Agency  U.S. National Profile  Z39.50 Interoperability Testbed 