1. IVOA Interoperability Meeting - Boston
IVOA Registry Working Group
OAI-based Harvesting
Ray Plante
IVOA Interoperability Meeting - Boston
26 May 2004

2. IVOA Interoperability Meeting - Boston
Harvesting in the Registry Framework
Full
Searchable
Registry VO
Projects
harvest
(pull)
Local
Publishing
Registry
Full
Searchable
Registry
Data
Centers
Local
Publishing
Registry
Local
Searchable
Registry
Harvesting is
about publishing
Specialized
Portals & Services
IVOA Interoperability Meeting - Boston
26 May 2004

3. IVOA Interoperability Meeting - Boston
Harvesting in the Registry Framework
Full
Searchable
Registry VO
Projects
harvest
(pull)
replicate
Local
Publishing
Registry
Full
Searchable
Registry
Data
Centers
Local
Publishing
Registry
Local
Searchable
Registry
Harvesting is
about publishing
Specialized
Portals & Services
IVOA Interoperability Meeting - Boston
26 May 2004

4. IVOA Interoperability Meeting - Boston
Harvesting in the Registry Framework
Full
Searchable
Registry VO
Projects
harvest
(pull)
replicate
Local
Publishing
Registry
Full
Searchable
Registry
Data
Centers
selective
harvesting
Local
Publishing
Registry
Local
Searchable
Registry
Harvesting is
about publishing
Specialized
Portals & Services
IVOA Interoperability Meeting - Boston
26 May 2004

5. IVOA Interoperability Meeting - Boston
Searching in the Registry Framework
Full
Searchable
Registry VO
Projects
Local
Publishing
Registry
Full
Searchable
Registry
Data
Centers
Local
Publishing
Registry
search
queries
Local
Searchable
Registry
Harvesting is not
about searching
Client
Applications
Specialized
Portals & Services
IVOA Interoperability Meeting - Boston
26 May 2004

6. Open Archives Initiative (OAI) Protocol for Metadata Harvesting
Existing standard for harvesting resource descriptions widely supported in digital library community
Supports aggregation of resource descriptions
Deployed successfully as part of NVO registry prototype
Currently part of framework supporting the NVO Data Inventory Service (DIS)
IVOA Interoperability Meeting - Boston
26 May 2004

7. IVOA Interoperability Meeting - Boston
OAI-PMH Features
Defines 6 operations:
Identify GetRecord
ListIdentifiers ListMetadataFormats
ListRecords ListSets
Features
Support for multiple description formats (“metadataPrefix”)
Harvesting by date (“from”, “until”)
Harvesting by category (“set”)
Marking records as deleted
Support for resumption tokens
IVOA Interoperability Meeting - Boston
26 May 2004

8. IVOA Interoperability Meeting - Boston
OAI as a Web Service
IVOA: presumed preference for Web Services
OAI PMH defined as set of HTTP Get services
Broad interest in seeing Web Service version
Evolving the standard toward WS:
NVO has prototyped WS versions
Gretchen Greene & Wil O’Mullane (STScI)
Charlie Cowert (SDSC)
Ray Plante (NCSA)
In contact with OAI community
Opportunity: present proposal for standard WS version to OAI community
IVOA Interoperability Meeting - Boston
26 May 2004

9. IVOA Interoperability Meeting - Boston
Reasons to adopt OAI
Existing, well-tested standard we don’t have to reinvent
Easy to implement
Demonstrated by NVO
Lots of existing OAI software tools
For clients and servers
Lowers cost of implementation
Interoperability with larger digital library community
Do these hold in the web service context?
Yes, if the WS version leverages the original OAI schema
IVOA Interoperability Meeting - Boston
26 May 2004

10. A Harvesting Standard based on OAI
Spelled out in alternate section 4 to the Registry Interface working draft
The OAI standard defined by…
The (existing) OAI-PMH v2.0 specification
Operations, behavior, message schema
OAI-PMH schema defines “envelope” for resource description
The (proposed) OAI WSDL interface
Mapping to WS interface
Imports the OAI-PMH schema
Standard IVOA use of OAI
OAI spec provides hooks for community-specific semantics
Resource description format
Sets
IVOA Interoperability Meeting - Boston
26 May 2004

11. IVOA Interoperability Meeting - Boston
IVOA Metadata Format
Define metadata format: “ivo_vor”
Description using the VOResource schema
Restricted to a resource sub-type defined in a “standard” extension schema
Today: one of the working draft extensions
Non-standard extensions should accessible via a non-standard metadata format name
Harvester can expect to fully comprehend the resource description
Dublin core format, “oai_dc”, required
For cross-disciplinary interoperability
Support is trivial via standard XSL stylesheet
IVOA Interoperability Meeting - Boston
26 May 2004

12. Sets: Named Categories of records
The OAI notion of “sets”
Each record may belong to zero or more named categories called “sets”
Sets may be defined by a community or the individual provider
Enables selective harvesting by category
Proposed use of sets for IVOA:
Implicit definition of a set for each standard resource sub-type
E.g. Organisation, Registry, SimpleImageAcess, etc.
Set name of the form “ivo_type”: e.g. “ivo_Registry”
Allows harvesting of specific types
Explicit definition of special sets:
ivo_managed: those records with authority ID that originates with that registry
ivo_standard: any record of a standard resource sub-type
Full registry replication by omitting set argument
IVOA Interoperability Meeting - Boston
26 May 2004

13. Other miscellaneous specifications
Required resource records:
One Registry record describing the registry itself
One Authority record for each AuthorityID it manages
One Organisation record for each publisher that registers an AuthorityID
The “Identify” operation response must include the registry record for the registry
e.g. <resource xsi:type=“Registry”>
IVOA Interoperability Meeting - Boston
26 May 2004

14. Related Harvester Interface
Additional standard operation to be supported by searchable registry (i.e. the harvester)
“Harvest Me”: a mechanism to tell a harvester that an update is available
Inputs:
ivo-id: the ID of the harvestable registry
harvestingType: HTTP Get or WS version
Should we allow either?
baseURL: endpoint for harvesting interface
lastUpdate: date of most recent update to registry contents
Harvester may choose when/if to harvest
IVOA Interoperability Meeting - Boston
26 May 2004

15. IVOA Interoperability Meeting - Boston
Conclusions
Reasons to adopt OAI-based harvesting
Existing, well-tested standard we don’t have to reinvent
Easy to implement
Lots of existing OAI software tools
Interoperability with larger digital library community
OAI fulfills all of the harvesting functionality set proposed in current WD
Proposal presumes a WS interface desired
Opportunity to contribute to DL community
Action:
Endorse preference to support standard if …
Meets needs
Has favorable cost/benefit ratio
Continue to develop within context of Registry Interface WD
Projects should study OAI spec carefully
If doesn’t meet above criteria, enumerate how
IVOA Interoperability Meeting - Boston
26 May 2004

16. IVOA Interoperability Meeting - Boston
Myths about OAI
OAI does more than we need, too heavy
If you do this right, you will reinvent most/all of the required functionality
It’s cheaper to do something simpler
It’s cheaper to adopt standard if you can leverage existing software
OAI+WS: aren’t these envelopes redundant?
The two envelopes serve different functions
OAI envelope: managing a set of related operations in a format-independent way
Response management Record management
responseDate identifier
request inputs datestamp
set membership
Envelopes are easily skipped
IVOA Interoperability Meeting - Boston
26 May 2004