Web: Minimal Metadata for Data Services Through DIALOGUE Neil Chue Hong AHM2007

Web: It’s Good To Talk Data Integration Applications: Linking Organisations to Gain Understanding and Experience (DIALOGUE) EPSRC supported sister project network grant running from Stimulate discussion between people involved in data access and integration o

Web: Minimum Requirements for Information Exchange Requirements for agreements so that information can be effectively interchanged between DAI technologies. o Identification of data sources o Description of data sources o Identification of data o Description of data representations What’s the least information I need?

Web: Service Types Core Services / Baseline Services o essential to infrastructure o security, registry, index, discovery Data Services / Data Access Services o exposes a queryable data resource Analytical Services / Data Processing Services / Computational Services o provide operations that act on data Data Transfer Services o provide transfer of data between endpoints Data Storage Services o provide management of data, inc replication

Web: Interoperability Points between Components Compatible naming for services Compatible naming for data objects Managed data transfer between any two endpoints Data formats Data discovery

Web: Searching for data Standard way of accessing a metadata catalogue Standard format for describing a data resource o sufficient to access it access protocols supported description of security policies transfer protocols supported available replicas QoS policies o sufficient to understand what’s in it quality of data provenance o sufficient to choose the right source quality of service –productivity, availability, responsiveness, reliability, accessibility

Web: Core Set of Metadata for all our Data Services Service name Service ID? o Does this evolve too quickly? o Is this only useful for resolution? o Should this be described elsewhere? Service version Service owner Service maintainer Service description: human readable summary Service types implemented Minimal set of annotation on operations to allow discovery Minimal set of management information? Link to service policies (including security, QoS)? Should we use WSDL and WS-MetadataExchange

Web: Extra Metadata for Data Access Services Access protocols supported o query languages supported o result representations supported Description of security policies Transfer protocols supported Available replicas QoS policies Is this DAIS? Extensions for each type: SQL Data Access Services, DICOM, XPath, CQL, etc.

Web: Extra Metadata for Data Transfer OGSA-DMI QoS – what terms o e.g. replica selection for transfers?

Web: Extra Metadata for Data Storage Amazon’s S3 (Simple Storage Service) Data Transport RG o storage operations and parameters SRM

Web: Extra Metadata for Data Transformation Services Schema mapping o store schema maps (e.g. A->B) rather than schema o but no current agreed way of representing a schema map o allow schema maps to be discovered o allow optimisation over maps e.g. A->B->C => A->C if present o Quality and trust of third party schema maps

Web: Core set of Metadata for representing a data object / record Identifier Structure / representational format Provenance data Human readable description Extended sets for o relational data o XML data o file collections Feature based processing, content based processing

Web: Core set of information to store in a provenance record for data integration Who created When created Which service created Service configuration parameters (inc inputs) But how do you evolve this information?

Web: What needs to be agreed What does need to be agreed to make different services interoperate? DIALOGUE allowed discussion amongst software engineers involved in data integration o paper to be published But look at Dublin Core… o … and the many formats for addresses