A Metadata Catalog Service for Data Intensive Applications Presented by Chin-Yi Tsai

2 Outline  Introduction  The Role of Metadata Services in Grid Data Management  Requirements for the Metadata Service  Components of a Metadata Service  MSC: A Metadata Catalog Service for Grids  Application Experiences  Scalability of the MCS  summary

3 Data-intensive application  Experimental analyses  Simulation in scientific disciplines Massive datasets are shared by a community of hundreds or thousands of researchers  Purpose  To manage these large data sets efficiently  Metadata or descriptive information about the data needs to be managed

4 High Level Diagram of the Metadata Catalog Architecture Client Application Web Server Database Connectivity Metadata Database (MySQL) Standard interface Metadata Catalog Service

5 Introduction  Metadata is information that describes data. Metadata Catalog Service  Design of a Metadata Catalog Service (MCS) that provides a mechanism for storing and accessing descriptive metadata and allows users to query for data items based on desired attribtues.  Accurate identification of desired data items is essential for correct analysis of experimental and simulation results.

6 Introduction (cont’d)  There are various types of metadata.  Replication metadata  Describe the contents of data items  Relate to the physical characteristics of data objects, such as size, access permission.  Distinguish between logical file metadata and physical file metadata. logical file metadataphysical filequery

7 A usage scenario of the Metadata Catalog Service Client Application Physical Storage System Replical Location Service Metadata Catalog Service MCS Web Server MCS Database Replica Index Node Local Replica Cat

8 Metadata types User Metadata Virtual Organization Metadata Domain-Specific Metadata Domain-Independent Metadata Physical Metadata Metadata Types Information about the characteristics of data on physical storage system Regardless of the application domain or virtual organization in which the data sets are created and shared. Specific to an application domain, a virtual organization or to particular user

9 The Role of Metadata Services in Grid Data Management  Medata Services as services that maintain mappings between logical name attributes for data items and other descriptive metadata attributes and respond to queries about those mappings.  Metadata Services play a key role in the publication and the discovery and access of data sets.

10 Publication  Publication is the process by which data sets and their associated attributes are stored and made acessible to a user community.  Domain-independent, domain-dependent, and virtual organization metadata attributes  To discover and access according to attributes  Some members of the community may use the Metadata Service to annotate the data sets with their own observations using user attributes and make these annotations available to a controlled subset of the community.

11 Discovery and Access  Discovery is the process of identifying data items of interest to the user. Client Application Physical Storage System Replical Location Service Metadata Catalog Service MCS Web Server MCS Database Replica Index Node Local Replica Cat

12 Requirements for the Metadata Service  Metadata Service must provide a mechanism for associating logical name attributes with domain- independent metadata attributes.  The Metadata Service must support queries on its contents.  The Metadata Service must implement policies regarding the consistency guarantees, authentication, authorization, and auditing capabilities provided by the service.

13 Requirements for the Metadata Service  The Metadata Service may support the ability to aggregate metadata into collections or views by associating aggregation attributes with logical name attributes.  The Metadata should provide the ability to store attributes that describe the record the transformations ona dataset.  The Metadata Service should provide good performance and scalability.

14 Components of a Metadata Service  A data model that includes mechanisms for aggregation of metadata mappings  A standard schema for domain-independent metadata attributes with extensibility for additional user-defined attributes  A set of standard service behaviors  Query mechanisms for accessing the database  A set of standard interfaces and APIs for storing and accessing metadata  A set of policies for consistency, access control and authorization, and auditing

15 MCS : A Metadata Catalog Service for Grids ( design and implementation )  The MCS data model  MCS Schema  MCS service implementation  MCS Query mechanism and APIs  MCS policies

16 The MCS Data Model  The most basic item in MCS data model is the logical file  Logical collecitons are user-defined aggregations that can consist of zero or more logical file and/or other logical collections.  A logical file may belong to at most one logical collection  Logical views are another type of aggregation that can consist of zero or more logical files, collections and/or other logical veiws.

17 The MCS Data Model Logical file Logical collectionLogical view

18 MCS Schema  Logical file metadata  main attributes of a logical file  Logical collection metadata  user-defined associations of logical files  Logical view metadata  user-defined aggregation of logical files, logical collections or other logical views  Authorization information  is associated the both individual logical files and logical collections  User information  Audit metadata  User-defined metadata  Annotation attributes  Creation history  External catalog metadata

field nametyperemarksdescription Data_idIntegerNon nullThe data identifier Logical_nameVarchar(250)Non nullThe logical file name VersionIntegerThe version of the daat Data_typeVarchar(250)The type of data Collection_idInteger Container_idInteger Container ServiceVarchar(250) Is_validIntegerNon null Creator_DnVarchar(250)Non null Last_Modifier_DnVarchar(250) Create_TimeDate/TimeNon null Last_Modify_TimeDate/Time Master_CopyVarchar(250) Logical file metadata

Logical collection metadata

Logical view metadata

Authorization information

 User information  About writers or modicifers of the logical files in the database  Audit metadata  Record information about actions that can be performed on the Metadata Service  User-defined metadata  Different application domains have their own metadata schemas  Annotation attributes  comments  Creation history  Information about how data items are geneated  External catalog metadata  Use this information to further query the external catalog

24 MCS Service Implementation Application Program Main() { mcsClient( ); mcsCreate( x ); } MCS Client SOAP Engine MCS Server MySQL Database Overview of the Implementation

25 MCS Query Mechanisms and APIs  The client API provides the following operations:  Querying the catalog for logical objects based on object attributes  Querying the static attributes of a logical object  Querying the user defined attributes of a logical object  Querying the contents of a logical view or a logical collection  Creating a logical file, collection or a view  Modifying the attributes of a logical object  Deleting a logical file, view or a collction  Annotating a logical object  Adding logical objects to view

26 MCS Policies  The MCS provides authentication and authorization capabilities on the logical files and logical colleciton attributes in MCS  The MCS provides auditing metadata  Creation information  log  To support other services  Such as replica managers that maintain consistency among data items

27 Application Experiences  To intergrate MCS into the software used by these applications  The Pegasus/LIGO Application  The Earth System Grid Application

28 The Pegasus/LIGO Application  Pegasus is used to map complex application workflows onto the available Grid resources  Pegasus uses MCS to discover existing application data products.  Pegasus uses the MCS and Replication Location Service  MCS only stores logical file names  Attributes that describe these data products, including the type of the data and the duration of data measurements, are stored in the MCS.  23 user defined attributes Client Application Physical Storage System Replical Location Service Metadata Catalog Service MCS Web Server MCS Database Replica Index Node Local Replica Cat

29 The Earth System Grid Application  The MCS is one component in an ESG testbed  ESG scientists use the MCS to discover and query for ESG files based on metadata attributes

30 Scalability of the MCS Database sizeLogical collectionLogical fileUser defined 100, ,000, ,000, Add and query operations

31 Scalability of the MCS With web interface Web service overhead

32 Scalability of the MCS

33 Scalability of the MCS

34 Scalability of the MCS

35 Scalability of the MCS

36 Scalability of the MCS

37 Scalability of the MCS

38 Summary  The design and implementation of a MCS  Store, access, and query  To make the service more extensible and to provdie a more general query model  Use of other database backnd technologies 