MCAT: A Metadata Catalog San Diego Supercomputing Center Part of the Storage Resource Broker (SRB)

Overview u What is metadata u MCAT architecture u List of (many!) MCAT attributes u MAPS

Elements of Data Intensive computing environments u Resources –Hardware: computing platforms, networks, storage systems –Software: DBs, file systems, operating systems, schedulers, applications u Methods –Access methods, APIs, data access and conversion u Data objects –Data sets and collections of data sets u Users and groups –Who is allowed to create/update/access resources, methods and data sets

Elements of MCAT u MAPS initialization (Metadata Attribute Presentation Structure) u Schema initialization u MAPS to schema converter to dynamic query generator u DB2 and Oracle Query systems u Answer extractor u Convert back to MAPS format

Metadata Stored in MCAT u Metadata = information about data objects –Describes properties and attributes of objects Examples 1. Identifier (internal, not seen by user) 2. Name 3. Types and formats 4. Size

MCAT attributes (cont.) 5. Comments 6. Liveness: (I.e., current state) deleted or exists or locked or under construction 7. Replica-number uSRB supports cloning of data uAn object may have many clones uSRB controls replica selection 8. Creation time-stamp 9. Creation-owner

MCAT attributes (cont.) 10. Collection name uEvery data object must be associated with a collection uA collection contains data objects and other collections (I.e., sub-collections) uObjects may only belong to one collection 11. Physical resource where object is located 12. Location inside the resource (e.g., a directory on a file system

MCAT Attributes (cont) 13. Access control list (ACL) uEntry is: uEach user is given one permission per data object uEach permissionID has an associated list of actions that are permitted uRead uWrite uControl ugrantTicket

MCAT attributes (cont.) 14. Audit record u uEach action on a data object can be audited uAction success or failure noted in audit trail

MCAT attributes (cont.) 15. Ticket uProvides holder with an action permit on the data object uCurrently only read uTicket-giver can impose restrictions: who can use it, when, how many times it can be used u

Attributes not yet supported u Partitioning of data objects u Versioning u Lineage (of data objects and methods) u Derivatives u Locking u Public and private keys on data objects or collections u Summaries or aggregations u Measurements

Resource-related metadata 1. Name 2. Type 3. Access address 4. Default location template (URL??) 5. Replica-numberA: copies of the same resource, any of copies are equivalent 6. Comments

Replicated resource concept u Logical resource u Formed as set of physical, possibly heterogeneous resources u Create a data object on a replicated resource: –object automatically replicated in each one of the component resources u Provides fault tolerance u Other logical resources: striped resources (round-robin), write-once resources, read- only resources

User-related attributes 1. Name 2. Type (privileged, normal, projects) 3. Address Phone 6. Pass phrase 7. Domain: e.g., ucsd, sdsc, caltech 8. User-groups: provides group ID and access control

Data Models and Data Exchange u Data models: standards for structuring information (e.g., Dublin core) u Data exchange formats: standard means to communicate metadata (e.g., XML) u MCAT uses its own data model and exchange format: MAPS –MAPS = metadata attribute presentation structure u Working on mappings to other formats

MAPS u MAPS query format derived from SQL –Large metadata catalogs require database systems –Metadata are normally given as attribute- value pairs whose search can easily be translated into SQL-like queries