Sponsored by the National Science Foundation 1 Nov 4, 2010 Inst & Meas WG Meeting at GEC9 Thur, Nov 4, 9am – 10:30am Introductions (9am) Topic 2: Meas Data Archive (MDA) service (9:05am) –Harry Mussman (GPO) and Larry Lannom (CNRI) Topic 5: Interfaces and APIs to manage I&M services (9:30am) –From Scalable Sensing Service, Sonia Fahmy (Purdue) –From OMF/OML, Jolyon White (NICTA) Topic 6: Techniques for measurement data flows (9:50am) –OML, Jolyon White (NICTA) –IPFIX, Tanja Zseby (Fraunhofer Institute FOKUS, Berlin) –NetCDF, Mike Zink (UMass Amherst) Next steps in defining I&M Arch (10:20am) –Approach in Europe, Tanja Zseby (Fraunhofer Institute FOKUS, Berlin) –Open Discussion Adjourn (10:30am)

Sponsored by the National Science Foundation 2 Nov 4, 2010 GENI I&M Architecture: Services and Interfaces

Sponsored by the National Science Foundation 3 Nov 4, 2010 Meas Data Archive (MDA) Srvc Identified in GENI Inst & Meas Architecture as a core service. Several tools/projects have referenced such a service, but none have implemented one. Definition/conceptual design effort started after I&M workshop in June, 2010, by: –Evan Zhang (now UMass Lowell) –Harry Mussman (GPO) –Larry Lannom, Giridhar Manepalli, Jim French and Christophe Blanchi (CNRI) –v1.0 of definition/conceptual design now complete and ready for review by WG Prototype of MDA service: –Will be built by CNRI during Spiral 3 –Who will use it by end of Spiral 3? Next steps: –Limited discussion today. Comments? Questions? –Additional reviews? when? how?

Sponsored by the National Science Foundation 4 Nov 4, 2010 Meas Data Archive (MDA) Srvc Initial View

Sponsored by the National Science Foundation 5 Nov 4, 2010 Meas Data Archive (MDA) Srvc Current View (v1.0)

Sponsored by the National Science Foundation 6 Nov 4, 2010 Meas Data Archive (MDA) Srvc Overview (1.0) The MDA srvc allows users to store, retrieve, browse, search, share and archive measurement data files, including their associated metadata. –The MDA srvc is not limited to measurement data, and can be used for other types of files needed by a researcher. The MDA srvc utilizes the mechanisms provided by the CF to authenticate and authorize users. –Assume: CF drops public keys of authorized users into MDA srvc; presence of key indicates an “account” on the MDA srvc. The MDA srvc is implemented by a User Workspace (UW) Srvc, which in turn utilizes an Digital Object Archive (DOA) Srvc.

Sponsored by the National Science Foundation 7 Nov 4, 2010 User Workspace (UW) Srvc (v1.0)

Sponsored by the National Science Foundation 8 Nov 4, 2010 User Workspace (UW) Srvc Overview (v1.0) The UW srvc utilizes a Linux server and file system structure so that the user can easily transfer files to (and from) a familiar Linux home directory structure –Home directory structure has been setup by and for the user. The UW srvc is expected to accumulate a large number (e.g., thousands) of files for a user, which from time to time can be discarded. –For example, It might accumulate directories of multiple files for each of 29 runs of a particular experiment. Eventually, run 29 is certified as the “golden run”, and all others can be discarded. A UW srvc is expected to be shared by a natural group(s), so that users in the group can easily share files. –Such a group might be a campus, project or slice.

Sponsored by the National Science Foundation 9 Nov 4, 2010 Digital Object Archive (DOA) Srvc (v1.0)

Sponsored by the National Science Foundation 10 Nov 4, 2010 Digital Object Archive (DOA) Srvc Overview (v1.0) The DOA srvc is used to archive files as objects, for long-term reference, and gives each object a persistent identifier. The DOA srvc trusts one or more UW srvc’s, and accepts files from them. The DOA srvc holds objects for a long time, with high availability, and makes them available to their owners and to other users, as authorized. –For example, the owner may reference an object in a research paper. –For example, the owner may make a measurement data set object available to other researchers to guide their research.

Sponsored by the National Science Foundation 11 Nov 4, 2010 Meas Data Archive (MDA) Srvc Conceptual Design (v1.0)

Sponsored by the National Science Foundation 12 Nov 4, 2010 Meas Data Archive (MDA) Srvc File Transfer Interface User can transfer meas data (and other types of) files between Exp Cntrl Srvc or Meas Srvc (or a component) and User Workspace (UW) Srvc, which contains a Linux home directory structure, setup by user –Supported protocols include: scp; ftp(s); grid (third-party) ftp; http(s). –When transferring a file, authentication and authorization are done using keys “dropped” by the CF into the involved services

Sponsored by the National Science Foundation 13 Nov 4, 2010 Meas Data Archive (MDA) Srvc Home Directory Structure Each user has a Home Directory structure in UW Srvc. User can set directory or file permissions for self, other users and identified groups

Sponsored by the National Science Foundation 14 Nov 4, 2010 Meas Data Archive (MDA) Srvc Measurement Data and Metadata Files All files have.ext to indicate nature and format of file Each Measurement Data file will always have a Metadata file, with an identical filename and a different extension User can generate metadata file where data is collected, and transfer it in parallel to the UW srvc; otherwise, metadata file will be generated in the UW srvc

Sponsored by the National Science Foundation 15 Nov 4, 2010 Meas Data Archive (MDA) Srvc Browse Files, Search and Edit Metadata User can browse files via ssh or web interfaces Use can search metadata via web interface. User can edit metadata, e.g., add annotation, via web interface

Sponsored by the National Science Foundation 16 Nov 4, 2010 Meas Data Archive (MDA) Srvc Transfer Files to Digital Object Archive Srvc User can select file(s) (including metadata) for transfer to Digital Object Archive (DOA) Srvc, get necessary persistent object identifier, transfer file as an object, and set object access rules –User can update object in DOA service –User can retrieve object from DOA service –User can allow others to browse, search and get objects from DOA service

Sponsored by the National Science Foundation 17 Nov 4, 2010 Meas Data Archive (MDA) Srvc Authentication, Authorization and Access Control AM authenticates and authorizes users through CF (usually based on credentials) AM has full administrator privileges in Linux server and file system AM creates Linux account for each user allocates resources, e.g., storage, and inserts public ssh keys Administrator via AM assigns Users to Linux Groups File System implements extended access control for directories or individual files based on Linux group structure

Sponsored by the National Science Foundation 18 Nov 4, 2010 Meas Data Archive (MDA) Srvc Identified in GENI Inst & Meas Architecture as a core service. Several tools/projects have referenced such a service, but none have implemented one. Definition/conceptual design effort started after I&M workshop in June, 2010, by: –Evan Zhang (now UMass Lowell) –Harry Mussman (GPO) –Larry Lannom, Giridhar Manepalli, Jim French and Christophe Blanchi (CNRI) –v1.0 of definition/conceptual design now complete and ready for review by WG Prototype of MDA service: –Will be built by CNRI during Spiral 3 –Who will use it by end of Spiral 3? Next steps: –Limited discussion today. Comments? Questions? –Additional reviews? when? how?