1. © 2006 Open Grid Forum PGI Use Cases: EGI, GROMACS, Data-intensive HTC Oxana Smirnova 26 October 2010, OGF30, Brussels

2. © 2010 Open Grid Forum 6 OGF IPR Policies Apply “ I acknowledge that participation in this meeting is subject to the OGF Intellectual Property Policy. ” Intellectual Property Notices Note Well: All statements related to the activities of the OGF and addressed to the OGF are subject to all provisions of Appendix B of GFD-C.1, which grants to the OGF and its participants certain licenses and rights in such statements. Such statements include verbal statements in OGF meetings, as well as written and electronic communications made at any time or place, which are addressed to: the OGF plenary session, any OGF working group or portion thereof, the OGF Board of Directors, the GFSG, or any member thereof on behalf of the OGF, the ADCOM, or any member thereof on behalf of the ADCOM, any OGF mailing list, including any group list, or any other list functioning under OGF auspices, the OGF Editor or the document authoring and review process Statements made outside of a OGF meeting, mailing list or other function, that are clearly not intended to be input to an OGF activity, group or function, are not subject to these provisions. Excerpt from Appendix B of GFD-C.1: ” Where the OGF knows of rights, or claimed rights, the OGF secretariat shall attempt to obtain from the claimant of such rights, a written assurance that upon approval by the GFSG of the relevant OGF document(s), any party will be able to obtain the right to implement, use and distribute the technology or works when implementing, using or distributing technology based upon the specific specification(s) under openly specified, reasonable, non- discriminatory terms. The working group or research group proposing the use of the technology with respect to which the proprietary rights are claimed may assist the OGF secretariat in this effort. The results of this procedure shall not affect advancement of document, except that the GFSG may defer approval where a delay may facilitate the obtaining of such assurances. The results will, however, be recorded by the OGF Secretariat, and made available. The GFSG may also direct that a summary of the results be included in any GFD published containing the specification. ” OGF Intellectual Property Policies are adapted from the IETF Intellectual Property Policies that support the Internet Standards Process.

3. © 2010 Open Grid Forum Overview EGI yields the largest set of requirements GROMACS and Data- intensive jobs are valid within EGI as well There are overlapping requirements See also doc16066 for all use cases 6 EGI doc16031 GROMACS doc16042 (was doc15576) Data-intensive computational jobs on cluster- based Grids doc16012

4. © 2010 Open Grid Forum EGI Use case European e-Science infrastructure Very diverse hardware resources Different middleware providers Globus, UNICORE, gLite, ARC, dCache, desktop solutions Wide range of applications High-level scenario: provision of arbitrary computational and/or storage resources to an arbitrary (group of) researcher(s) The wide scope of the use case implies a very large set of requirements (62 identified initially) If OGF standards will address EGI requirements, large part of other use cases will be covered as well 6

5. © 2010 Open Grid Forum GROMACS use case GROMACS-based Molecular Dynamics in Bio-molecular Systems The first formalized PGI use case Application uses MPI or PVM to parallelize the jobs Long simulation, can last 2-5 days Produces large data files on output Present in both HPC and cluster environments 6

6. © 2010 Open Grid Forum Data-intensive processing Data intensive processing on cluster-based Grids Scenarios from High Energy Physics applications Focus is on serial jobs constituting complex workflows Target systems: Linux clusters a single workflow may spread over dozens of clusters in different countries Heavy use of mass storage services Necessity to optimise input/output data handling Users are organised in huge Virtual Organisations (thousands of members), with groups and roles 6

7. © 2010 Open Grid Forum Matching requirements from the PGI inventory EGIGROMACSData-intensive Compute 34, 35, 37, 43, 44, 45, 61, 63, 64, 66, 68, 72, 76, 78, 80, 81, 82, 84, 86, 90, 94, 100, 101, 104, 105, 108, 113, 114, 119, 126, 133, 141, 143, 144 34, 35, 43, 44, 45, 61, 63, 64, 66, 68, 76, 78, 81, 82, 86, 90, 101, 104, 105, 108, 112, 114, 119, 143, 144 34, 35, 43, 44, 45, 56, 63, 64, 66, 68, 76, 78, 80, 81, 82, 85, 86, 90, 94, 100, 101, 104, 105, 108, 112, 114, 116, 119, 126, 133, 141, 143, 144 Info 1, 4, 6 Security 9, 11, 17, 20, 21, 22, 23, 28, 29, 30, 32 11, 28, 299, 11, 28, 29, 30, 32 Accounting 36 Non- functional 145, 146, 147, 148, 149, 150, 156 148, 149, 156145, 146, 148, 149, 150, 156 Other 160 6 In BOLD are those already blessed by PGI

8. © 2010 Open Grid Forum Common requirements 6 Common requirements for EGI/GROMACS/Data- intensive use cases Compute 34, 35, 43, 44, 45, 61, 63, 64, 66, 68, 76, 78, 81, 82, 86, 90, 94, 101, 104, 105, 108, 114, 119, 143, 144 Info 1, 4, 6 Security 11, 28, 29 Accounting - Non-functional 148, 149, 156 Other -

9. © 2010 Open Grid Forum Matching numbers to requirements 34 An easy way of launching applications or pre-configured /pre-installed software w/o specifying location details. Installed/pre-configured ones should be exposed as well as part of the resource description. The available software should be exposed through the service and in turn be requestable for resource/applications statements in JSDL. E.g. extending the JSDL with "pre-configured sw pieces" and/or "software libraries", e.g. global namespace of this applications might be beneficial as well., etc. 35 Application management, for example: pre-installed applications, abstract notion of the application (i.e. specify w/o executable locations), 43The Execution Service MUST provide a way to create Activities. 44Activities submitted by Clients to an Execution Service MUST be specified using a well-defined Job Description document 45 On creation of an Activity, the Execution Service MUST return to the Client an Activity ID permitting the Client to perform subsequent actions (Query, Cancel,...) on this precise Activity 61 The Execution Service SHOULD support any Activity running massively-parallel processes using MPI and large-scale HPC Systems 63The client must be able to obtain the status of an activity 64 The Client must be able to obtain the activity status information with different levels of verbosity (basic, detailed, more detailed) 66 The Instance of Execution Service MUST permit clients to request the list of all Activities managed by this instance on which the Client has authorization 68 The Client must be able to query the detailed Information of an Activity, and MUST able to obtain the detailed Activity Information (more than just the status, for example: GLUE2 Activity information). 76 In order to permit the Client to perform client-directed processing of submitted Activities (for example client-directed data staging), the Execution Service SHOULD manage 'Hold' points for Activities. Fault if not supported. 78 Requirement to support a incoming/pending/queued state (i.e. waiting to be executed) as a sub-state of running, having submitted a job to a batch system like LoadLeveler, Torque, and such like pending means it was accepted, is scheduled, but not running yet (kind of waiting to be executed). 81 On Activity submission, if the Execution Service fails to create the Activity for any reason, the Execution Service MUST return to the Client a descriptive error 6

10. © 2010 Open Grid Forum Matching numbers to requirements 6 82 It is important that Job Description elements MUST NOT be ignored by the service and rather throw a corresponding fault in the case that the service does not support this particular client request or doesn't understand it. (e.g. JSDL spec states that all elements must be understood by the service before ignoring elements marked as optional) 86 The Execution Service offers the possibility to 'cancel/terminate/kill/destroy' an Activity. The cancel means that active data- staging processes SHOULD be cancelled and active executions in RMS MUST be cancelled. 90 Execution Service SHOULD perform a certain to be agreed set of validations (potentially several steps defined by PGI). Not all of them are necessarily needed to be done as part of the activity creation. An incomplete validation maybe done at activity creation but a full validation should be completed later, e.g. support of scientists that perform manual data-stagings might need some very well validated jobs in order to not redo a lot of manual data-stagings - Further validations are subject to be decided by service implementation 94 Requirement to have a purge (maybe called wipe) operation. Removing all presence (except logs & usage records) of the activity (when it is not longer active or once ffinished). This fcuntionality is only allowed on any final state according to the PGI state model. The functionality is not supposed to kill Activities, that is why its only allowed on final states. 101The Job Description document specification MUST permit the Client to request automatic data stage-in and stage-out 104 The Job Description SHOULD allow to specify n different alternative(!) data sources of any single input data.(semantics: data may be fetched from some of the alternative sources) 105 Job Description supports, 1) service-directed data-staging (i.e. specify data-stagings),2) client-initiated data-staging (i.e. specifiy data-staging but service will do nothing only control whether files existing), 3) manual data-staging (i.e. no specification of data-staging elements) 108 Job Description comes with support for requesting multi-processor Activities (for example: threads/node, network topology, task/core mappings, multi-threading and such like). 114 Job Description document should offer a new re-usable structure to describe a software requirement, to express relations (OR, AND, GREATER, SMALLER,...) of software requests and other requests. 119The Execution service MUST support a common state model 143Data Staging MUST support an agreed set of protocols in PGI, e.g. HTTP(S), scp, gridftp, mailto, RNS/ByteIO 144 Minimize the amount of calls required to interact with Execution Service (for example: in one WS: vector operations, service exposes the location where the client can stage any input data, service exposes the location where the client can fetch any output data.)

11. © 2010 Open Grid Forum Matching numbers to requirements 6 Info 1 All grid entities (if possible) MUST be described using the GLUE model - if not possible extensions for the GLUE model are necessary 4Each service must publish information regarding its service properties 6 Instance of Execution Service MUST provide access to attributes/meta data that describe this instance using GLUE. (V-2 note: The appropriate sections of the XML rendering of GLUE need to be identified.). Different levels of verbosity should be offered. Security 11 Each Service MUST publish the Authentication and Authorization methods accepted by its Endpoints in conformance with GLUE recommendations, example: endpoint capabilities for security like WS-Security and X.509, WS-SecurityPolicy within the GLUE element, list of authenticaiton and authorization methods (e.g. profiles like WS-SecureCommunication ) - extend capability_t values security.authentication.ssl and security.authorization.ssl - do we extend with Strings only or do we insert XML (WS-SecurePolicy) 28 Service has to act on behalf of another identity. Services often need to delegate those Credentials to other Service. Therefore, Services MUST offer interoperable mechanism for Credential delegation. 29 We need delegation and have to define exactly what delegation is and its methods. E.g the delegation procedure might require multiple steps (one approach might be the delegation portType), e.g. gridsite delegation method - bringing delegation to the portType level using operations. Non-functional 148 Software components (Services and Clients) MUST generate and propagate meaningful error messages, including context description 149Specifications SHOULD prevent the occurrence of SPOFs and bottlenecks 156Need to provide high availability and reliability

12. © 2010 Open Grid Forum 6 Full Copyright Notice Copyright (C) Open Grid Forum (2010). All Rights Reserved. This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. The limited permissions granted above are perpetual and will not be revoked by the OGF or its successors or assignees.