Conference name Company name INFSOM-RI Speaker name The ETICS Job management architecture EGEE ‘08 Istanbul, September 25 th 2008 Valerio Venturi INFN - ETICS

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Outline Deploying to the Grid infrastructures Identification of deployment issues Re-engineering the ETICS services Submitting ETICS jobs to the EGEE infrastructure Technical issues Organizational issues Submitting ETICS jobs using UNICORE Submitting ETICS jobs to EC2

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Integrating ETICS in Grid Infrastructures Problem statement Software projects maintain a variable-sized pool of machines for the various tasks of development and maintenance Building software releases Running continuous builds of the development release Testing the software For a number of reasons (needs of high-availability for release-rush, needs of possibly a lot of different platforms) this machines are likely to have high idle times. When a software projects happens to have other computing needs, it would great benefit to be able the idle time of their resources for their other computing needs.

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Integrating ETICS in Grid Infrastructures Problem statement Providing resources for software development tasks is what ETICS does for its customers. So far the machines are owned and managed by the ETICS project, thus eliminating the problem of allocating resources. Not sustainable. Where is gLite going to be build in a couple of years from now? We propose integrating the ETICS services in infrastructures (such as EGEE) to solve the sustainability problem and the more general problem of wisely using the resources projects allocate to software development tasks.

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Integrating the ETICS services Analysis of deployment issues 1 Build/Test Artefacts Configuration Web Service Report/Metrics DB Configuration DB Execution Engine Command Line User Interface Repository Web Service ETICS Infrastructure Web Portal

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Build/Test Jobs Management in ETICS Architecture Execution Engine Physical Worker Nodes Virtual OS Images ETICS Submission Engine

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Build/Test Jobs Management in ETICS Re-engineering the ETICS services This wouldn't be necessary if the various infrastructures had interoperable interfaces and security mechanisms Could have used BES and VOMS to submit to both gLite and UNICORE We will separate the part of the ETICS service that handle the jobs submission so that it can be deployed separately A submitting interface implemented using Metronome over a Condor pool, gLite over the EGEE infrastructure, and UNICORE over DEISA Operations to submit, cancel, getStatus Why not a standard interface like BES? ETICS jobs are pretty simple, doesn't really fit the bill

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS Jobs to EGEE Overview Configuration Web Service Submitter Repository Web Service ETICS Infrastructure EGEE Infrastructure VOMS JDL VOs

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS Jobs to EGEE Technical issues We don't envision big problems from a technical point of view, as gLite is thought to be easily integrated The WMS has a WS interface that makes it pretty easy to use Jobs are described using an established language as JDL The GLUE Schema allows for a (quite) standard description of resource requirements Use information published by sites through the GLUE Schema elements to land on nodes with the required architectures, operating system and compiler Future works on virtualizations will eliminate this problem Unfortunately VOMS doesn't a WS interface Hopefully gLite can get to it in the future

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS Jobs to EGEE Organizational issues In order to have access to the EGEE infrastructure, you have to be in one of the EGEE VOs. Two solutions Apply for an ETICS VO, moving some resources to the infrastructures and gaining share quotas Use the current EGEE Vos With either solutions, we have to decide whether Use delegation to submit the ETICS jobs with the user credentials Have the ETICS service registered to the VO and knows which credentials to use for submitting jobs

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS Jobs to EGEE An ETICS Virtual Organization? Apply for an ETICS VO Moving part of the nodes that are currently under exclusive ETICS management to the EGEE infrastructure Getting a quota with which to run build and test jobs In this case ETICS will manage its internals resource sharing, assigning priority and quotas to its projects.

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS Jobs to EGEE An ETICS Virtual Organization? PROS Simple solution ETICS may provide resources for building projects that are not in a VO and so don't have resources on the infrastructure CONS Not complete integration, ETICS would “buy” resources on the infrastructure and act as a broker to projects using it Does ETICS fit the profile for a VO Typically scientific collaborations

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS Jobs to EGEE Integrating with the current EGEE VOs ETICS uses an EGEE VO credentials for submitting build jobs, thus consuming resources assigned to that VO If a VO decide to use ETICS for building its software, it can use its resources shares for doing that For example, the gLite middleware builds could be accounted to DTEAM Two ways of doing that Using delegation The VO is able to grant single users the permission to build and test, and have fine grained control Having the ETICS service credentials registered in the VO, and the ETICS services maintaining a mapping to know which VO credentials to use for each builds

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS Jobs to EGEE Integrating with the current EGEE VOs Real-life example, building the gLite middleware The DTEAM VO in EGEE is used by gLite developers for testing the developments The DTEAM VO may have developer, integrator, tester, roles (or groups)‏ The DTEAM VO assigns shares to those roles, so that importance of the work is reflected This can be done dynamically, so to handle possible bug-fix- release-rush situations Developer, integrator and tester submits their jobs and these are executed on the infrastructure according to the policy decided by the VO Can be handled both by using delegation or the ETICS user in the VO

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS Jobs to EGEE Integrating with the current EGEE VOs PROS Complete integration, resource shares are arranged within EGEE and not between EGEE and ETICS CONS Projects that don't have EGEE shares won't build Registering the ETICS service credentials to the VO may scare someone Delegation would be the way round this The VO would assign to the ETICS service the same level of trust it assigns to its user The system provides controls to stop misuses

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS Jobs to EGEE Mixed approach We will go both ways, and will use the ETICS VOs We'll be able to provide resources for those projects that don't have shares on the infrastructures Will compete for resources, and likely have low priority We'll let VOs use their shares to build and test their software According to the priority that the infrastructures assign them, and the VO choices

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Submitting ETICS jobs to UNICORE The ETICS project is going to implement a submitter able to send jobs to UNICORE, so to be able to integrate into infrastructures using it (such as DEISA) Submits to UNICORE OGSA-BES Describe the job using JSDL

The ETICS Job Management Architecture INFSO-RI Istanbul, September 2008 Conclusions The ETICS System provides a grid-based remote build and test functionality The current implementation is based on Condor, but generalization and extension to additional infrastructures is required The ETICS submission mechanism has been generalized to be able to plugin different middleware implementations We are adding support for EGEE and gLite and investigating the possible authorization and usage models based on the VO concept Further extensions to UNICORE and other implementations are foreseen