1. INFSO-RI-508833 Enabling Grids for E-sciencE www.eu-egee.org NA4 Biomed Applications EGEE-fr meeting, March 24, 2005 Christophe Blanchet (CNRS IBCP) NA4 Biomedical Applications Slides courtesy of V. Breton (1st EU review) and J. Montagnat (6th biomed meeting)

2. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 2 Enabling Grids for E-sciencE INFSO-RI-508833 Biomed status EU review output with specific comments to NA4 –subject of next session –TA changes may modify deliverable dates Focus on demonstration of grid interest for applications New applications starting –LPC: remote medicine experience with China –WHO: human evolutionary genetics –other EoI LCG2 migration to SL3 going on –CC-IN2P3 announced end of RH7.3 at end of this month Highlight on gLite –migration plan requested –still complaining of lack of biomed feedback / lack of testing –prototype testbed expected to bring much more robustness

3. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 3 Enabling Grids for E-sciencE INFSO-RI-508833 The role of the pilot applications – HEP and Biomedicine Initial area of focus to establish a strong user base on which to build a broad EGEE user community Provide early feedback to the infrastructure activities on their experience with application deployment and VO management Act as guinea pigs and provide early feedback to the middleware developers on their experience with new services

4. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 4 Enabling Grids for E-sciencE INFSO-RI-508833 The characteristics of biomedical pilot applications Prototype level at project day 1 VO was created after the project kicked-off Very decentralized: application developers use the grid at their own pace Very demanding on services –Compute intensive applications –Applications requiring large amounts of short jobs –Need for interactivity or guaranteed response time Resources were focused on the deployment of large scale applications on LCG-2 –Integration of Biomed VO used to identify issues relevant to all VOs to be deployed during EGEE lifetime –Decentralized usage of the infrastructure highlights different weaknesses from the more centralized HEP data challenges

5. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 5 Enabling Grids for E-sciencE INFSO-RI-508833 Status of Biomedical VO PADOVA BARI 15 resource centres ( )  17 CEs (>750 CPUs)  16 SEs 4 RBs: CNAF, IFAE, LAPP, UPV RLS, VO LDAP Server: CC-IN2P3 4 RBs 1 RLS 1 LDAP Server

6. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 6 Enabling Grids for E-sciencE INFSO-RI-508833 Biomedical VO: production jobs on EGEE

7. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 7 Enabling Grids for E-sciencE INFSO-RI-508833 Biomedical VO: production jobs on EGEE

8. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 8 Enabling Grids for E-sciencE INFSO-RI-508833 Biomedical applications –3 batch-oriented applications ported on LCG2  SiMRI3D: medical image simulation  xmipp_MLRefine: molecular structure analysis  GATE: radiotherapy planning –3 high throughput applications ported on LCG2  CDSS: clinical decision support system  GPS@: bioinformatics portal (multiple short jobs)  gPTM3D: radiology images analysis (interactivity) –New applications to join in the near future  Especially in the field of drug discovery DEMO

9. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 9 Enabling Grids for E-sciencE INFSO-RI-508833 Experience with LCG2 middleware Two categories of applications had different levels of success –Batch-oriented application (high performance): well adapted EGEE infrastructure, gridification has significant impact on performances –More dynamic applications (high throughput): gridification has been difficult with current middleware. Turn-arounds were needed to bypass some limitations Still a high failure rate reported on LCG2 (order of 25%) –Users tend to adapt manually their application (selection of sites to submit job, store data...) –Irregular through time (instability of the infrastructure) –This makes it difficult to estimate the failure ratio The SA1-biomed interaction loop is being set up –Significant improvement in feedback and solutions since Dec’04

10. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 10 Enabling Grids for E-sciencE INFSO-RI-508833 Evolution of biomedical applications Growing interest of the biomedical community –Partners involved proposing new applications –New application proposals (in various health-related areas) –Enlargement of the biomedical community (EMBRACE FP6 NoE in Bioinformatics) Growing scale of the applications –Progressive migration from prototypes to pre-production services for some applications –Increase in scale (volume of data and number of CPU hours) Towards pre-production –Several initiatives to build user-friendly portals and interfaces to existing applications in order to open to an end-users community

11. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 11 Enabling Grids for E-sciencE INFSO-RI-508833 Feedback to LCG-2 middleware developers and infrastructure From HEP applications –Experiment Integration Support group and Grid Applications Group produced documents summarizing problems encountered in use of LCG-2 From Biomed applications –Very significant exchanges related to the set-up of the biomed VO and the deployment of relevant services –Request to use MPI

12. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 12 Enabling Grids for E-sciencE INFSO-RI-508833 Major issues and their mitigation Issue 1: the planning and execution of the migration of applications currently deployed on LCG2, both HEP and non-HEP, to the new gLite middleware Issue 2: the provision of management and support structures for the integration of multiple user communities, and taking into account the significant increase in the number of EGEE active users Issue 3: the availability of security-enhanced services for data manipulation and job execution

13. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 13 Enabling Grids for E-sciencE INFSO-RI-508833 Issue 3: Security-enhanced services Security-enhanced services for data manipulation and job execution are critical for industrial partners and non-HEP scientific applications gLite 1.0 will provide new security-enhanced services –VOMS for VO management –The File and Replica Catalog provides support to ACLs –These services must be tested by the biomedical pilot applications Additional support needed –ACL support at Storage Element level Secure access to licensed software – will be investigated via specific cases with members of the Industrial Forum

14. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 14 Enabling Grids for E-sciencE INFSO-RI-508833 Biomed status PTF –New requirements database (Savannah)  http://savannah.cern.ch/support/?group=egeeptf http://savannah.cern.ch/support/?group=egeeptf –You are invited to go, see, and comment! –Now in the end of middleware developers: feedback coming MWSG –prioritization on security related tasks –4 biomed-related security components  Data encryption and keys management  Pseudonimity  Dynamic connectivity service  User key management –Work started on data encryption and keys management (now in the hands of JRA3) –Lot of discussion with JRA1 data management activity

15. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 15 Enabling Grids for E-sciencE INFSO-RI-508833 Biomed status 3 rd ARDA workshop –intensive gLite testing Demonstration session in Athens –GPS@ interested, any others? User survey form –biomed as a pilot Please do react to sollicitations... –...user survey form –...configuration meeting –...migration plans –...etc

16. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 16 Enabling Grids for E-sciencE INFSO-RI-508833 Reviewers recommendation to NA4 1- Have all current applications migrated to gLite with a very good user satisfaction rating on application development support and grid infrastructure operations 2- Building on the experiences of previous FP5 grid projects, capture full requirements of future user groups, assess needs for new Grid services and plan accordingly for later implementation 3- Clarify the true motivation of users from new application areas right from the beginning. In the extreme, are users from the new areas really interested in applying grid technologies to explore new ways to produce more and better scientific results or simply interested in accessing large and cheap resources?

17. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 17 Enabling Grids for E-sciencE INFSO-RI-508833 Response to recommendation 1: application migration to gLite General NA4 strategy: difficult to migrate before users are convinced –Show stoppers: robustness and scale (number of sites/nodes) –Need to change OS system (RH7.3-> SL) Scale: EGEE-0 and EGEE-1 need to share resources –gLite and LCG-2 should coexist on the same physical machines (worker nodes) General roadmap –Common testing activity will allow to set-up preproduction service –Once preproduction service is available, validation of a subset of applications on preproduction service. –Once validated, the other applications will migrate

18. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 18 Enabling Grids for E-sciencE INFSO-RI-508833 Biomed migration to gLite Biomed experiments are waiting for a system with at least the same functionalities and robustness as the current middleware. –Last reports mention instability (on the prototype) –See the end of March release CDSS (Clinical Decision Support System) first pilot to migrate –UPV actively testing gLite prototype –Expecting a service-based infrastructure Migration plans for gPTM3D, GPS@ and remote medicine –no date set yet Other applications wait and see Some testing started at CINES computing center

19. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 19 Enabling Grids for E-sciencE INFSO-RI-508833 Response to recommendation 2 Status: Process to collect requirements is set-up and operational –Extension to projects or applications connected to EGEE  External projects and user communities have entries into the requirements database Needs: –Collect user requirements more broadly taking advantage of previous FP5 projects  Resources needed to evaluate outcome of FP5 projects –Define requirements process from their collection to the implementation of grid services  How are project requirements taken into account for middleware implementation ? requirements capture, need assessment for future implementation

20. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 20 Enabling Grids for E-sciencE INFSO-RI-508833 Response to recommendation 3 New application selection process up to now –EGAAP Initial philosophy is to attract people, not protect project resources –Two step process to accept new applications  First light filter: EGAAP, acting as a scientific committee  Second technical filter and formal approval by PEB Need to further strengthen the selection process –Section 9.11, p15: “Capture user and application requirements in details and determine the impact on Grid infrastructure and services. The decision to deploy any new application (and if yes when) should be based on a very careful analysis of all associated technical and non-technical risks.” Careful evaluation of new applications prior to EGAAP approval requires SA1 involvment Clarify the true motivation of users from new application areas

21. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 21 Enabling Grids for E-sciencE INFSO-RI-508833 Reviewers recommendation for future work New to address complex workflows Testing and software packaging will be critical to success. Reinforce these also intellectually very demanding activities even further. The ambition to quasi-fulfill industrial or commercial requirements should be relaxed until the end of the current contract. If examples can be found of scientific work that could not have been done without the EGEE development (e.g. necessary distributed process, truly enormous calculations), they will be a strong motivation for future development.

22. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 22 Enabling Grids for E-sciencE INFSO-RI-508833 Responses to recommendations for future work On-going activity on deploying complex workflows –Integration of TRIANA in GENIUS (INFN-Catania) –Interfacing of TAVERNA with LCG2 (CNRS-I3S) Collaboration with TRIANA group from GridLab is pursuing very well –First full writer/reader of LCG-2 DAG JDL files coded in TRIANA –Latest version of TRIANA deployed on GILDA at Catania and integrated in GENIUS –Tests are undergoing Taverna: worflow management system (from MyGRID UK e-science project) –Development of WS wrapper to LCG2 command line interface How to demonstrate a better usage of the grids by scientific communities ? (1/2)

23. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 23 Enabling Grids for E-sciencE INFSO-RI-508833 Responses to recommendations for future work Examples of scientific work impossible without a grid to be identified –HEP –Large scale docking for drug discovery on tropical diseases –… Could EGEE resources be dedicated to non HEP data challenges for short periods of time ? How to demonstrate a better usage of the grids by scientific communities ? (2/2)

24. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 24 Enabling Grids for E-sciencE INFSO-RI-508833 NA4 15 month plan No major change to the text already available on EDMS –Revision under way following reviewers feedback Migration to gLite of a subset of applications Virtuous cycle –Improve application identification and selection process –Make sure the user communities already on LCG2 are happy –Set up the tools to survey user satisfaction

25. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 25 Enabling Grids for E-sciencE INFSO-RI-508833 NA4 15 month plan: issues Issues from present version identified before review –prepare the transition from LCG-2 to gLite foreseen after the first EC review. It is the biggest challenge in light of the applications already active on the infrastructure –get reliable indicators of decentralized activity on the production infrastructure as most of the statistics produced so far are based on estimates done by hand. –prepare the transition from NA4 supported application deployment to autonomous usage of the infrastructure for the first scientific communities selected by EGAAP. –project has underestimated effort in managing the virtuous cycle and this is putting unforeseen demands on time necessary for liaison involving all the actors e.g. NA4, NA1, NA2, NA3, SA1.

26. EGEE France Federation, Marseille, March 24 th 2005, Christophe Blanchet 26 Enabling Grids for E-sciencE INFSO-RI-508833 Proposed changes to TA Move MN4.2 Milestone to PM14 Reduce number of deliverables by merging DNA4.3.3 Final Report of Application Identification and Support Activity, including Application Migration status M24-> M22 DNA4.4 -> DNA4.3.3 Second revision of EGEE Application Migration Progress reportM21DNA4.3.3 First revision of EGEE Application Migration Progress report M15 -> M18 DNA4.3.2 Second external review of Applications Identification and Support with feedback M24MNA4.3 First external review of Applications Identification and Support with feedback M12 -> M14 MNA4.2