1. Final Review Meeting 16 th March 2010 Brussels (Belgium) www.d4science.eu D4Science Production Infrastructure Contract n°: RI-212488 Pedro Andrade (CERN) pedro.andrade@cern.ch www.d4science.eu

2. 2 D4Science Final Review Meeting Brussels, 16 th March 2009 Introduction  Introduction  Major Achievements  Activities and Roles  Operational Procedures  Infrastructure Operation  Infrastructure Exploitation  Conclusions

3. 3 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Introduction The objective of the Service Activity is to make available and maintain a stable production infrastructure by providing Virtual Research Environments to the project scientific user communities (EM and FARM) Several challenges to address  Improve and extend the previous DILIGENT prototype testbed running gCube v0.9  Address new user communities and new user requirements  Make better usage of gCube functionality for autonomic infrastructure management  Improve the operations and software release procedures

4. 4 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Major Achievements gCube Releases  7 major release cycles and 11 maintenance cycles  From org.gcube.1.0 to org.gcube.1.6  6 major releases and 8 maintenance releases in 2 nd year  2400 deployment tests executions  44000 test cases executions  Latest release composed by 181 software archives  395 components including services, libraries, and portlets  Release preparation based on multiple support tools  Fully documented by developer, user, and admin guides  Made available via the gCube distribution site

5. 5 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Major Achievements Production Infrastructure Operation  Infrastructure evolution based on 3 milestones  5 site providers: CNR, ESA, FAO, NKUA, UNIBASEL  1 central coordination: CERN  Running gLite middleware and gCube software  Part of EGEE production infrastructure MSA1.1 July 08 1 Community 4 Sites MSA1.2 February 09 2 Communities 4 Sites MSA1.3 November 09 2 Communities 5 Sites

6. 6 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 VREs Availability  2 User Communities  2 VOs and 5 VREs Major Achievements GVM VRE AquaMaps VRE FCPPS VRE MSA1.1 Jan08Apr08Jul08Oct08Jan09Apr09Jul09Oct09Jan10 MSA1.2MSA1.3 r1.0 r1.1r1.2r1.3r1.4 r1.5 r1.6 GCM VRE ICIS VRE FARM EM

7. 7 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Activities and Roles Developers Subsystem Managers Integrators Testers Release Managers VRE Users VRE Designers Site Managers Infrastructure Admins Data Managers VO Admins VRE Managers Support Team gCube Development gCube Integration & Testing Infrastructure Deployment VREs Deployment VRE Exploitation

8. 8 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Infrastructure operation is based on well defined procedures SSoftware: Release Cycle, Maintenance Cycle NNodes: Deployment, Certification, Downtime IInfrastructure: Monitoring, Accounting SSupport: Incident Management VVO/VRE: VO Creation, VRE Creation, Validation, Data Staging, User Registration Based on ITIL methodology and concepts Consolidation and extension of EGEE procedures Supported by many collaboration tools (ETICS, TRAC, Wiki) Operational Roles and Procedures

9. 9 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Release Cycle Procedure Organized in a sequence of tasks Well defined roles for each tasks Clear transition between tasks Operational Procedures Preparing Cycle Delivering Components Delivering Subsystems Releasing to Integration Building & Packaging Deployment Testing Functional Testing Releasing to Production

10. 10 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Operational Procedures VO Creation Procedure  Services deployment  VO Manager deploys the mandatory services of the enabling layer  VO Manager deploys the services to support specific scenarios  Configuration  VO Manager applies post-deployment configurations: e.g. preparing service map, registering mandatory resources  Management  VO Manager updates the VO operation conditions (RIs, users, etc)  Data Manager imports the defined data collections Most tasks are automatically implemented/enforced by gCube

11. 11 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Operational Procedures Incident Management Procedure  Manages all incidents affecting the production service  1 st level support by Support Team (help desk)  2 nd level support by TCom members (functional escalation)  PEB for final decisions, if needed (hierarchical escalation)

12. 12 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Infrastructure Operation The infrastructure hardware is provided by 5 project partners Enough to support the needs of present VREs and VOs Support from other EGEE infrastructure sites added the availability of 2982 WNs and 228 TB disk (ASGC, INFN, etc) SiteHardware * gCube Nodes gLite WNsDowntimes CNR25 381015 ESA13 2399 FAO1 2-0 NKUA8 121025 UNIBASEL6 6-2 TOTAL53 605951 (*) multiple cores

13. 13 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009  100% increase since Q4  Q4 to Q5 grow due to MSA1.2  CNR contribution is ~50% Infrastructure Operation  r1.2 upgrade: almost 2 weeks  r1.5 upgrade: few hours

14. 14 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Infrastructure Operation Monitoring of infrastructure status and size  gCube nodes  Based on two distinct gCube subsystems gCube Information System: Live Monitor, IV, Advanced Monitor gCube Messaging System: Producer, Broker, Consumer  Monitoring tailored views for different operational needs  Aggregated data coming from all infrastructure nodes  Stores infrastructure status historical data  Active notification mechanism to Site Managers and VO Admin  gLite nodes  Based on EGEE monitoring tools: SAM, GridMap, GStat, etc

15. 15 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Infrastructure Operation Accounting of infrastructure exploitation and load  gCube nodes  Based on gCube Messaging System (as for monitoring)  The accounting information can be of two types: Infrastructure load: accounting information related to the service level communication: running instance invoked, method invoked, caller scope, caller gHN, invocation time Infrastructure exploitation: accounting information related to the user exploitation of the infrastructure via the portal: login, browse collection, simple search, advanced search, google search, quick search, content retrieval  gLite nodes  Based on EGEE monitoring tools: APEL, DGAS

16. 16 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Infrastructure Operation

17. 17 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Infrastructure Operation The infrastructure operation is facilitated by several factors  Common representation of infrastructure resources  Allowed a common approach to query and monitor resources  Well defined and certified gCube software archives  Allowed the execution of certified software  Simplified execution of infrastructure deployment  Allowed automatic and remote deployment of gCube  Common solutions based on platform tools  Allowed accounting based on call interceptions Cost effective operation of distributed resources where new VOs and VREs can be created in few minutes

18. 18 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Infrastructure Exploitation 30 Nodes CNR NKUA ESA FAO UNIBASEL 25 Data EEA MERIS AATSR 69 Metadata es ISO19115 eiDB 15 Data AquaMaps Fact Sheets Country Maps 28 Metadata FARM_dc aquamaps NodesCollectionsFunctionality 29 Nodes CNR NKUA FAO UNIBASEL Integration with gPod Geographical and text search Search by metadata Personal workspace Objects annotation Report generation Maps Generation Production VOs/VREs

19. 19 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Infrastructure Exploitation Additional VOs/VREs Support for other project related activities  Demonstration, training, and dissemination  VRE available to all project members  Deployment of Demo VRE under the Ecosystem VO Support for user communities external to D4Science  Collaboration with King’s College (UK)  Experimentation in the humanities domain  Deployment of gMan VRE under the Ecosystem VO AquaMaps is a production VRE but was not originally planned

20. 20 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Conclusions After two years providing a production service  All resources (hardware, data, and services) planned in the begin of the year made available  Distinct VREs created for project user communities (EM and FARM) and other external communities  gCube software stabilized and improved through several release and maintenance cycles  Operations procedures well implemented and consolidated A gCube based Infrastructure is ready to provide production quality VREs to different user communities under cost effective maintenance effort

21. 21 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 THANKS !! QUESTIONS ??

22. 22 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 AUXILIARY SLIDES

23. 23 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Effort spent by activity

24. 24 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 gCube Releases

25. 25 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 VO VRE VREs Deployment and Exploitation Infrastructure

26. 26 www.d4science.eu D4Science Final Review Meeting Brussels, 16 th March 2009 Services and Nodes Management From the experience deploying VOs/VREs for EM & FARM: New VREs can be created in few minutes ACTIONROLETOOLTIME VO deploy gHNsSite ManagerCLI1 min stage data collectionsData ManagerPortalhours register resourcesResource OwnerPortal10 min deploy VO servicesVO AdminPortal30 min manage VO users and rolesVO AdminPortal5 min VRE define VRE functionality and dataVRE DesignerPortal5 min deploy VRE services and layoutVRE ManagerPortal10 min manage VRE users and rolesVRE ManagerPortal5 min