D4Science and ETICS Building and Testing gCube and gCore Pedro Andrade CERN EGEE’08 Conference 25 September 2008 Istanbul (Turkey)

2 D4Science D4Science is a FP7 project establishing a knowledge infrastructure of data, services and processing resources through the provision of Virtual Research Environments D4Science simplifies the discovery, access and production of existing and new data through a collaborative environment D4Science continues and consolidates the work started in the DILIGENT project opening it to new communities D4Science provides a production environment for ESA & FAO D4Science and ETICS 25 September 2008, Istanbul (Turkey)

3 gCube gCube is the software implementing the D4Science resources sharing vision via Virtual Research Environments gCube software:  Provides a multi-domain Service Oriented Infrastructure  Gathers data, services and hardware as system resources  Supports on-demand virtual collaborations  Reduces the cost of managing research infrastructures  Tailored for e-Science (content oriented, data intensive)  Is complaint with consolidated and emerging standards D4Science and ETICS 25 September 2008, Istanbul (Turkey)

4 gCube gCube resources:  Computing and storage  VRE services  content management  search and index  annotations  Collections  raw data  content  metadata  Processes to manage other resources D4Science and ETICS 25 September 2008, Istanbul (Turkey)

5 gCore gCore is a framework developed to reduce the complexity of designing and implementing gCube SOI complaint services gCore is composed by:  gCore Framework: An application framework to consolidate and develop existing/new gCube services  gCore Container: A self-contained software container distributed to facilitate the deployment of gCube services D4Science and ETICS 25 September 2008, Istanbul (Turkey)

6 D4Science in ETICS Two projects:  org.gcube: 24 subsystems  org.gcore: 2 subsystems 225 components:  160 software components  51 services, 84 libraries, and 25 portlets  Including services samples, external libs, test libs (18 components)  65 distribution components  64 gCube Software Archives  gCore Hosting Node Around 500 configurations D4Science and ETICS 25 September 2008, Istanbul (Turkey)

7 Preparing Cycle Delivering Components Delivering Subsystems Releasing to Integration Building & Packaging Deployment Testing Functional Testing Releasing to Production D4Science Release Cycle Bug Fixing Patching the Production D4Science and ETICS 25 September 2008, Istanbul (Turkey) etics component configurations etics subsystem configurations etics project configurations etics-build etics-test

8 Build The objective of the build activity is to ensure api-level compliance of components and the creation of components’ artifacts. ETICS is exploited to:  Model build scripts and manage build dependencies  Execute on-demand remote builds (via WA and CLI)  Execute nightly local builds for development configurations  Produce distribution artifacts (via dedicated components)  Feed the D4Science reporting application with xml raw reports D4Science and ETICS 25 September 2008, Istanbul (Turkey)

9 Deployment Testing The objective of the deployment testing activity is to ensure that the gCube software archives:  Can be remotely and automatically deployed  Once deployed, that it starts and is activated correctly ETICS is exploited to:  Manage the executing the deployment tests  Define the deployment test tool dependencies  Generate deployment test reports D4Science and ETICS 25 September 2008, Istanbul (Turkey)

10 Deployment Test D4Science and ETICS 25 September 2008, Istanbul (Turkey) ETICS Server Condor Node Service 2Service 1Service A Service 2Service 1Service A

11 Static Analyzers The objective of the adoption of static analyzers is to improve the quality of gCore and gCube software ETICS is planned to be exploited to:  Run jFindbugs and jDepend (under adoption) Other analyzers (not exploited via ETICS):  Check coverage of javadoc documentation (checkstyle)  Monitor the content of code repository  e.g. pre-compiled artifacts, etc..  Certification of distribution components  check structure, correctness, etc.. D4Science and ETICS 25 September 2008, Istanbul (Turkey)

12 Current Releases D4Science and ETICS 25 September 2008, Istanbul (Turkey)

13 Current Releases D4Science and ETICS 25 September 2008, Istanbul (Turkey)

14 Improvements Parametric tests on remote infrastructure  Single remote test with multiple parameterized tests Co-scheduling mechanism to be further exploited  Condor nodes firewall limitations Notification mechanism  ETICS is designed to support distributed teams  No notifications between different ETICS roles  components availability  successful build/test execution D4Science and ETICS 25 September 2008, Istanbul (Turkey)

15 Improvements ETICS configurations contains many information  Can an ETICS API be useful ? Reports summary information  Summary page with built, test, and plug-ins execution  Organized by configuration Plug-ins availability  Info about existing ones?  How to add external plug-ins? D4Science and ETICS 25 September 2008, Istanbul (Turkey)

16 Conclusions Solid exploitation:  DILIGENT/D4Science using ETICS since the early days  ETICS is the default tool in D4Science build an test tasks  ETICS completely adopted by all D4Science members Solid collaboration:  D4Science and ETICS plans kept aligned  Efficient support from ETICS D4Science and ETICS 25 September 2008, Istanbul (Turkey)