INFSOM-RI Toward a Quality Certification Process for Grid Research Projects: an ETICS feasibility Study Andrea Manieri Adriano Rippa - Engineering Ingegneria Informatica s.p.a. - Engineering Ingegneria Informatica s.p.a. On behalf of ETICS Project

INFSOM-RI Open Grid Forum 20 - Manchester May Why bother? R&D project results are often interesting, but:R&D project results are often interesting, but: –They aim only at proving concepts –They suffer of deviation from plans due to their investigation nature –Developers (sometimes) don’t pay enough time to testing, documentation, etc –Developers are not so skilled in software engineering After the end of a projects:After the end of a projects: –None (out of the consortium) is willing to use such sw –The software itself is difficult to maintain and improve So, many times, effort and brilliant results are lost!So, many times, effort and brilliant results are lost!

INFSOM-RI Open Grid Forum 20 - Manchester May That are only dreams? Software components fully tested and documentedSoftware components fully tested and documented Maintenance of the know-how for build, deploy and run the softwareMaintenance of the know-how for build, deploy and run the software Software components that build on different platformSoftware components that build on different platform Software bundle with all the right dependencies and the right libraries to be used by other developersSoftware bundle with all the right dependencies and the right libraries to be used by other developers

INFSOM-RI Open Grid Forum 20 - Manchester May Quality assurance… QA is a matter of:QA is a matter of: –In which way sw has been created (procedural) –How sw is written (structural) –How sw perform (behavioural) Currently we have:Currently we have: –QA standards (I.e. CMMi, ISO, ITIL, …) –SW Testing –So-called Quality of Service (at least partially)

INFSOM-RI Open Grid Forum 20 - Manchester May What’s missing, then? A systematic approach toA systematic approach to –Build software with all the correct dependencies –Extensively Test each components That allows gathering some measuresThat allows gathering some measures –Related to the build performance –Reporting tests results and –Other characteristics of the code With which derive the overall quality of the software produced!With which derive the overall quality of the software produced! DISCLAIMER: this study will focus, initially, on Research ProjectsDISCLAIMER: this study will focus, initially, on Research Projects

INFSOM-RI Open Grid Forum 20 - Manchester May Yet another Quality Assurance model? Current models are useful but:Current models are useful but: – are process oriented – provide theoretical guidelines on the organization The problem:The problem: –Guidelines can be applied but it’s hard to systematically verify goodness of results

INFSOM-RI Open Grid Forum 20 - Manchester May ETICS feasibility study To study and propose a grid software Certification model that canTo study and propose a grid software Certification model that can –Be automated and –Be not subjective –Used within Research Projects ETICS will analyse the feasibility of a Certification for grid software:ETICS will analyse the feasibility of a Certification for grid software: –To support the safe adoption of software generated by research projects – To help researchers in produce good grid applications

INFSOM-RI Open Grid Forum 20 - Manchester May The timeline FIRST BETA VERSION FINAL BETA VERSION OCTOBER FIRST OFFICIAL VERSION DECEMBER ECHOGRID/EUChinagrid Conf. Beijing April MAYFEBRUARY OGF 20/EGEE UF 7-11 May Belief Conference RIO – BRASIL June OGF Oct QUALIPSO Conference Dec 2007 Now

INFSOM-RI Open Grid Forum 20 - Manchester May ETICS: the project Started in January 2006Started in January 2006 Duration: 2 yearsDuration: 2 years Funded by the European Commission: ~1.4 M€Funded by the European Commission: ~1.4 M€ 5 partners5 partners 2 early adopter projects2 early adopter projects –EGEE/gLite –DILIGENT/gCube Spin-off of the integration and test activities of EGEE-I for gLite middlewareSpin-off of the integration and test activities of EGEE-I for gLite middleware ETICS Service:ETICS Service: –Improve the way Grid software is built and tested –Increase sharing in the Grid communities

INFSOM-RI Open Grid Forum 20 - Manchester May The ETICS Partners Build system, software configuration, service infrastructure, dissemination, EGEE, gLite, project coord. Software configuration, service infrastructure, dissemination Web portals and tools, quality process, dissemination, DILIGENT Test methods and metrics, unit testing tools, EBIT The Condor batch system, distributed testing tools, service infrastructure, NMI

INFSOM-RI Open Grid Forum 20 - Manchester May ETICS is a framework… based on a grid infrastructure where it’s possible to:based on a grid infrastructure where it’s possible to: –Run any scripts over the code (e.g ant ) –Run the unit or component testing –Run a script to deploy one or more services –Run a test to verify the conformity of some API or guidelines (e.g IPV6 compliance) –… –Then collect all these data to –perform (trend) analysis –gather measures and calculate metrics…

INFSOM-RI Open Grid Forum 20 - Manchester May Where ETICS tools apply DESIGN and CODE WRITING CODE BUILD and INTEGRATION TESTING including COMPLIANCE EXECUTION ETICS context

INFSOM-RI Open Grid Forum 20 - Manchester May A Land of nobody? WRONG!WRONG! Many projects on-going (i.e. QUALIPSO for the quality in the open source environment..)Many projects on-going (i.e. QUALIPSO for the quality in the open source environment..) Plenty of build tools (GUMP, Maven, *-forge, Savanna, etc)Plenty of build tools (GUMP, Maven, *-forge, Savanna, etc) –Often open source… Several testing tools (Pulse, Visual Build, etc)Several testing tools (Pulse, Visual Build, etc) –Often commercial…

INFSOM-RI Open Grid Forum 20 - Manchester May Terminology Measure: a value (number or category) assigned to an attribute of an entity (e.g. software component)Measure: a value (number or category) assigned to an attribute of an entity (e.g. software component) Measurement: The act or process of assigning a number or category to an entity to describe an attribute of that entity.Measurement: The act or process of assigning a number or category to an entity to describe an attribute of that entity. Metric: A function whose inputs are software measures and whose output is a single numerical value that can be interpreted as the degree to which software possesses a given attribute that affects its quality.Metric: A function whose inputs are software measures and whose output is a single numerical value that can be interpreted as the degree to which software possesses a given attribute that affects its quality. Perspective: A set of homogenous software attributes that lead to some quality assumptionsPerspective: A set of homogenous software attributes that lead to some quality assumptions

INFSOM-RI Open Grid Forum 20 - Manchester May Grid Quality Assurance Certification Model (GQACM ): a 3D QA Model Let’s suppose A, B, C to be different perspectivesLet’s suppose A, B, C to be different perspectives Common QA Standards are Waterfall-likeCommon QA Standards are Waterfall-like GQACM uses 3 independent perspectives to provide client-friendly resultsGQACM uses 3 independent perspectives to provide client-friendly results –Users can be interested in some attributes and less in others.. –So-made perspectives can have sublevels… A B C A B C

INFSOM-RI Open Grid Forum 20 - Manchester May GQACM: Preliminary remarks GQACM is independent from the ETICS toolsGQACM is independent from the ETICS tools However, any GQACM implementation must use automationHowever, any GQACM implementation must use automation – the most useful characteristic of the model is that it can be automatically applied! GQACM has three perspectives (I.e. independent points of view) to derive the quality degree of a software :GQACM has three perspectives (I.e. independent points of view) to derive the quality degree of a software : – Code – Platforms – Standards C

INFSOM-RI Open Grid Forum 20 - Manchester May The GQACM: Perspectives Code PerspectiveCode Perspective –Usual metrics depending on programming paradigm –… to qualify the degree of goodness of the code (static analysis) Platform PerspectivePlatform Perspective –The type and quantity of different platform (I.e. family of platforms) where software built correctly and can be automatic deployed Standard Perspective Standard Perspective –Software tested for compliance to applicable standards (e.g. BES, JSDL, SRM, WS-I, IPV6) These perspectives are independent each otherThese perspectives are independent each other

INFSOM-RI Open Grid Forum 20 - Manchester May Perspectives composition Code PerspectiveCode Perspective –Coding Conventions –Code Commenting –Other QA metrics related to the corresponding programming language Platform perspectivePlatform perspective –The set of platforms (declared, most used, all available?) Standard perspectiveStandard perspective –The set of (available) standards against the compliance is proven.

INFSOM-RI Open Grid Forum 20 - Manchester May Test execution metrics C1 (method level):C1 (method level): –if C is the cyclomatic complexity (of the method) and T is the number of executed tests then RM1 = C/T C2 (class level):C2 (class level): –if CK RFC is the the number of methods which can be invoked from an instance of the class target to answer a message ) and T is the number of executed tests on it then RM1 = RFC/T. C3 (class level):C3 (class level): –if E is the number of events which interact with the class and T is the number of executed tests then RM4 = E/T C4 (class level):C4 (class level): –if CK NOC is the Number Of Children and T is the number of executed tests on the class then RM8 = NOC/T

INFSOM-RI Open Grid Forum 20 - Manchester May Code comment metrics C5:C5: –if CLOC are lines of comments, then RM5 = CLOC/LOC C6:C6: –if F is the number of functions and C is the number of comment paragraphs then RM6 = F/C C7:C7: –Automatic documentation programs use: e.g. doxygen

INFSOM-RI Open Grid Forum 20 - Manchester May Coding convention metrics 1/2 Only one class definition per header fileOnly one class definition per header file Filename = ClassnameFilename = Classname Filename in lowercaseFilename in lowercase No cycles in “include” operationsNo cycles in “include” operations No C instruction (in C++ programming)No C instruction (in C++ programming) Coherent Programming ConventionCoherent Programming Convention –Curly Brackets positioning –First character in variables names –Lower case in macros names –Coherent use of new lines –Semicolons only at the end of a line

INFSOM-RI Open Grid Forum 20 - Manchester May Coding convention metrics 2/2 –No Never used functions –No Not reachable code paths –No Not defined prototypes –No Never used constants –No Never used macros –No Never used types definitions –No Never used variables –Too frequent “breaks” use –No absence of variables de-alloaction –No Null pointers –Management of one instruction long “if” or one instruction long “cycle” –Shared resources management

INFSOM-RI Open Grid Forum 20 - Manchester May other metrics Errors correction abilityErrors correction ability –C7: errors trend between a test and another after corrections Other software related metricsOther software related metrics –(e.g. CK Metrics for OO software)

INFSOM-RI Open Grid Forum 20 - Manchester May GQACM: Users benefits Users will:Users will: –Have a clear picture about the quality and … –…will be able to select components and software of certified quality level Concretely for different roles:Concretely for different roles: –Developer will be able to certify its software –Integrator will be safe in compose new software starting from existing certified components of the appropriate quality degree –Final User will be sure to use certified software

INFSOM-RI Open Grid Forum 20 - Manchester May Yes, but! I’m not able to add any overhead to my projectI’m not able to add any overhead to my project –This model (and the capability of automate) will reduce the effort in performing continuous build and tests (e.g. coverage tests) on different releases How much costs adopting it?How much costs adopting it? –Nothing, the model will be discussed publicly and the final version will be released under open license (e.g. Creative Commons) –The ETICS framework is provided as a service running on a dedicated infrastructure, free of charge for Research Project (until dec 2007) My organisation is certified ISO/CMMi?My organisation is certified ISO/CMMi? –See next slides…

INFSOM-RI Open Grid Forum 20 - Manchester May ISO 9126 is an international standard for the evaluation of software.is an international standard for the evaluation of software. is divided into four parts which addressis divided into four parts which address –quality model –external metrics –internal metrics –quality in use metrics. The quality model established in the first part of the standard, ISO , classifies software quality in a structured set of characteristics and sub-characteristics divided into attributes.The quality model established in the first part of the standard, ISO , classifies software quality in a structured set of characteristics and sub-characteristics divided into attributes.

INFSOM-RI Open Grid Forum 20 - Manchester May GQACM integration in an ISO 9126 certified organization Facts:Facts: –ISO 9126 define quality attributes. –GQACM support the measure and evaluation of quality metrics. Hypothesis:Hypothesis: –ISO9126 Functionality: adequacy. –ISO9126 asks for correct functions with specific goals. –GQACM may check not dead functions and other coding convention. –ISO9126 Maintainability: all. –ISO9126 asks to develop a software that is easy to analyse, easy to modify, Stable… –GQACM metrics (such as the CK for OO) helps to verify in concrete these requirements

INFSOM-RI Open Grid Forum 20 - Manchester May GQACM integration in an ISO 9126 certified organization ISO9126 Functionality: accuracy.ISO9126 Functionality: accuracy. –ISO9126 asks that a software have to give results according with requirements. –GQACM certification can run functional test (or plug-in external test suits) of declared compliances. ISO9126 Portability: adaptability, installabilityISO9126 Portability: adaptability, installability –ISO9126 asks OS adaptability and installability for software of certified organizations. –GQACM may test deployment of SW on different platforms (HW+OS) Our thesis:Our thesis: –ISO9126 quality metrics can be measured and evaluated based on GQACM –Any tool implementing it and can be integrated as supporting ISO9126 adoption

INFSOM-RI Open Grid Forum 20 - Manchester May CMMI It provides organizations with guidelines for effective organisational processesIt provides organizations with guidelines for effective organisational processes It can guide organisational process improvement across a project, a division, or an entire organizationIt can guide organisational process improvement across a project, a division, or an entire organization It helps to integrate traditionally separate organizational functionsIt helps to integrate traditionally separate organizational functions It sets goals and prioritiesIt sets goals and priorities

INFSOM-RI Open Grid Forum 20 - Manchester May GQACM integration in a CMMi certified organization FactsFacts –CMM is process oriented (focused on the organisation). –GQACM is product oriented. Hypotesis: Hypotesis: –CMM2 –asks to “control and value” –GQACM propose the measures and metrics to be collected, hence controlling and evaluating specific sw related items –CMM3 –asks to “use internal standards and have a reference model” –GQACM proposes code and implementation conventions (how to organise and structure code, how to define tests, etc…)

INFSOM-RI Open Grid Forum 20 - Manchester May GQACM integration in a CMM certified organization CMM4CMM4 –asks to “have measured and controlled processes using quantitative and statistical techniques” –GQACM propose the continuous collection of quality data to perform trends analysis. CMM5CMM5 –asks to have an “improvement process and quality based on measurement” –GQACM provide numerical reference to set the improvement processes!! Our ThesisOur Thesis –GQACM is not in contrast with CMMI and can be integrated in organization as a tool to support CMMI adoption

INFSOM-RI Open Grid Forum 20 - Manchester May ETICS B&T tool and GQACM ETICS:ETICS: –Can perform many measurements and calculate metrics  code perspective –Can build and test software on several distinct platforms  platform perspective –Can test the software compliance with standards  standard perspective –Can automate all the processes (build, deployment, test)

INFSOM-RI Open Grid Forum 20 - Manchester May Where to find more on ETICS? ETICS portal: portal:

INFSOM-RI Open Grid Forum 20 - Manchester May Where to find more on ETICS? ETICS wiki: wiki: ETICS client getting started client getting started ETICS tutorials tutorials Under development

INFSOM-RI Open Grid Forum 20 - Manchester May Conclusions ETICS provides a multi-platform environment for building, testing and checking the quality of distributed and grid applicationsETICS provides a multi-platform environment for building, testing and checking the quality of distributed and grid applications ETICS focus is on automation of builds, tests, reporting, collection and analyse of quality measures … to simplify life when managing complex software management tasksETICS focus is on automation of builds, tests, reporting, collection and analyse of quality measures … to simplify life when managing complex software management tasks We want it to be as simple and effective as possible to use. For this we need your feedback:We want it to be as simple and effective as possible to use. For this we need your feedback:

INFSOM-RI Open Grid Forum 20 - Manchester May Thanks Thanks you for taking the time to learn about ETICS Have a good day Give us feedback