Toward a Quality Certification Model for Grid Research Projects: an ETICS feasibility Study Andrea Manieri Adriano Rippa - Engineering Ingegneria Informatica s.p.a. On behalf of ETICS Project Chi siamo, cosa facciamo

The starting point Current quality assurance standards are useful but: are process oriented provide theoretical guidelines on the organization The problem: Guidelines can be applied but it’s hard to systematically verify goodness of results Our solution: A quality assurance model fully automatable in measuring and verifying activities not subjective ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

ECHOGRID/EUChinagrid The timeline Now OGF 21 15-19 Oct ALL HANDS MEETING 23-25 May OGF 20/EGEE UF 7-11 May FIRST BETA VERSION FINAL BETA VERSION FIRST OFFICIAL VERSION FEBRUARY MAY OCTOBER DECEMBER ECHOGRID/EUChinagrid Conf. Beijing 24-25 April QUALIPSO Conference Dec 2007 Belief Conference RIO – BRASIL 25-28 June ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Terminology Measure: a value (number or category) assigned to an attribute of an entity (e.g. software component) Measurement: The act or process of assigning a number or category to an entity to describe an attribute of that entity. Metric: A function whose inputs are software measures and whose output is a single numerical value that can be interpreted as the degree to which software possesses a given attribute that affects its quality. Perspective: A set of homogenous software attributes that lead to some quality assumptions ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Gird Quality Assurance Certification Model (GQACM ): a 3D QA Model Let’s suppose A, B, C to be different perspectives Most of Common QA Standards (e.g. CMMI) are Waterfall-like GQACM uses 3 independent perspectives to provide client-friendly results Users can be interested in some attributes and less in others.. So-made perspectives can have sublevels… A B C A C B ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

GQACM: Preliminary remarks GQACM is independent from the ETICS tools However, any GQACM implementation must use automation – the most useful characteristic of the model is that it can be automatically applied! GQACM has three perspectives (I.e. independent points of view) to derive the quality degree of a software : Code and test Platforms Standards ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Perspectives composition Code Test Perspective: to qualify the goodness of the code and of the test Coding Conventions Code Commenting Other QA metrics related to the corresponding programming language Unit and functional tests Platform perspective: to qualify B&T requirements on different platforms The set of GQACM available platforms Standard perspective: to qualify the compliance with standards (e.g. IPV6, BES, JSDL, SRM, WS-I) The set of GQACM available standards ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Test execution metrics Cyclomatic complexity CCT (method level): if C is the cyclomatic complexity (of the method) and T is the number of executed tests then CCT = C/T RFCT (class level): if CK RFC is the the number of methods which can be invoked from an instance of the class target to answer a message ) and T is the number of executed tests on it then RFCT = RFC/T ET (class level): if E is the number of events which interact with the class and T is the number of executed tests then ET = E/T NOCT (class level): if CK NOC is the Number Of Children of a class and T is the number of executed tests on the class then NOCT = NOC/T ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Code comment metrics FC: AD: if F is the number of functions and C is the number of comment paragraphs then FC = F/C AD: Automatical documentation programs use: e.g. doxygen ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Coding convention metrics 1/2 Only one class definition per header file Filename = Classname Filename in lowercase No cycles in “include” operations No C instruction (in C++ programming) Coherent Programming Convention Curly Brackets positioning First character in variables names Lower case in macros names Coherent use of new lines Semicolons only at the end of a line ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Coding convention metrics 2/2 No Never used functions No Not reachable code paths No Not defined prototypes No Never used constants No Never used macros No Never used types definitions No Never used variables Too frequent “breaks” use No absence of variables de-alloaction No Null pointers Management of one instruction long “if” or one instruction long “cycle” Shared resources management ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

other metrics Other software related metrics (e.g. Chidamber and Kemerer (CK) Metrics for OO software) ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Final Score1/2 For every perspective must be genrated: a score between 0 and 100 a list of passed and not passed tests The way to calculate the score is different for every perspective. For code perspective The score is simply calculated using this mathematical formula: (X/N)*100 where X is the number of successes, N the number of used metrics ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Final Score 2/2 For platform perspective For standard perspective If DP is the number of declared compliant platforms, CP the number of really compliant platforms, TP the number of GQACM available platforms , CTP the total number of compliant platforms , then the score can be calculated with this mathematical formula: (CP/DP)*50+(CTP/TP)*50 For standard perspective DS is the number of declared compliant standards, CS is the number of really compliant standards, TS is the number of available standards, CTS is the total number of compliant standards (declared + not declared), then score can be calculated with this mathematical formula: (CS/DS)*50+(CTS/TS)*50 ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Score visualization The graphical view of the score is related to the used test tool In general, all tools have to use at least the “classical version”: Final score: <total>/<max possible score> ([<perspective><score>]) Where Perspective can be: Code (C), Platforms (P), Standards (S) Score: 0-100 (where 0 is 0% compliant, 100 is 100% compliant) Total score: sum of the scores E.g. component X: GQACM score = 151/300 (C90-P51-S10) ETICS could complement the standard basic visualization using a 3d graph to give the user a better feeling interface ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

GQACM integration in an ISO 9126 certified organization Facts: ISO 9126 define quality attributes . GQACM support the measure and evaluation of quality metrics. Hypothesis: ISO9126 Portability: adaptability, installability ISO9126 asks OS adaptability and installability for software of certified organizations. GQACM may test deployment of SW on different platforms (HW+OS) ISO9126 Maintainability: all. ISO9126 asks to develop a software that is easy to analyse, easy to modify, Stable GQACM metrics (such as the CK for OO or coding convetions metrics) helps to verify in concrete these requirements ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

GQACM integration in an ISO 9126 certified organization ISO9126 Functionality: accuracy. ISO9126 asks that a software have to give results according with requirements. GQACM certification can run functional test (or plug-in external test suits) of declared compliances. ISO9126 Functionality: adequacy. ISO9126 asks for correct functions with specific goals. GQACM may check not dead functions. Our thesis: Many ISO9126 quality metrics can be measured and evaluated based on GQACM Any tool implementing it and can be integrated as supporting ISO9126 adoption ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

GQACM integration in a CMMi certified organization Facts CMM is process oriented (it gives guidelines). GQACM is product oriented. Hypotesis: CMM2 asks to “control and value” GQACM propose the measures and metrics to be collected, hence controlling and evaluating specific sw related items CMM3 asks to “use internal standards and have a reference model” GQACM proposes code and implementation conventions (how to organise and structure code, how to define tests, etc.) ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

GQACM integration in a CMM certified organization asks to “have measured and controlled processes using quantitative and statistical techniques” GQACM propose the continuous collection of quality data to perform trends analysis. CMM5 asks to have an “improvement process and quality based on measurement” GQACM provide numerical reference to set the improvement processes!! Our Thesis GQACM is not in contrast with CMMI and can be integrated in organization as a tool to support CMMI adoption ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

ETICS B&T tool and GQACM Could perform many measurements and calculate metrics (e.g. JPmdPlugin and JFindbugsPlugin)  Code and test perspective Can build and test software on several distinct platforms  Platform perspective Could test the software compliance with standards  Standard perspective Can automate all the processes (build, deployment, test) ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Conclusions GQACM is a certification model Automatable 3D Not in contrast with classical standards Free GQACM doesn’t require big efforts to be used ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007

Questions? Feedbacks? Thanks http://www.eu-etics.org ETICS 3rd All-Hands Meeting - Universiti of Wisconsin, Madison 23-25/05/2007