1. www.eu-etics.org INFSOM-RI-026753 Toward standardization of an automated software quality model: the Grid-QCM Adriano Rippa Engineering Ingegneria Informatica S.p.A. ETICS Final Review CERN, Geneva - 15 February 2008

2. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 2 Summary Introduction to QA conceptsIntroduction to QA concepts The starting point of the studyThe starting point of the study The Grid Quality Certification Model (Grid-QCM)The Grid Quality Certification Model (Grid-QCM) Possible Questions, Timeline and feedbackPossible Questions, Timeline and feedback ConclusionsConclusions

3. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 3 Quality Assurance refers to several concepts Quality of the implementation processQuality of the implementation process –High level steps of the software production cycle suggesting what the organization must do (not how) to have effective development processes that “may lead” to good software. Quality of the requirements managementQuality of the requirements management –Correct collection/management of requirements and relation with the customer and stakeholders, to reduce the percentage of failures due to misinterpreted requirements. Quality of the serviceQuality of the service –Performances of the service Finally quality of the software…Finally quality of the software… QUALITY Quality of Product Quality of Process Grid-QCM CMM ITIL ISO – 900x

4. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 4 Some QA Terminology According to ISO 9126 documentation we can define:According to ISO 9126 documentation we can define: Measure: the number or category assigned to an attribute of an entity by making a measurement (sometimes used as synonymous of metric)Measure: the number or category assigned to an attribute of an entity by making a measurement (sometimes used as synonymous of metric) Metric: The defined method to measure an attribute and the scaleMetric: The defined method to measure an attribute and the scale Measurement: The use of a metric to assign a value (which may be a number or category) from a scale to an attribute of an entity)Measurement: The use of a metric to assign a value (which may be a number or category) from a scale to an attribute of an entity)

5. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 5 Quality is a matter of measure! State-of-the-art provide hundreds of metricsState-of-the-art provide hundreds of metrics –The most common –Cyclomatic complexity –Lines of Code –Function Points –Mean Time Between (To) Failure –Bugs density –… –Other approaches (Goal Question Metric - GQM) promote user defined metrics –Anomalies distribution –Effort used to solve anomalies –Cost of not founded anomalies –… An exhaustive list is provided within deliverable D5.7An exhaustive list is provided within deliverable D5.7

6. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 6 Quality Standards There are many standards to asses the quality of the processes of an organizationThere are many standards to asses the quality of the processes of an organization –CMM –ISO family (e.g. ISO 9126, ISO 900X) –ITIL –AQAP But QA means initial investments and managing QA means devote resources to it!But QA means initial investments and managing QA means devote resources to it! Several studies show that lot of companies (e.g. many SMEs) can’t afford the initial effort and don’t recognise the promised increase of value.Several studies show that lot of companies (e.g. many SMEs) can’t afford the initial effort and don’t recognise the promised increase of value. –Only ~70 companies in the world are certified at level 5 –50 of them are in India –source: Gartner –Just 25% of the companies in the world are level 2 or above –source: Kulik, Weber: “Software Metrics Best Practices – 2001” and “Software Metrics State of the Art – 2000”

7. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 7 QA in non commercial short-live project (e.g. research projects) is missing! Current quality assurance standards are useful butCurrent quality assurance standards are useful but –They need lot of time to be applied. The organisation need to be structured and certified. What for the short-live consortia? –They provide only theoretical guidelines which need to be adapted and implemented. What for homogeneity and comparability of results? –It’s hard to systematically verify goodness of results: managing tools needed –They need resources to be devoted to –People need training and certification needs inspections and time to be achieved

8. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 8 QA in grids: our solution Grid-QCM is a model for quality assurance that isGrid-QCM is a model for quality assurance that is –fully automatable in measuring and verifying activities to reduce investments and management effort, –not subjective, to certify the object not the process nor the organization, –product oriented, not process oriented, …easily adoptable within (Grid) short-live Projects…easily adoptable within (Grid) short-live Projects

9. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 9 Grid-QCM: Preliminary Remarks (1/2) Using the ETICS tool people can have Grid-QCM implemented for free in the b&t application.Using the ETICS tool people can have Grid-QCM implemented for free in the b&t application. Grid-QCM has been developed within a Grid project and to asses the quality of grid software research projects but it can be used for any software applications.Grid-QCM has been developed within a Grid project and to asses the quality of grid software research projects but it can be used for any software applications. Grid-QCM has been developed according toGrid-QCM has been developed according to –The feedback received from expert people and potential users –several standards: –Grid-QCM has been described according to ISO standards (e.g. ISO/IEC 25000, 14598) –Grid-QCM has been restructured according to ISO 25041 –Quality attributes has been named using the same terminology as ISO 9126

10. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 10 The ETICS vision Grid-QCM ETICS SW (v.2.0) ETICS grid infrastructure CERN, INFN, UoW (NMI) Definesmetrics Runsmeasures Allows Code analysis Allowsautomation

11. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 11 Grid-QCM: Structure Grid-QCM is structured in Evaluation Modules (EM).Grid-QCM is structured in Evaluation Modules (EM). The set of evaluation techniques are grouped in families. Every family is an Evaluation ModuleThe set of evaluation techniques are grouped in families. Every family is an Evaluation Module 5 Evaluation Modules:5 Evaluation Modules: –Static analysis –Coding style –Structural testing –Functional testing –Standards compliance

12. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 12 Evaluation Modules 1/2 Static analysisStatic analysis –Quality characteristics: –Reliability – maturity –Maintainability – analysability –Maintainability – changeability –Maintainability – testability –Static analysis of classes. Statistics on measures are used as predictor of quality characteristics. Coding styleCoding style –Quality characteristics: –Maintainability – analysability –Static analysis of the source code.

13. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 13 Evaluation Modules 2/2 Structural testingStructural testing –Quality characteristics: –Functionality – accuracy –Reliability – maturity –Structural testing to classes identified more likely to have many errors. Functional testingFunctional testing –Quality characteristics: –Functionality – accuracy –Functionality – interoperability –Reliability – maturity –Portability – adaptability –Portability - installability –Platform compliance and to functional abilities of the software Standards complianceStandards compliance –Quality characteristics: –Functionality – standards compliance

14. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 14 Grid-QCM: Final Score Final score should be provided according to the following schema. The items which should be available for the users are:Final score should be provided according to the following schema. The items which should be available for the users are: –A table summarizing the results –A list of passed and failed tests –All the important information as: –Software product (e.g. name, version, executable code, documentation..) –Platform (name, version, date) –Quality characteristics (name, evaluation result, evaluation module identification) –Standard compliance (for each standard: name, version, date) –Identification of evaluation report (organization, report number, date) –Identification of certification body (organization, contact information) –Certification data (dates, certification number) –Electronic signature of certification record

15. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 15 Possible Criticism, Our Answer I can’t add any overhead to my projectI can’t add any overhead to my project –This model (and the capability of automate) reduce the effort in performing continuous build and test activities (e.g. coverage tests) on different SW releases. What about the cost?What about the cost? –Using the ETICS tool people can have the model implemented for free! My organisation is certified ISO/CMMI so I…My organisation is certified ISO/CMMI so I… –The model is a standalone quality certification model. –However it can be easily integrated in yet ISO/CMMI certified organisations.

16. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 16 Current ETICS metrics and Grid-QCM MetricsType Programming language Grid-QCM: INVOLVED EMs complexitystatic Java, Python EM CODING STYLE EM STATIC ANALYSIS design quality staticJava EM CODING STYLE EM STATIC ANALYSIS nr of „possible” bugs static Java, C/C++, Python, Perl, PHP EM CODING STYLE EM STRUCTURAL TESTING nr of „possible” bugs dynamicC/C++ EM STRUCTURAL TESTING unit test dynamic Java, Python EM FUNCTIONAL TESTING EM STRUCTURAL TESTING coveragedynamicJava EM FUNCTIONAL TESTING EM STRUCTURAL TESTING profiling information dynamic Java, C/C++ EM FUNCTIONAL TESTING

17. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 17 The timeline OCTOBER DECEMBER ECHOGRID/EUChinagrid Conference 24-25 April Beijing (CHINA) MAYFEBRUARY OGF 20/EGEE UF 7-11 May Manchester (UK) Belief Conference 25-28 June Rio de Janeiro (BRA) OGF 21 15-19 Oct Seattle (USA) QUALIPSO Conference 16-17 Jan 2008 Rome (IT) Now EGEE ’07 1-5 Oct Budapest (Hun) EELA 3° Conference 3-5 Dec 2007 Catania (IT) ESA 3rd GRID & e- Collaboration Workshop 16-17 Jan 2008 Frascati (IT)

18. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 18 Grid-QCM: feedback OGF 20OGF 20 Possibility to automate Possibility to automate Unclear formal name of the model Unclear formal name of the model Criticism: GQACM was difficult to rememberCriticism: GQACM was difficult to remember Solution: New name: Grid-QCMSolution: New name: Grid-QCM Belief/EELA ConferenceBelief/EELA Conference Automation, CMMI/ISO compatibility Automation, CMMI/ISO compatibility Structure of some parts of the model Structure of some parts of the model Criticism: Metrics not well split according to their roleCriticism: Metrics not well split according to their role Solution: Organisation in Evaluation ModelsSolution: Organisation in Evaluation Models EGEE’07EGEE’07 Automation, ETICS + Grid-QCM = free, people asked for specific information Automation, ETICS + Grid-QCM = free, people asked for specific information Metrics for the process Metrics for the process Criticism: Lack of process metricsCriticism: Lack of process metrics Solution: it’s out of the scopeSolution: it’s out of the scope

19. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 19 Grid-QCM: feedback OGF 21OGF 21 Automation, CMMI/ISO compatibility, people asked for specific information Automation, CMMI/ISO compatibility, people asked for specific information ISO9000 compatibility ISO9000 compatibility Criticism: no mention about ISO9000 compatibilityCriticism: no mention about ISO9000 compatibility Solution: it’s out of the scopeSolution: it’s out of the scope QualiPSo ConferenceQualiPSo Conference Automation, Used Standards, ETICS + Grid-QCM = free Many people interested in specific information Automation, Used Standards, ETICS + Grid-QCM = free Many people interested in specific information

20. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 20 Conclusions Grid-QCM is a certification modelGrid-QCM is a certification model –Automatable –Implemented for free in the ETICS tool –Not in contradiction with classical standards –Ready to interact with classical standards –Not limited to research projects –Not limited to grid software Grid-QCM require less human effort to be used because it is almost fully automatableGrid-QCM require less human effort to be used because it is almost fully automatable ETICS tool is ready to implement Grid-QCMETICS tool is ready to implement Grid-QCM During the ETICS 2 project, if approved,During the ETICS 2 project, if approved, –Grid-QCM will be validated on-the-field with at least four projects –Grid-QCM will be proposed for standardisation under ISO.

21. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 21 Q&A http://www.eu-etics.org

22. INFSOM-RI-026753 ETICS Final Review - Grid-QCM - CERN, 15 February 2008 22 Grid-QCM: Summarizing Table example Consolidated eval. result EM Static EM Coding EM structural EM Functional EM Std compliance EM eval. result MMGGGY FunctionalityGGG AccuracyGGG InteroperabilityGG ComplianceY(Y) ReliabilityMMGG MaturityMMGG MaintainabilityMMG AnalyzabilityMMG ChangeabilityMM TestabilityMM PortabilityGG AdaptabilityGG InstallabilityGG E = Excellent G = Good M = Medium F = Fair P = Poor