1. www.eu-etics.org INFSOM-RI-026753 A Quality Certification Model for Grid Research Projects the ETICS feasibility Study Adriano Rippa (adriano.rippa@eng.it) adriano.rippa@eng.it - Engineering Ingegneria Informatica s.p.a. - Engineering Ingegneria Informatica s.p.a. On behalf of the ETICS Project

2. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 2 Summary Introduction to QA conceptsIntroduction to QA concepts The starting point of the studyThe starting point of the study The proposed Grid Quality Certification Model (GQCM)The proposed Grid Quality Certification Model (GQCM) Comparing GQCM and other QA standardsComparing GQCM and other QA standards Timeline and publicationsTimeline and publications ConclusionsConclusions

3. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 3 Quality Assurance refers to several concepts Quality of the implementation processQuality of the implementation process –High level steps of the software production cycle suggesting what the organization must do (not how) to have effective development processes that “may lead” to good software. Quality of the requirements managementQuality of the requirements management –Correct collection/management of requirements and relation with the customer and stakeholders, to reduce the percentage of failures due to misinterpreted requirements. Quality of the serviceQuality of the service –Performances and correctness of the service Finally quality of the software…Finally quality of the software… QUALITY Quality of Product Quality of Process GQCM CMM ITIL ISO – 900x Could be removed.

4. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 4 Some QA Terminology According to ISO 9126 documentation we can define:According to ISO 9126 documentation we can define: Measure: the number or category assigned to an attribute of an entity by making a measurement (sometimes used as synonymous of metric)Measure: the number or category assigned to an attribute of an entity by making a measurement (sometimes used as synonymous of metric) Metric: The defined method to measure an attribute and the scaleMetric: The defined method to measure an attribute and the scale Measurement: The use of a metric to assign a value (which may be a number or category) from a scale to an attribute of an entity)Measurement: The use of a metric to assign a value (which may be a number or category) from a scale to an attribute of an entity)

5. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 5 Quality is a matter of measure! State-of-the-art provide hundreds of metricsState-of-the-art provide hundreds of metrics –The most common –Cyclomatic complexity –Lines of Code –Function Points –Mean Time Between(to) failure –Bugs density –… –Other approaches (Goal Question Metric - GQM) promote user defined metrics –Anomalies distribution –Effort used to solve anomalies –Cost of not founded anomalies –…

6. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 6 Quality Standards There are many standards to asses the quality of the processes of an organizationThere are many standards to asses the quality of the processes of an organization –CMM –ISO family –ITIL –AQAP But QA means initial investments and managing QA means devote resources to it!But QA means initial investments and managing QA means devote resources to it! Several studies show that lot of companies (e.g. many Small/Medium Enterprises) can’t afford the initial effort and don’t recognise the promised increase of value.Several studies show that lot of companies (e.g. many Small/Medium Enterprises) can’t afford the initial effort and don’t recognise the promised increase of value. –Only ~70 companies in the world are certified at level 5 –50 of them are in India –Only 25% of the companies in the world are level 2 or above

7. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 7 QA in research project is missing! Current quality assurance standards are useful butCurrent quality assurance standards are useful but –They need lot of time to be applied. The organisation need to be structured and certified. What for the short- live consortia? –They provide only theoretical guidelines which need to be adapted and realised. What for the objectivity? –It’s hard to systematically verify goodness of results: managing tools needed –They need resources to be devoted to –People need training and certification needs inspections and time to be achieved

8. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 8 QA in grids: our proposal GQCM is a model for quality assurance that isGQCM is a model for quality assurance that is –fully automatable in measuring and verifying activities to reduce investments and management effort, –not subjective, to certify the object not the process nor the organization, –product oriented, not process oriented, …easily adoptable within (Grid) Research Projects…easily adoptable within (Grid) Research Projects

9. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 9 GQCM: Preliminary Remarks (1/2) GQCM is independent from the ETICS toolsGQCM is independent from the ETICS tools Any GQCM implementation needs just “automation”Any GQCM implementation needs just “automation” GQCM has been developed within a Grid project and to asses the quality of grid software research projects but it can be used for any software applicationsGQCM has been developed within a Grid project and to asses the quality of grid software research projects but it can be used for any software applications GQCM has been rounded off according to the gotten feedbacks from expert people and potential users who confirmed the usefulness of the model to reduce costs and time spend and to have something of more practical.GQCM has been rounded off according to the gotten feedbacks from expert people and potential users who confirmed the usefulness of the model to reduce costs and time spend and to have something of more practical.

10. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 10 GQCM: Preliminary Remarks (2/2) GQCM has been developed according to several standards:GQCM has been developed according to several standards: –GQCM has been described according to several ISO standards (e.g. ISO/IEC 25000, 14598) –GQCM has been restructured according to ISO 25041 –Quality attributes has been named using the same terminology as ISO 9126

11. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 11 GQCM: Structure GQCM is structured in Evaluation Modules (EM).GQCM is structured in Evaluation Modules (EM). The set of evaluation techniques are grouped in families. Every family is an EMThe set of evaluation techniques are grouped in families. Every family is an EM 5 EMs:5 EMs: –EM: Static analysis –EM: Coding style –EM: Structural testing –EM: Functional testing –EM: Standards compliance

12. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 12 Evaluation Modules 1/5 EM Static analysisEM Static analysis –Quality characteristics: –Reliability – maturity –Maintainability – analysability –Maintainability – changeability –Maintainability – testability –Technique: –Static analysis of classes. Statistics on measures are used as predictor of quality characteristics. –Applicability: –Object oriented programming languages. –Input: –source code, each class of the code is analysed This and the next 4 slides could became 3 instead than 5 removing the applicability and the input susections

13. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 13 Evaluation Modules 2/5 EM Coding styleEM Coding style –Quality characteristics: –Maintainability – analysability –Technique: –Static analysis of the source code. –Applicability: –Most programming language –Input: –source code

14. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 14 Evaluation Modules 3/5 EM Structural testingEM Structural testing –Quality characteristics: –Functionality – accuracy –Reliability – maturity –Technique: –Structural testing. The intention of this EM is to test specific classes that are identified by static measures as being statistically more likely to have many errors. –Applicability: –Object oriented programming languages –Input: –source code

15. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 15 Evaluation Modules 4/5 EM Functional testingEM Functional testing –Quality characteristics: –Functionality – accuracy –Functionality – interoperability –Reliability – maturity –Portability – adaptability –Portability - installability –Technique: –Functional testing –The purpose of this EM is both to check platform compliance and to check to functional abilities of the software product. –Applicability: –General –Input: –compiled code –user documentation

16. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 16 Evaluation Modules 5/5 EM Standards complianceEM Standards compliance –Quality characteristics: –Functionality – standards compliance –Technique: –Functional testing –This EM has a good potential for automating the test. –Applicability: –Standard specific –Input: –compiled code

17. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 17 GQCM: Final Score Final score should be provided according to the following schema. The items which should be available for the users are:Final score should be provided according to the following schema. The items which should be available for the users are: –A table summarizing the results –A list of passed and non passed tests –All the important information as: –Software product (e.g. name, version, executable code, documentation..) –Platform (name, version, date) –Quality characteristics (name, evaluation result, evaluation module identification) –Standard compliance (for each standard: name, version, date) –Identification of evaluation report (organization, report number, date) –Identification of certification body (organization, contact information) –Certification data (dates, certification number) –Electronic signature of certification record

18. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 18 GQCM: Summarizing Table example Consolidated eval. result EM Static EM Coding EM structural EM Functional EM Std compliance EM eval. resultMMGGGY FunctionalityGGG AccuracyGGG InteroperabilityGG ComplianceY(Y) ReliabilityMMGG MaturityMMGG MaintainabilityMMG AnalyzabilityMMG ChangeabilityMM TestabilityMM PortabilityGG AdaptabilityGG InstallabilityGG G = Good M = Medium P = Poor

19. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 19 GQCM: FAQ I can’t add any overhead to my projectI can’t add any overhead to my project –This model (and the capability of automate) will reduce the effort in performing continuous build and test activities (e.g. coverage tests) on different releases How much costs adopting it?How much costs adopting it? –Nothing, the model will be discussed publicly and the final version will be released under open license (e.g. Creative Commons) –The ETICS framework is provided as a service running on a dedicated infrastructure, free of charge for Research Project My organisation is certified ISO/CMMi so I…?My organisation is certified ISO/CMMi so I…? –See next slides…

20. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 20 GQCM integration in an ISO 9126 certified organization Facts:Facts: –ISO 9126 define quality attributes –GQCM support the measure and evaluation of quality metrics Why are they compatible each other?Why are they compatible each other? –Many ISO9126 aspects can be evaluated by GQCM  GQCM apply what ISO 9126 asks –Any tool implementing GQCM and can be integrated as supporting ISO9126 adoption SoSo –GQCM can be both an independent quality model and a “tool” to be easily compliant with ISO 9126

21. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 21 GQCM integration in a CMMi certified organization FactsFacts –CMM is process oriented (it gives guidelines). –GQCM is product oriented. Hypothesis: Hypothesis: CMM2CMM2 –asks to “control and value” CMM3CMM3 –asks to “use internal standards and have a reference model” CMM4CMM4 –asks to “have measured and controlled processes using quantitative and statistical techniques” CMM5CMM5 –asks to have an “improvement process and quality based on measurement”. Our ThesisOur Thesis –GQCM is not in contrast with CMMI and can be integrated in organization as a tool to support CMMI adoption

22. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 22 Current ETICS calculable metrics and GQCM MetricsType Programming language GQCM: INVOLVED EMs complexitystaticJava EM STATIC ANALYSIS design quality staticJava EM STATIC ANALYSIS EM STRUCTURAL TESTING nr of security bugs static C/C++, Python, Perl, PHP EM CODING STYLE EM STRUCTURAL TESTING nr of warnings staticJava EM CODING STYLE EM STRUCTURAL TESTING coveragedynamicJava unit test dynamicJava EM FUNCTIONAL TESTING unit test reports staticJava EM FUNCTIONAL TESTING profilerdynamicJava

23. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 23 The timeline OCTOBER DECEMBER ECHOGRID/EUChinagrid Conference 24-25 April Beijing (CHINA) MAYFEBRUARY OGF 20/EGEE UF 7-11 May Manchester (UK) Belief Conference 25-28 June Rio de Janeiro (BRA) OGF 21 15-19 Oct Seattle (USA) QUALIPSO Conference 16-17 Jan 2008 Rome (IT) Now EGEE ’07 1-5 Oct Budapest (Hun) EELA 3° Conference 3-5 Dec 2007 Catania (IT)

24. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 24 GQCM: publications 2 papers submitted and accepted2 papers submitted and accepted When/WhereWhen/Where EELA third conferenceEELA third conference 3-5 December 20073-5 December 2007 Catania/ItalyCatania/Italy QUALIPSO first conferenceQUALIPSO first conference 16-17 January 200816-17 January 2008 Rome/ItalyRome/Italy A quality oriented conferenceA quality oriented conference

25. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 25 GQCM: feedbacks The model has got good feedbacks and acknowledgementsThe model has got good feedbacks and acknowledgements People are asking for more informationPeople are asking for more information Some issues about the name of the model (solved)Some issues about the name of the model (solved) One issue about ISO9000 compliance (outside the scope?)One issue about ISO9000 compliance (outside the scope?) Requests for metrics about the process  outside the scope?Requests for metrics about the process  outside the scope?

26. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 26 Conclusions GQCM is a certification modelGQCM is a certification model –Automatable –Free –Not in contrast with classical standards –Ready to interact with classical standards –Not limited to the ETICS build and test tool –Not limited to research projects –Not limited to grid software GQCM require less human effort to be used because it is almost fully automatableGQCM require less human effort to be used because it is almost fully automatable ETICS tool is ready to implement GQCMETICS tool is ready to implement GQCM

27. INFSOM-RI-026753 ETICS 2nd Review - CERN 15/02/2008 27 Q&A http://www.eu-etics.org