OGF 23 Quality for Grid & Grid for Quality: sharing experiences and perspectives of Grid and Open Source communities Engineering Ingegneria Informatica S.p.A. INFSOM-RI Adriano Rippa Barcelona, 2nd June 2008 Grid-QCM Automation in Quality Assurance

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Summary The project Introduction to QA concepts The starting point of the study The Grid Quality Certification Model (Grid-QCM) The Grid-QCM dissemination timeline Some FAQs Grid-QCM adoption in a CMMi lev.3 organization Conclusions

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 The ETICS2 project

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 ETICS2 The ETICS second phase Started in March 2008 Duration: 2 years Funded by the European Commission Extended name: eInfrastructure for Testing Integration and Configuration of Software 2 Purpose: Automate the way a software is built, tested and certified. It provides Fully automatic build, test and certification tools A repository for the artefacts A Grid based solution optimal performances and multiple platform and test options

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Introduction to QA concepts

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Grid-QCM focus Product VS Process Quality refers to several concepts like: Quality of the Implementation process Quality of the requirements management Quality of the service Quality of the software

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Terminology Product VS Process According to ISO 9126 documentation we can define: Measure: the number or category assigned to an attribute of an entity by making a measurement (sometimes used as synonymous of metric) Metric: The defined method to measure an attribute and the scale Measurement: The use of a metric to assign a value (which may be a number or category) from a scale to an attribute of an entity)

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 The starting point

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 State of the Art We have tons of stuff in the SoA! State of the art provides Hundreds of metrics Approaches to define users own metrics Many standards to asses the quality of processes and products So, whats the matter?!?

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 The matter Effort and resources QA means initial investments and Managing QA means devote resources to it Many companies (e.g. SMEs) cannot afford the initial effort and do not recognise the promised increase of value. Only ~70 companies in the world are CMMi lev.5 certified 50 of them are in India source: Gartner Just 25% of the companies in the world are CMMi lev.2 or above source: Kulik, Weber: Software Metrics Best Practices – 2001 and Software Metrics State of the Art – 2000

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Current standards issues Why studying for one more QA certification model Current quality assurance standards are useful but They need lot of time to be applied. People need training Certification needs inspections to be achieved What for the short-lived consortia? They provide only theoretical guidelines which need to be adapted and implemented. What for homogeneity and comparability of results? Its hard to systematically verify goodness of results: managing tools needed

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 The Grid Quality Certification Model

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Grid-Quality Certification Model Our proposal to help developers Grid-QCM is fully automatable in measuring and verifying activities to reduce investments and management effort not subjective, to certify the object not the process nor the organization product oriented, not process oriented easily adoptable within short-lived Projects

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Grid-Quality Certification Model preliminary remarks Grid-QCM has been developed according to The feedback received from expert people and potential users ISO standards guidelines: Grid-QCM has been described according to ISO 25000, Grid-QCM has been structured according to ISO Quality attributes has been named according to ISO 9126

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Grid-Quality Certification Model The ETICS vision Grid-QCM ETICSSoftware ETICS grid infrastructure Definesmetrics Runsmeasures AllowsTest Allowsautomation

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 The structure Evaluation modules Grid-QCM is structured in Evaluation Modules (EM). The set of evaluation techniques are grouped in families. Every family is an Evaluation Module 5 Evaluation Modules: Static analysis Coding style Structural testing Functional testing Standards compliance

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Evaluation Modules 1/2 Static analysis Quality characteristics: Reliability – maturity Maintainability – analysability Maintainability – changeability Maintainability – testability Static analysis of classes. Statistics on measures are used as predictor of quality characteristics. Coding style Quality characteristics: Maintainability – analysability Static analysis of the source code.

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Evaluation Modules 2/2 Structural testing Quality characteristics: Functionality – accuracy Reliability – maturity Structural testing to classes identified more likely to have many errors. Functional testing Quality characteristics: Functionality – accuracy Functionality – interoperability Reliability – maturity Portability – adaptability Portability - installability Platform compliance and to functional abilities of the software Standards compliance Quality characteristics : Functionality – standards compliance

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Grid-QCM: Final Score Final score will consist in: A table summarizing the results with several possible views A list of passed and failed tests All the important information as: Software product Platform Quality characteristics Standard compliance Identification of evaluation report Identification of certification body (organization, contact information) Certification data (dates, certification number) Electronic signature of certification record

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 The Grid-QCM dissemination timeline

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Grid-QCM worldwide Since 2007 OCT DEC ECHOGRID/EUChinagrid Conference April Beijing (CHINA) MAY FEB 07 OGF 20/EGEE UF 7-11 May Manchester (UK) Belief Conference June Rio de Janeiro (BRA) OGF Oct Seattle (USA) QUALIPSO Conference Jan 2008 Rome (IT) Now EGEE Oct Budapest (Hun) EELA 3° Conference 3-5 Dec 2007 Catania (IT) ESA 3rd GRID & e- Collaboration Workshop Jan 2008 Frascati (IT) JAN 08 JUNE OGF June Barcelona (SPA) ETICS review 15 Feb Geneva (CH)

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Some FAQs

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Some Questions and Answers Yes, but… Whats good about Grid-QCM for my project? This model (and the capability of automation) reduces the effort in performing continuous build and test activities (e.g. coverage tests) on different SW releases. What are the costs? Using the ETICS tool people can have the model integrated in the build and test application free of charge My organisation is certified ISO/CMMI can I use Grid-QCM? The model is a standalone quality certification model. However it can be easily integrated in yet ISO/CMMI certified organisations.

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Grid-QCM in a CMMi lev.3 organisation

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Grid-QCM in a CMMi lev.3 certified company The Engineering experience Engineering Ingegneria Informatica S.p.A is certified as follows: CMMi lev. 3 For all production divisions ISO 9001 For all the ICT sector in which the organization is involved Since 1994 ( ISO9001:2000 from December 2002) NATO AQAP 2110/160 Since December 1996 (from 1996 to 2005 there were different rules) The whole Grid-QCM work is born from internal needs of The research area The production area

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Engineering QA tools Keeping the certification Engineering tools to reach and maintain the CMMi lev. 3 certification: Spago4Q ( measures processes, performances and bugs through the integration with other tools Internally developed and available for free Examples of calculated metrics: Test Coverage Anomalies distribution Effort used to solve anomalies Cost of not founded anomalies Metrics about requirements Metrics about risk management …

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 It is not enough… What ETICS and Grid-QCM can do for ENG …Some quality aspects are not covered Grid-QCM + the ETICS suite Measures other quality aspects such as: Quality of the code Compliance with standards Compliance with OS … Can be integrated with the Spago4Q suite So, Engineering is managing the activities to make Grid-QCM + ETICS a tool to keep the CMMi lev.3 certification

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Conclusions

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Conclusions And further steps Grid-QCM is a certification model Automatable Implemented by default in the ETICS tool Ready to interact with classical standards Adopting Grid-QCM requires little human effort During the ETICS 2 project Grid-QCM is going to be validated on-the-field with at first two (Dec. 08) and then four projects (Dec. 09) Grid-QCM will be proposed for the ISO standardisation (2009)

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June addresses We wish Grid-QCM and ETICS to be as simple and effective as possible to use. For this we need your feedback

Grid-QCM Automation in Quality Assurance INFSOM-RI Open Grid Forum 23 Barcellona, 2° June 2008 Q&A Thank you!