1. Toward standardization of an automated software quality model: the Grid-QCM
Adriano Rippa
Engineering Ingegneria Informatica S.p.A.
ETICS Final Review
CERN, Geneva - 15 February 2008

2. Summary Introduction to QA concepts The starting point of the study
The Grid Quality Certification Model (Grid-QCM)
Possible Questions, Timeline and feedback
Conclusions
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

3. Quality Assurance refers to several concepts
Quality of the implementation process
High level steps of the software production cycle suggesting what the organization must do (not how) to have effective development processes that “may lead” to good software.
Quality of the requirements management
Correct collection/management of requirements and relation with the customer and stakeholders, to reduce the percentage of failures due to misinterpreted requirements.
Quality of the service
Performances of the service
Finally quality of the software…
QUALITY
Quality of Product
Quality of Process
CMM
Grid-QCM
ITIL
ISO – 900x
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

4. Some QA Terminology According to ISO 9126 documentation we can define:
Measure: the number or category assigned to an attribute of an entity by making a measurement (sometimes used as synonymous of metric)
Metric: The defined method to measure an attribute and the scale
Measurement: The use of a metric to assign a value (which may be a number or category) from a scale to an attribute of an entity)
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

5. Quality is a matter of measure!
State-of-the-art provide hundreds of metrics
The most common
Cyclomatic complexity
Lines of Code
Function Points
Mean Time Between (To) Failure
Bugs density …
Other approaches (Goal Question Metric - GQM) promote user defined metrics
Anomalies distribution
Effort used to solve anomalies
Cost of not founded anomalies
An exhaustive list is provided within deliverable D5.7
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

6. Quality Standards
There are many standards to asses the quality of the processes of an organization
CMM
ISO family (e.g. ISO 9126, ISO 900X)
ITIL
AQAP
But QA means initial investments and managing QA means devote resources to it!
Several studies show that lot of companies (e.g. many SMEs) can’t afford the initial effort and don’t recognise the promised increase of value.
Only ~70 companies in the world are certified at level 5
50 of them are in India
source: Gartner
Just 25% of the companies in the world are level 2 or above
source: Kulik, Weber: “Software Metrics Best Practices – 2001” and “Software Metrics State of the Art – 2000”
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

7. Current quality assurance standards are useful but
QA in non commercial short-lived project (e.g. research projects) is missing!
Current quality assurance standards are useful but
They need lot of time to be applied. The organisation need to be structured and certified. What for the short-lived consortia?
They provide only theoretical guidelines which need to be adapted and implemented. What for homogeneity and comparability of results?
It’s hard to systematically verify goodness of results: managing tools needed
They need resources to be devoted to
People need training and certification needs inspections and time to be achieved
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

8. QA in grids: our solution
Grid-QCM is a model for quality assurance that is
fully automatable in measuring and verifying activities to reduce investments and management effort,
not subjective, to certify the object not the process nor the organization,
product oriented, not process oriented,
…easily adoptable within (Grid) short-lived Projects
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

9. Grid-QCM: Preliminary Remarks (1/2)
Using the ETICS tool people can have Grid-QCM implemented by default in the b&t application.
Grid-QCM has been developed within a Grid project and to asses the quality of grid software research projects but it can be used for any software applications .
Grid-QCM has been developed according to
The feedback received from expert people and potential users
several standards:
Grid-QCM has been described according to ISO standards (e.g. ISO/IEC 25000, 14598)
Grid-QCM has been restructured according to ISO 25041
Quality attributes has been named using the same terminology as ISO 9126
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

10. ETICS grid infrastructure
The ETICS vision
Allows
Test
Grid-QCM
Allows
automation
ETICS SW (v.2.0)
Defines
metrics
ETICS grid infrastructure
CERN, INFN, UoW (NMI)
Runs
measures
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

11. Grid-QCM: Structure Static analysis Coding style Structural testing
Grid-QCM is structured in Evaluation Modules (EM).
The set of evaluation techniques are grouped in families. Every family is an Evaluation Module
5 Evaluation Modules:
Static analysis
Coding style
Structural testing
Functional testing
Standards compliance
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

12. Evaluation Modules 1/2 Static analysis Coding style
Quality characteristics:
Reliability – maturity
Maintainability – analysability
Maintainability – changeability
Maintainability – testability
Static analysis of classes. Statistics on measures are used as predictor of quality characteristics.
Coding style
Static analysis of the source code.
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

13. Evaluation Modules 2/2 Structural testing Functional testing
Quality characteristics:
Functionality – accuracy
Reliability – maturity
Structural testing to classes identified more likely to have many errors.
Functional testing
Functionality – interoperability
Portability – adaptability
Portability - installability
Platform compliance and to functional abilities of the software
Standards compliance
Functionality – standards compliance
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

14. Grid-QCM: Final Score
Final score should be provided according to the following schema. The items which should be available for the users are:
A table summarizing the results
A list of passed and failed tests
All the important information as:
Software product (e.g. name, version, executable code, documentation..)
Platform (name, version, date)
Quality characteristics (name, evaluation result, evaluation module identification)
Standard compliance (for each standard: name, version, date)
Identification of evaluation report (organization, report number, date)
Identification of certification body (organization, contact information)
Certification data (dates, certification number)
Electronic signature of certification record
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

15. Possible Criticism, Our Answer
I can’t add any overhead to my project
This model (and the capability of automate) reduce the effort in performing continuous build and test activities (e.g. coverage tests) on different SW releases.
What about the cost?
Using the ETICS tool people can have the model integrated in the b&t application
My organisation is certified ISO/CMMI so I…
The model is a standalone quality certification model.
However it can be easily integrated in yet ISO/CMMI certified organisations.
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

16. Current ETICS metrics and Grid-QCM
Type
Programming language
Grid-QCM: INVOLVED EMs
complexity
static
Java, Python
EM CODING STYLE
EM STATIC ANALYSIS
design quality
Java
nr of „possible” bugs
Java, C/C++, Python, Perl, PHP
EM STRUCTURAL TESTING
dynamic
C/C++
unit test
EM FUNCTIONAL TESTING
coverage
profiling information
Java, C/C++
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

17. The timeline Now EGEE ’07 1-5 Oct Budapest (Hun) OGF 21 15-19 Oct
Seattle (USA)
ESA 3rd GRID & e-Collaboration Workshop Jan 2008
Frascati (IT)
OGF 20/EGEE UF
7-11 May
Manchester (UK)
Now
EELA 3° Conference 3-5 Dec 2007
Catania (IT)
OCTOBER
DECEMBER
FEBRUARY
MAY
QUALIPSO Conference
16-17 Jan 2008
Rome (IT)
ECHOGRID/EUChinagrid
Conference April
Beijing (CHINA)
Belief Conference
25-28 June
Rio de Janeiro (BRA)
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

18. Grid-QCM: feedback OGF 20 Belief/EELA Conference EGEE’07
Possibility to automate
Unclear formal name of the model
Criticism: GQACM was difficult to remember
Solution: New name: Grid-QCM
Belief/EELA Conference
Automation, CMMI/ISO compatibility
Structure of some parts of the model
Criticism: Metrics not well split according to their role
Solution: Organisation in Evaluation Models
EGEE’07
Automation, integration of Grid-QCM in the b&t tool, people asked for specific information
Metrics for the process
Criticism: Lack of process metrics
Solution: it’s out of the scope
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

19. Grid-QCM: feedback OGF 21 QualiPSo Conference
Automation, CMMI/ISO compatibility, people asked for specific information
ISO9000 compatibility
Criticism: no mention about ISO9000 compatibility
Solution: it’s out of the scope
QualiPSo Conference
Automation, Used Standards,
Default integration of Grid-QCM in the b&t tool Many people interested in specific information
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

20. Conclusions Grid-QCM is a certification model
Automatable
Implemented by default in the ETICS tool
Not in contradiction with classical standards
Ready to interact with classical standards
Not limited to research projects
Not limited to grid software
Grid-QCM require less human effort to be used because it is almost fully automatable
ETICS tool is ready to implement Grid-QCM
During the ETICS 2 project, if approved,
Grid-QCM will be validated on-the-field with at least four projects
Grid-QCM will be proposed for standardisation under ISO.
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

21. Q&A http://www.eu-etics.org
ETICS Final Review - Grid-QCM - CERN, 15 February 2008

22. Grid-QCM: Summarizing Table example
E = Excellent
G = Good
M = Medium
F = Fair
P = Poor
Consolidated eval. result
EM Static
EM Coding
EM structural
EM Functional
EM Std compliance
EM eval. result M G Y
Functionality
Accuracy
Interoperability
Compliance
(Y)
Reliability
Maturity
Maintainability
Analyzability
Changeability
Testability
Portability
Adaptability
Installability
ETICS Final Review - Grid-QCM - CERN, 15 February 2008