1. Quality Assurance & Standards MechEng SE3 (non-)lecture 12 Slides by Phil Gray

2. 2 MechEng SE3 2008-09 (non-)Lecture 12 Where to go for more… l Sommerville »7 th & 8 th Editions, Chap. 27 »6 th Edition, Chap. 24

3. 3 MechEng SE3 2008-09 (non-)Lecture 12 The Capability Maturity Model l Description of the characteristic levels of quality in the software development process l Developed by the Carnegie Mellon Software Engineering Institute

4. 4 MechEng SE3 2008-09 (non-)Lecture 12 The Capability Maturity Model l Initial »Ad hoc, chaotic l Repeatable »Basic tracking processes l Defined »Document, standardised processes l Managed »Measures of process and product quality collected; process understood and controlled l Optimizing »Continuous process improved via feedback

5. 5 MechEng SE3 2008-09 (non-)Lecture 12 CMM and Quality l Key feature of CMM is centrality of process and product quality l How is this achieved? l Answer: Quality Assurance or QA

6. 6 MechEng SE3 2008-09 (non-)Lecture 12 What is Quality Assurance l From Software QA/Test Resource Center: ".. Software QA involves the entire software development process - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented to 'prevention'.. “

7. 7 MechEng SE3 2008-09 (non-)Lecture 12 Sommerville on Quality Assurance l for Sommerville: »Quality Assurance – framework of procedures and standards »Quality Plan – selection and adaptation of procedures and standards for a project »Quality Control – carrying out processes that ensure procedures and standards are followed l the term ‘Quality Assurance’ sometimes refers to all of the above

8. 8 MechEng SE3 2008-09 (non-)Lecture 12 What’s involved in software quality assurance? l At the level of an institution, company or standards body »Develop procedures and standards »Perform certification –to prove that the QA mechanism used is acceptable and effective l At the level of a particular project »Prepare a quality plan –specifying processes, deliverables, measures of quality (metrics, standards) »Carry out quality control –Collect data l So-called metrics l Compared to standards –Conducting reviews l checking reality against plan and against standards l At all levels »Change attitudes –convince staff that quality is important –develop a “quality culture”

9. 9 MechEng SE3 2008-09 (non-)Lecture 12 economic justification l Reported gains »up to 50% reduction in development time »85% of faults removed via inspections (over 1 million lines of code) »90% reduction in maintenance effort required l Costs »Typically 5-10% of development effort

10. 10 MechEng SE3 2008-09 (non-)Lecture 12 techniques for quality reviews l Progress review »Examination of progress with respect to plans l Quality review »Examination of project artefacts with respect to attributes of quality l Inspection »an FTR that tries to identify likely areas for faults and to identify lack of conformity to standards »Includes code walkthrough

11. 11 MechEng SE3 2008-09 (non-)Lecture 12 stages of review process l planning l review preparation l individual preparation l review meeting l re-work l re-review l follow-up

12. 12 MechEng SE3 2008-09 (non-)Lecture 12 review team l minimum 3, maximum 6 l roles »author »moderator »reader »plus –scribe –QA staff –specialists –dependent developers –maintainers

13. 13 MechEng SE3 2008-09 (non-)Lecture 12 review dos and don’ts l DO make it peer group review, applicable to all stages of software development »method of finding faults cheaply »method of training and learning »method of control »method of encouraging “egoless teamwork” l DON’T make it »a problem-solving session – faults should be identified, but solutions should not be patched together ‘in committee’ »a managerial appraisal of personnel

14. 14 MechEng SE3 2008-09 (non-)Lecture 12 standards for assessment l documents »structure »section numbering and title styles »spelling, grammar, style »accuracy and appropriateness of content l diagrams »semantic correctness »syntactic and lexical correctness (use of symbols, connectivity rules) »number of nodes per page

15. 15 MechEng SE3 2008-09 (non-)Lecture 12 standards for assessment 2 l programs »use of comments »indentation style »module length »completeness »consistency »cohesion and coupling »maintainability »N.B. these semantic standards difficult to measure

16. 16 MechEng SE3 2008-09 (non-)Lecture 12 code inspection guidelines l tracing requirements »check off each requirement against piece(s) of code »(possibly) provide a cross-reference of document tracing to the review team »can use code walkthrough l questioning assumptions »any assumptions not justified by the requirements? »sizes and volumes of data consistent with requirements

17. 17 MechEng SE3 2008-09 (non-)Lecture 12 code inspection guidelines 2 l program structure »program structure sensible? »data structures updated properly (wrt DFDs) l scoping »variables as tightly scoped as possible? »global variables used only where absolutely necessary? »local subprograms used where appropriate?

18. 18 MechEng SE3 2008-09 (non-)Lecture 12 code inspection guidelines 3 l optimisation / factoring »overlapping subprograms which can be combined? »opportunities for code re-use? »code optimised where needed to satisfy efficiency requirements? l algorithms »algorithm efficiency appropriate for data volume assumptions »standard algorithms / libraries used where appropriate

19. 19 MechEng SE3 2008-09 (non-)Lecture 12 code inspection guidelines 4 l description of functionality »links from requirements to code via design documents »adequate embedded comments

20. 20 MechEng SE3 2008-09 (non-)Lecture 12 reviewing the inspection process l statistics collected on »details of items inspected »list of faults found & classification »resources required for re-working »number of people involved & time l analysis provides »fault checklists »management reports on effectiveness of inspections l it’s the process being assessed, not the authors or inspectors

21. 21 MechEng SE3 2008-09 (non-)Lecture 12 What is a Standard? "A standard is a document approved by a recognized body, that provides, for common and repeated use, rules, guidelines, or characteristics for products, processes or services with which compliance is not mandatory.” A Guide to Project Management Body of Knowledge, 1996

22. 22 MechEng SE3 2008-09 (non-)Lecture 12 Standards Organisations l International Organization for Standardization (ISO) » non-governmental » develops standards for various technical fields (more than 11000) » 120 national members, which are themselves standards organisations

23. 23 MechEng SE3 2008-09 (non-)Lecture 12 Standards Organisations l BSI (UK) l ANSI (USA) l DIN (Germany) l ETSI (European) l IEC (International) l Other standard setting bodies »IET »BCS »EU »W3C »OMG

24. 24 MechEng SE3 2008-09 (non-)Lecture 12 Software Engineering/IT Standards l ISO/IET/EU have specific standards that can be used for measuring product and process quality l E.g., »ISO/IEC TR 14471:1999 : Information Technology – Software Engineering – Guidelines for the adoption of CASE tools »ISO 9241-1 Ergonomic requirements for office work with visual display terminals »W3C –Standard = “recommendation” –E.g., XML 1.1

25. 25 MechEng SE3 2008-09 (non-)Lecture 12 ISO 9000 l international standard for quality management and quality assurance » states what must be in a quality management system » first established in 1987 » derived from BS 5750, a British standard l ISO 9001 » applies to products involving design » Latest version is ISO 9001:2000 » 9001-3 interprets 9001 for software development

26. 26 MechEng SE3 2008-09 (non-)Lecture 12 ISO 9000 (cont’d) l certification is not carried out by ISO » carried out by independent certification bodies » organisation is awarded a Certificate of Conformity

27. 27 MechEng SE3 2008-09 (non-)Lecture 12 What’s in ISO 9000? l covers 20 topics, including » Quality System » Design Control » Process Control » Inspection & Testing » Contract Review » Quality Records » Internal Quality Audits » Training

28. 28 MechEng SE3 2008-09 (non-)Lecture 12 What’s in ISO 9000? (cont’d) l standard is abstract; it sets ends not means l for example, “The supplier shall establish and maintain a documented quality system as a means of ensuring that product confirms to specified requirements.” [from Section 4.2 of ISO 9001]

29. 29 MechEng SE3 2008-09 (non-)Lecture 12 TickIt l developed by DTI l provides a nationally accredited certification body l interpretation of ISO 9000 » related to ISO 9001-3

30. 30 MechEng SE3 2008-09 (non-)Lecture 12 TickIt (cont’d) l gives concrete guidelines on how software development should conform to the standard » based on developing a scheme of internal audits » audits related to standards compliance » auditors require training » audits involve document reviews and staff interviews

31. 31 MechEng SE3 2008-09 (non-)Lecture 12 QA Standards: A Good Thing? l pro » makes quality assurance assessable » as of 1993, 40 000 organisations in 93 countries have adopted the standard » US survey – 89% reported greater operational efficiency – 48% reported increased profitability

32. 32 MechEng SE3 2008-09 (non-)Lecture 12 QA Standards: A Good Thing? l con » fosters “command & control” style of management – emphasises inflexible compliance with a set of rigid written rules » standards rely heavily on assessors’ judgements – standards are not completely objective

33. 33 MechEng SE3 2008-09 (non-)Lecture 12 QA Standards: A Good Thing? » staff will pay attention to controls, not the things affected by the controls – attention to quality inspection and monitoring can deflect from attention to quality itself – like the problem of exams distorting education