Peer Reviews A. Winsor Brown Jan. 13, 2010 CS577b Spring 2010 Peer Reviews A. Winsor Brown Jan. 13, 2010 (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Goals of Presentation Background CeBASE types of defects vs. methods of removal COQUALMO Model Reviews per IEEE J-STD-016-1995: [repeat from 577a03] Standard for Information Technology Software Life Cycle Processes Software Development Acquirer-Supplier Agreement Quality Models and Metrics Peer Reviews as practiced in CS577 Fagan’s Inspections (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

CeBASE Empirical Data Shows Center for experience BAsed Software Engineering An early 2000 joint effort with UMD (Vic Basili); Funded by NSF Factor of 100 Growth in Software Cost-to-Fix from Requirements to Test [Other studies show a Factor of 1000] Technique Selection Guidance: Peer Reviews vs. Test (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Factor-of-100 Growth in Software Cost-to-Fix (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Technique Selection Guidance “Under specified conditions, …” Peer reviews are more effective than functional testing for faults of omission and incorrect specification (UMD, USC) Functional testing is more effective than reviews for faults concerning numerical approximations and control flow (UMD, USC) (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

COQUALMO The COQUALMO model contains two sub-models 1) the Defect Introduction model: Uses the required subset of COCOMO cost drivers and three internal baseline defect rates (requirements, design, code and test baselines) 2) the Defect Removal model: Uses three (orthogonal) defect removal profile levels, Automated Analysis People Reviews Execution Testing and Tools along with the prediction produced by the defect introduction model to estimate resultant defect density (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

COQUALMO (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

IEEE J-STD-016-1995 Standard for Information Technology Software Life Cycle Processes Software Development Acquirer-Supplier Agreement ls usable with any development strategy: structured to better accommodate incremental, evolutionary, and other development models than the traditional “waterfall” model. It is structured to avoid time-oriented dependencies and implications, provides alternatives to formal reviews (that can force a waterfall development model), and explains how to apply the standard across multiple builds or iterations. (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Reviews per IEEE J-STD-016-1995 Joint technical reviews – objectives: a) Review evolving software products, using as criteria the software product evaluation criteria in annex L; review and demonstrate proposed technical solutions; provide insight and obtain feedback on the technical effort; surface and resolve technical issues. b) Review project status; surface near- and long-term risks regarding technical, cost, and schedule issues. c) ... d) e) .... (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Reviews per IEEE J-STD-016-1995 Joint management reviews – objectives: a) Keep management informed about project status, directions being taken, technical agreements reached, and overall status of evolving software products. b) Resolve issues that could not be resolved at joint technical reviews. c) Arrive at agreed-upon mitigation strategies for near- and long-term risks that could not be resolved at joint technical reviews. d) Identify and resolve management-level issues and risks not raised at joint technical reviews. e) Obtain commitments and acquirer approvals needed for timely accomplishment of the project. (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Agenda Reviews per IEEE J-STD-016-1995: Quality Models and Metrics  Peer Reviews as practiced in CS577 (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Quality Model Types All four types represented: Process, Product, Property, Success (P3S) Product What’s a defect? Problem reports Property Defects [over time]: Removal and residual injection rates Defect Density Success Defect removal rate Problem/Trouble Reports Open over time Process Macro: Defect injection and removal; workflow Micro: ETVX, defect removal techniques, etc. (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Product Models Related to Quality What’s a defect? An instance of non-conformance with the initiating requirements, standards, or exit criteria or other “checklists” Can exist in the accuracy/completeness of requirements, standards, and associated interface/reference documents Determined ONLY by the responsible Author of an artifact Typically start out as concerns in informal or agile reviews What’s an “issue” Concerns that can NOT be fixed by the author of the artifact under review In developments with large number of people or cycles, issues are usually tracked to closure. (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Defect Categories Severity Major Minor A Condition that causes an operational failure, malfunction, or prevents attainment of an expected or specified result Information that would lead to an incorrect response or misinterpretation of the information by the user An instance of non-conformance that would lead to a discrepancy report if implemented as is Minor A violation of standards, guidelines, or rules, but would not lead to a discrepancy report Information that is undesirable but would not cause a malfunction or unexpected results (bad workmanship) Information that, if left uncorrected, may decrease maintainability (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Defect Categories (continued) Class Missing Information that is specified in the requirements or standard, but is not present in the document Wrong Information that is specified in the requirements or standards and is present in the document, but the information is incorrect Extra Information that is not specified in the requirements or standards but is present in the document (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Defect Categories (continued) Type Unavoidable Unavoidable defects (AKA changes) arise because of the methods, techniques or approaches being followed necessitate changes. Examples include changes arising because of the dynamics of learning, exploration in IKIWISI situations, code or screen contents reorganizations taken on as an "afterthought", replacement of stubs or place-holders in code, etc. Such situations are often "planned for" and expected to occur. Avoidable Changes in analysis, design, code or documentation arising from human error, and which could be avoided through better analysis, design, training, etc. Examples include stub replacement that violates win conditions or requirements such as execution time, memory space: for instance the replacement of a "stub" which breaks a critical timing constraint. (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Defect Categories (continued) Severity Class Type Major Missing Avoidable Wrong Unavoidable Minor Extra (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Agenda Reviews per IEEE J-STD-016-199u5: Quality Models and Metrics Peer Reviews as practiced in CS577  (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Role Based Agile Internal/Informal Review The main activities are following Planning Overview (optional) Preparation Review Meeting Rework Participants (four recommended) Review Leader (recommended: quality focal point for cs577) “Coder” (or next type of person in the artifact’s chain) “Tester” (reviews external interfaces; generates gedanken test approaches) Author (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

Agile Internal/Informal Review Planning Overview Preparation Concern Log Problem List Review Rework Review Result Summary (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

ETVX Paradigm Sw Development Process Role Based Peer Review Relationships: A software development process, e.g., specifying, designing, coding Role Based Peer Review Tasks distributed to team members Entry Criteria Task Validate Exit Emphasize SIMPLE process management. Simple because we are not accounting for forks & joins: processes that produce multiple products (that are used by different subsequent processes) or use multiple inputs from different predecessor processes. At the highest level we see a major task with transition where the validate box would be in this case the PDR. Below that we see the division of work amongst the software development team. On the right hand side of the foil we see Fagan's Inspection as the in-process, work product validation. (c) 2003-10 AWBrown & CSSE Jan. 13, 2010

CS577 IICM-Sw Defect Reporting Concepts Range of Defect identification and reporting mechanisms One at a time: Problem report system Multiple issues/problems found by a single reviewer: Agile Artifact Review (AAR) – only two types of forms (Issues/Concern Log and Defect List) Agile Internal/Informal Review (AIR): Two types of forms Agile Formal Review: Three different types of forms Internal/Informal Review: Four different types of bigger forms Formal Review: Four different types of bigger forms Fagan’s Inspection: Five different types of forms (c) 2003-10 AWBrown & CSSE Jan. 13, 2010