1. Software Process Reviews/Audits
Process Overview by
Tom Gilchrist, CSQA, CSQE,

2. Overview of SW Audit Process SW Audit Examples
Before we start…
Information in this presentation are my opinions and not necessary those of my employer.
SQA Context
Overview of SW Audit Process
SW Audit Examples
SASQAG 10/17/2002 2

3. Deterministic vs. Non Deterministic Quality vs. Value
Some Terms/Ideas
Process
Deterministic vs. Non Deterministic
Quality vs. Value
SASQAG 10/17/2002 3

4. Software Quality Assurance
Check software products and processes to verify that they comply with the applicable procedures and standards. (Process Reviews or Audits)
Review and measure the quality of software products and processes throughout development. (Dynamic & Static Testing)
Provide software project management (and other appropriate parties) with the results of reviews and process checks.
Work with the software project during early stages to establish plans, standards, and procedures to keep errors from occurring in the first place.
SASQAG 10/17/2002 4

5. Formal Definition
Audits provide an independent evaluation of software products or processes to ascertain compliance to standards, specifications, and procedures based on objective criteria that included documents that specify:
The form or content of the product to be produced
The process by which the products shall be produced
How compliance to standards or guidelines shall be measured.
IEEE STD 1028, (1988)
SASQAG 10/17/2002 5

6. First Party Audit Second Party Audit Third Party Audit Audit Types
Within you company or organization
Second Party Audit
Sometimes called “external audits”
By a Customer on his Supplier
By a Supplier on you.
Third Party Audit
Outside third party is contracted to do the audit.
SASQAG 10/17/2002 6

7. Audit/Process Review Principles
Conducted by individuals who are organizationally independent of the developers.
Begin early in the requirements phase and continue throughout the development process.
Professionally planned, conducted and documented.
Follow-up on corrective action.
Project Management is involved in the Audit process and is responsible for rework and process improvements.
SASQAG 10/17/2002 7

8. What Software Audit Should Do
Determine:
Compliance to requirements
Conformance to plans, policies, procedures, and standards
Drive process improvement based on:
Adequacy of plans, policies, procedures, and standards
Effectiveness and efficiency of plans, policies, procedures, and standards
Assess personnel familiarity to requirements and documentation
Assure availability, use and adherence to software standards
SASQAG 10/17/2002 8

9. Quality Assurance Plan Event Date Requests from management
What Triggers an Audit?
Quality Assurance Plan
Event
Date
Requests from management
Requests from developers
Requests from customers
Integration with process improvement activities
Outside requirements — regulatory
Gut feel
SASQAG 10/17/2002 9

10. Scope: Requirements, Time, and Target
External
Standards
Spread around organization
Cover all functions and activities
Try to hit things early
Move towards process audits
Audit
Target
Organizational
Procedures and
Methods
SASQAG 10/17/2002 10

11. Process Review/Audit Process
Developers
Auditor
Project Manager
Plan
(Requirements, Scope, & Checklist)
Start
Prepare
Audit
Conduct
Audit
Write-up
Report &
Findings
Review
with
Manager
Findings? NO
YES
Corrective
Actions OK
Closeout
Audit &
File
END
Follow-up
Audit
Re-Work
SASQAG 10/17/2002 11

12. Identify Requirements
Policies/Standards Corporate, Group, IEEE
Processes/Plans SCMP, SQAP, SDP, Project Plan
Procedures Change Management, Design Reviews, Document Standards,
Testing
Task Instructions Library updates, unit testing, peer reviews
Success of an audit is directly proportional to preparation, research and analysis conducted before the audit is performed.
SASQAG 10/17/2002 12

13. Functional (ascertainably true or false)
Requirement Types
Functional (ascertainably true or false)
Quality (range of acceptable values)
SASQAG 10/17/2002 13

14. Types of Audits (Internal)
Quality System Audits
Product Audit
Process Audit
Project Audit
CM Audit
SASQAG 10/17/2002 14

15. Collect Factual Information Analyze and Evaluate the Evidence
Evidence Collection
Collect Factual Information
Analyze and Evaluate the Evidence
Draw Conclusions
Generate Findings
SASQAG 10/17/2002 15

16. Corrective Action of Findings
Determine Action
Immediate Remedial Action
Process Improvement/Fix
Acceptable Risk
Identify Root Cause
Corrective Actions Plan
Manage CA Plan to completion
Analyze Effects of CA
SASQAG 10/17/2002 16

17. Develop Audit Checklist
Focus on clear requirements (or unclear to fix)
Select subset of requirements
Focus on important steps/products
Write clear concise questions
Canned checklist vs. straw horse
SASQAG 10/17/2002 17

18. Checklist Sample Requirement Checklist Item Details Observations
Results (P/F)
Company Standard ABC-234, page 7
Does project QA plan will have a list of deliverables subject to Peer Reviews?
Check SQA document for a list of approved peer reviews and which documents are to be reviewed. (if no documents are found, then fail. If no peer review procedures are referenced, then fail)
Project SQA Plan
Were the number of audits completed equal to the number planned?
Check to see which audits were planned for the last 60 days. Check for evidence that the audit was completed and if there were findings, that a CA plan was signed.
Were the number of peer reviews completed equal to the number planned?
For each peer review type, check the CM records for the past 60 days to see if the document type specified in the QA plan was checked into CM for the first time. If so, check for records of the peer review being completed as per peer review process cited in SQA plan.
SASQAG 10/17/2002 18

19. Ask open-ended questions Know the types of answers expected
Interviewing
Ask open-ended questions
Know the types of answers expected
Focus on Process and not People
Seek Corroboration and Evidence
SASQAG 10/17/2002 19

20. Sample Interview Questions
How do you track your progress?
Do you have a CM Plan?
Tracing
What are you working on?
Is it a configured item?
Do you have an approved CR or PR?
Is the version you are working on checked out of CM?
SASQAG 10/17/2002 20

21. Desirable Auditor Characteristics
Emotional
Interviews
Group dynamics
Oral reports
Empathy
Don’t take things personally
Mechanical
Sampling
Root Cause Analysis
Intellectual
Writing
Planning
Speaking
Detail Oriented
Concise
SASQAG 10/17/2002 21

22. Desirable Auditor Characteristics (Cont.)
Knowledge of Audit process
Knowledge of target (SW) processes
Knowledge of techniques
Professional attitude
Good listener
Inquisitive/analytical
Communicates at all levels
Detailed Notes and Observations
Diplomatic
SASQAG 10/17/2002 22