1. A Day in the Life of an SQA Manager
Robert Sabourin
President
AmiBug.Com, Inc.
Montreal, Canada
Tuesday, January 01, 2019
© Robert Sabourin, 2004

2. A Day in the Life of an SQA Manager
Robert Sabourin , Software Evangelist
President
AmiBug.Com Inc.
Montreal, Quebec, Canada
Tuesday, January 01, 2019
© Robert Sabourin, 2004

3. AmiBug.Com, Inc. Software Development & SQA Consulting Services
Training, Coaching and Professional Development
Light Effective Process
Team Building and Organization
We help people to get things done!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

4. I am a Bug In the style of a children's book.
Robert & Catherine Sabourin
ISBN:
In the style of a children's book.
Explains elements of software
development process in a fun easy
to read format.
Tuesday, January 01, 2019
© Robert Sabourin, 2004

5. Workshop D105 A Day in the Life of an SQA Manager
Tuesday, January 01, 2019
© Robert Sabourin, 2004

6. Learning Objective
With the successful completion of this workshop the student will be able to:
- Identify important activities done by software testing professionals and SQA Managers
- Define interaction of SQA Managers and other roles in the software development team
Tuesday, January 01, 2019
© Robert Sabourin, 2004

7. Overview Folk Wisdom of the ages Quotable quotes!
Why are we here? Purpose.
Who do we meet? Roles and Responsibilities.
What if we find a bug? Business context.
What kind of day is this? Phases and Workflow.
Rules of engagement? SQA Bill of Rights.
Tuesday, January 01, 2019
© Robert Sabourin, 2004

8. Which email should I read first?
“New test automation tools and tips?”
“CFO quarterly financial reporting principals”
“Sales funnel update”
“Trade show schedules”
“The Dilbert Zone”
“Welcome to QII!”
“New test strategy for current project”
“Bug Reporting Tactics and process update”
Tuesday, January 01, 2019
© Robert Sabourin, 2004

9. A Day in the Life of a Tester
Folk Wisdom
of the Ages!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

10. A Day in the Life of an SQA Manager
It’s all about people! (and the occasional bug too)
Tuesday, January 01, 2019
© Robert Sabourin, 2004

11. Fundamental Question
Robert Sabourin: The fundamental question of software engineering is:
‘How do you know when you are finished?’
Tuesday, January 01, 2019
© Robert Sabourin, 2004

12. Crosby on Quality “Quality is defined as conformance to requirements”
“Quality is not a measure of GOODNESS”
Phil B. Crosby, Quality is Free
Tuesday, January 01, 2019
© Robert Sabourin, 2004

13. Deming Quality approach (PDCA)
Plan, Do Check, and Act:
Plan what you want to implement.
Do the pilot implementation.
Check the results of the pilot.
Act on the results by tweaking the process before the next project.
Tuesday, January 01, 2019
© Robert Sabourin, 2004

14. Pareto Principal Vilfredo Pareto, 1848 - 1923, Economist
80% of the wealth was in the hands of 20% of the population
Tuesday, January 01, 2019
© Robert Sabourin, 2004

15. Pareto Principal
Joseph Juran, present, Quality Control Engineer
1950 Quality Control Handbook
20% of the study population accounts for 80% of the measure under consideration
Tuesday, January 01, 2019
© Robert Sabourin, 2004

16. Edsger W. Dijkstra
“Program testing can be used to show the presence of bugs, but never to show their absence”
Tuesday, January 01, 2019
© Robert Sabourin, 2004

17. A Day in the Life of an SQA Manager
Why are we here?
Tuesday, January 01, 2019
© Robert Sabourin, 2004

18. Purpose of Testing Common definition: Broader definition:
To find bugs before our customers do!
Broader definition:
The role of testing is to provide objective input to facilitate business decisions!
Keeps stakeholders aware of all issues or concerns that relate to shipping a product!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

19. Purpose
“At our organization we have a special focus on helping increase the value of our organization, we look at things from a business prospective always keeping in mind the key stakeholders, our customers, employees and shareholders”
Tuesday, January 01, 2019
© Robert Sabourin, 2004

20. Purpose
“The role of SQA in our organization is to provide objective input to facilitate business decisions (wise smart and good decisions)”
“SQA keeps internal stakeholders aware of all the issues that relate to shipping a product”
Tuesday, January 01, 2019
© Robert Sabourin, 2004

21. Purpose
To be an effective SQA manager you must be an “on purpose” SQA manager
On Time
On Quality
On Budget
Are meaningless unless you are On Purpose
Tuesday, January 01, 2019
© Robert Sabourin, 2004

22. Purpose of SQA Team The E-SQA Manager on Service Model
We have a service model for SQA generalized to Software Engineering
Metrics collection tracking as a service
Analysis as a service
Configuration management and construction as a service
Integration and System testing services
Formal inspection services
Tuesday, January 01, 2019
© Robert Sabourin, 2004

23. A Day in the Life of a Tester
Who do we meet?
Tuesday, January 01, 2019
© Robert Sabourin, 2004

24. Some Roles in a Software Project
Development
Testing
Logistics
User Education
Product Management
Project/Program Management
Architecture
Build master
Tuesday, January 01, 2019
© Robert Sabourin, 2004

25. Microsoft® SDD Team Model
SDD teams at Microsoft ®
small effective multidisciplinary teams
interdependent roles and responsibilities
clear goals and responsibilities
focused and empowered
teams generally work at one site
Tuesday, January 01, 2019
© Robert Sabourin, 2004

26. Microsoft® SDD Team Model
Tuesday, January 01, 2019
© Robert Sabourin, 2004

27. Microsoft® SDD Team Model
Six Clearly Defined Roles
Development
Testing
Logistics
User Education
Product Management
Program Management
Tuesday, January 01, 2019
© Robert Sabourin, 2004

28. Microsoft® SDD Team Model
Product Management Role
Customer advocate
Manages
requirements
business case
customer expectations
marketing of product launch
Drives
feature, schedule tradeoffs
Tuesday, January 01, 2019
© Robert Sabourin, 2004

29. Microsoft® SDD Team Model
Program Management Role
leader facilitator and coordinator of the project
drives the development process
make sure the product ships on time
represents the team to general management
Tuesday, January 01, 2019
© Robert Sabourin, 2004

30. Microsoft® SDD Team Model
Development Management Role
specifies physical design which meets requirements
estimate time and effort to build features
builds features
prepares product for release
Tuesday, January 01, 2019
© Robert Sabourin, 2004

31. Microsoft® SDD Team Model
Testing Role defined simply and clearly in the sense of SQA not corporate QA
ensure all issues are KNOWN to team
develop testing strategy and plans
FACILITATE BUSINESS DECISIONS
PROVIDE OBJECTIVE INFORMATION
Tuesday, January 01, 2019
© Robert Sabourin, 2004

32. Microsoft® SDD Team Model
User Education Role
drives usability and user performance enhancement trade-offs
advocate of the end-user (as opposed to the customer!)
on line help, training, user interface, documentation etc
Tuesday, January 01, 2019
© Robert Sabourin, 2004

33. Microsoft® SDD Team Model
Logistics Role
operations
support
procurement
deployment
channel relationships
Tuesday, January 01, 2019
© Robert Sabourin, 2004

34. Some Other Roles Software Architect generally part of development
Head techie
Decides technical solution
Ensures evolution of technology
Tuesday, January 01, 2019
© Robert Sabourin, 2004

35. Some Other Roles Software Development Coordinator
Drives operational logistics
Time keeper
formal inspections moderator
Metrics
logistics, program management, development
general administration
Tuesday, January 01, 2019
© Robert Sabourin, 2004

36. A Day in the Life of an SQA Manager
What if we find a Bug?
Tuesday, January 01, 2019
© Robert Sabourin, 2004

37. Definition of a Bug
To make our job more fun, whenever we have a concern with software, we call it a “bug”.
Tuesday, January 01, 2019
© Robert Sabourin, 2004

38. Example Bug Flow What do we do when we catch a bug?
Tuesday, January 01, 2019
© Robert Sabourin, 2004

39. Example Bug Flow Put it in a jar to look at it!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

40. About Bugs Bugs are not Good or Bad Tuesday, January 01, 2019
© Robert Sabourin, 2004

41. About Bugs Some bugs are important and have a high priority!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

42. About Bugs Some bugs are dangerous and have a high severity!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

43. About Bugs
Setting the priority and severity of a bug is a business decision
Changing business conditions impact the priority and severity of a bug!
Always review previous decisions in light of changing business context
Ensure staff assigning priority and severity are aware of all relevant business drivers
Tuesday, January 01, 2019
© Robert Sabourin, 2004

44. Bug Quadrants
Tuesday, January 01, 2019
© Robert Sabourin, 2004

45. Business Decisions SQA: Development: Product Management:
Objective input
Development:
Technical implementation
Product Management:
Customer driven requirements
Tuesday, January 01, 2019
© Robert Sabourin, 2004

46. Quadrant Changing
Same technical bug can be in a different quadrant depending on the business context
Monitor business drivers!
Focus find and fix quadrant -1- bugs high priority/high severity
Tuesday, January 01, 2019
© Robert Sabourin, 2004

47. Finished? How do you know you are finished? Tuesday, January 01, 2019
© Robert Sabourin, 2004

48. You know you are finished when …
… the only bugs left are the ones that Product Management and Development agree are acceptable (based on objective SQA input) ...
Tuesday, January 01, 2019
© Robert Sabourin, 2004

49. You know you are finished when …
… the only bugs left are the ones that Product Management and Development agree are acceptable (based on objective SQA input) …
At least for now!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

50. Bug Reporting Entered Reviewed Prioritized Assigned Unassigned Fixed
REFUSE
Entered
Reviewed
Prioritized
Assigned
CHECK
TRIAGE
DESIGNATE
CORRECT
MANDATE
Unassigned
Fixed
Closed
CONFIRM
FAILURE
Tuesday, January 01, 2019
© Robert Sabourin, 2004

51. Example Bug Flow Business Decisions Product Manager & Development Lead
Based on objective input from Test Lead
Tuesday, January 01, 2019
© Robert Sabourin, 2004

52. Example Bug Flow Business Decisions
Bug prioritization is an important business decision
Tuesday, January 01, 2019
© Robert Sabourin, 2004

53. Example Bug Flow Principal
Anyone internal directly involved in a project can enter a bug
Encouraged to do so!
Remember never loose a bug
Tuesday, January 01, 2019
© Robert Sabourin, 2004

54. A Day in the Life of a Tester
Exactly what kind of day is this?
Tuesday, January 01, 2019
© Robert Sabourin, 2004

55. Getting Things Done Development BUG REQ FLOW FLOW Release Cycle
- Who manages them?
- How are they prioritized?
- Where can I find them?
- Are the communicated?
- Do they get reprioritized?
- Are business drivers known?
- Are technical risks known?
Getting Things Done
Development
BUG
FLOW
REQ
FLOW
- Are builds delivered?
- Where do developers work?
- Configuration management?
- Source control? Baseline?
- Transition? Periodic?
- Smoke tests?
- Owners:Dev IT DBA SQA?
- Who manages them?
- What are they?
- Where can I find them?
- When are they updated?
- Why are they changing?
- How are they evolving?
- Do we observe turbulence?
Release
Cycle
Tuesday, January 01, 2019
© Robert Sabourin, 2004

56. Rational Unified Process Iterations and Workflowm n a y I t o ( s ) . # 1 2 + c p E b C u T h W k f w A
Requirements
Design
Implementation
Test
Analysis
Tuesday, January 01, 2019
© Robert Sabourin, 2004

57. Different work takes place as part of each phase of development .
Rational Unified
Process (RUP)
Transition
Construction
Inception
Elaboration
Core Workflow
Maintenance
Testing
Development
Design
Analysis
Requirements
Different work takes place as part
of each phase of development .
Tuesday, January 01, 2019
© Robert Sabourin, 2004

58. Phase
Rational Unified
Process (RUP)
Transition
Construction
Inception
Elaboration
Core Workflow
Maintenance
Testing
Development
Design
Analysis
Requirements
Testers take part in each core workflow involved in the development organization.
Tuesday, January 01, 2019
© Robert Sabourin, 2004

59. Elaboration, Procedures
Phase
Rational Unified
Process (RUP)
Transition
Construction
Inception
Elaboration
Core Workflow
Maintenance
Testing
Development
Design
Analysis
Requirements
Inspections, Reviews
React & adapt to change
Quality Factors
Planning, Strategy
Testing Objectives
Elaboration, Procedures
Test Design
Test Cases, Scripts
Exploration
Execution, Regression
Test Support, Update
Inspections, Reviews
Tuesday, January 01, 2019
© Robert Sabourin, 2004

60. Getting Things Done Trend Chart Open Bugs Tuesday, January 01, 2019
© Robert Sabourin, 2004

61. Getting Things Done
Tuesday, January 01, 2019
© Robert Sabourin, 2004

62. Getting Things Done
Tuesday, January 01, 2019
© Robert Sabourin, 2004

63. Getting Things Done Test Lead Must Track Efforts Every Day!
Number of Tests to elaborate?
Number of Tests Passed?
Number of Tests Failed?
Number of High Priority Bugs to be fixed?
Number of Bug Fixes to confirm?
Count every day and plot trend graphs!
Publish results.
Tuesday, January 01, 2019
© Robert Sabourin, 2004

64. Getting Things Done Trend Chart Open Bugs By Type
Tuesday, January 01, 2019
© Robert Sabourin, 2004

65. Getting Things Done Testing Schedule Tuesday, January 01, 2019
© Robert Sabourin, 2004

66. Philosophy We have precious little time to run tests!
We must always be prepared!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

67. Time
Tuesday, January 01, 2019
© Robert Sabourin, 2004

68. Getting Things Done Testing Planning Identify Test Objectives
What are we concerned about?
What does the application have to do?
Technical Risk
Compare risk of failure from developers point of view?
New code? Unknown technology? Unstable?
Business Importance
Relative business importance of testing objective?
Any test objective more important than any other?
Allocate effort
Total budget effort
Spread across testing objectives
Define chunks
Define a series of testing activities in chunks of about two hours.
Include chunks for Test Elaboration, Exploration and Execution
Adapt to change daily
Technical and business risks change daily
New test objectives and priorities are discovered
Tuesday, January 01, 2019
© Robert Sabourin, 2004

69. Getting Things Done Test Scheduling Adapt to change
Revised risks?
New test objectives? New chunks?
Triage Testing Chunks
Assign testing chunks to testing team members.
Analysis and exploration to more senior team members
DAILY
Prioritize Bugs
Relative business importance of testing objective?
Any test objective more important than any other?
Track Progress
Total budget effort
Spread across testing objectives
Smoke Test
Should be test new build
BUILD
Regression Test
Does application still work as expected?
Did we accidentally break something?
Stress Testing
How well does the application behave in harsh conditions?
Treated as an experiment.
Tuesday, January 01, 2019
© Robert Sabourin, 2004

70. Getting Things Done Testing Activities Elaboration Exploration
Define test procedure and test cases.
Methods, techniques, test cases. All must be repeatable.
Exploration
Primarily to identify areas of weakness or instability!
Use exploratory testing techniques.
PLATFORM
Execution
Follow defined test procedures or execute automated test scripts.
While testing identify bugs! (use checklists!)
Track Progress
Track defects Open open over time. (TREND CURVE)
Track test chunk status over time. (SPREED SHEET)
Stress Test Experiment
Execute in collaboration with development.
As required report bugs and always report stress test results!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

71. "No! Try not, Do. Or do not. There is no try."
Tuesday, January 01, 2019
© Robert Sabourin, 2004

72. So what is Exploratory Testing?
Tuesday, January 01, 2019
© Robert Sabourin, 2004

73. Exploratory Testing
An approach to “test design” that employs general-systems heuristics and specific-systems expertise to enable rapid development of test cases.
Approach formalized by James Bach (
Used in General Functionality and Stability Test Procedure for Windows 2000 Application Certification
Tuesday, January 01, 2019
© Robert Sabourin, 2004

74. Mandate to explore
William Clark
Meriwether Lewis
The object of your mission is to explore the Missouri river,
& such principal streams of it, as, by its course and communication
with the waters of the Pacific ocean...may offer the most direct &
practicable water communication across this continent for the
purposes of commerce.
- Thomas Jefferson's letter to Meriwether Lewis, June 1803
Tuesday, January 01, 2019
© Robert Sabourin, 2004

75. Make intelligent decisions
Take notes about your decisions
Map out where you have been
Others can use the result
Tuesday, January 01, 2019
© Robert Sabourin, 2004

76. Chart as you explore
Further exploration yields a good idea of the state of the world!
One bit at a time
Tuesday, January 01, 2019
© Robert Sabourin, 2004

77. Exploration Notes - Tabular - Chronological - Schematic - Point form
- Concise
Tuesday, January 01, 2019
© Robert Sabourin, 2004

78. Exploratory Testing Test cases
Not known in advance
Defined & executed “on the fly” while you learn about the product
Testers need to “hone up” their skills in making maps!
Consistent note taking style
Practice
Tuesday, January 01, 2019
© Robert Sabourin, 2004

79. Exploratory Testing During test we must capture
Function, options or sub-functions being explored
Test cases attempted
Comments, notes, images or attachments
Hints, reminders and observations which may be useful to future testers
Date, Platform, Build or Configuration under test
Name of person running test
Oracles, “strategy to assess correctness”
Other relevant details
Tuesday, January 01, 2019
© Robert Sabourin, 2004

80. Exploratory Test Process
Confirm Test Objective
Ensure context known
Ensure HW and SW OK
All tools available
Kick Off
Chunk of 90 to 120 min
Test, Plan, Discover
Prepare
Run
Wrap up
Collect all notes data
Complete
Review results with
Test Lead
Review
Follow Up
Reassess goals
Piece together map
Tuesday, January 01, 2019
© Robert Sabourin, 2004

81. People “It’s all about people ... and the occasional bug.”
An SQA Manager must defend the rights of the team!
Tuesday, January 01, 2019
© Robert Sabourin, 2004

82. People SQA Bill Of Rights
Right to know the business context for assigned activities. Staff must be able to answer the question: “What is the business reason for doing this assigned activity?”
Right to know what it means to finish assigned activity
Tuesday, January 01, 2019
© Robert Sabourin, 2004

83. People SQA Bill Of Rights
Right to know what software being tested is supposed to do and if assumptions are to be made the right to double check with product or development management before testing activity starts
Right to get software which the development team honestly believes works
Tuesday, January 01, 2019
© Robert Sabourin, 2004

84. People SQA Bill Of Rights Right to have fun at work
Right to learn new work methods, techniques and technologies
Right to try out innovations which may fail
Tuesday, January 01, 2019
© Robert Sabourin, 2004

85. People SQA Bill Of Rights
Right to speak directly with developer responsible for code being tested
Right to report a bug discovered even if it may already be in the bug list (never lose a bug)
Right to know how much effort to spread across a testing assignment
Tuesday, January 01, 2019
© Robert Sabourin, 2004

86. Thank You
Questions?
Tuesday, January 01, 2019
© Robert Sabourin, 2004

87. Just for Fun
Tuesday, January 01, 2019
© Robert Sabourin, 2004