A Day in the Life of an SQA Manager Robert Sabourin President AmiBug.Com, Inc. Montreal, Canada rsabourin@amibug.com www.amibug.com Tuesday, January 01, 2019 © Robert Sabourin, 2004

A Day in the Life of an SQA Manager Robert Sabourin , Software Evangelist President AmiBug.Com Inc. Montreal, Quebec, Canada rsabourin@amibug.com Tuesday, January 01, 2019 © Robert Sabourin, 2004

AmiBug.Com, Inc. Software Development & SQA Consulting Services Training, Coaching and Professional Development Light Effective Process Team Building and Organization We help people to get things done! Tuesday, January 01, 2019 © Robert Sabourin, 2004

I am a Bug In the style of a children's book. Robert & Catherine Sabourin ISBN: 0-9685774-0-7 www.amazon.com www.fatbrain.com In the style of a children's book. Explains elements of software development process in a fun easy to read format. Tuesday, January 01, 2019 © Robert Sabourin, 2004

Workshop D105 A Day in the Life of an SQA Manager Tuesday, January 01, 2019 © Robert Sabourin, 2004

Learning Objective With the successful completion of this workshop the student will be able to: - Identify important activities done by software testing professionals and SQA Managers - Define interaction of SQA Managers and other roles in the software development team Tuesday, January 01, 2019 © Robert Sabourin, 2004

Overview Folk Wisdom of the ages Quotable quotes! Why are we here? Purpose. Who do we meet? Roles and Responsibilities. What if we find a bug? Business context. What kind of day is this? Phases and Workflow. Rules of engagement? SQA Bill of Rights. Tuesday, January 01, 2019 © Robert Sabourin, 2004

Which email should I read first? “New test automation tools and tips?” “CFO quarterly financial reporting principals” “Sales funnel update” “Trade show schedules” “The Dilbert Zone” “Welcome to QII!” “New test strategy for current project” “Bug Reporting Tactics and process update” Tuesday, January 01, 2019 © Robert Sabourin, 2004

A Day in the Life of a Tester Folk Wisdom of the Ages! Tuesday, January 01, 2019 © Robert Sabourin, 2004

A Day in the Life of an SQA Manager It’s all about people! (and the occasional bug too) Tuesday, January 01, 2019 © Robert Sabourin, 2004

Fundamental Question Robert Sabourin: The fundamental question of software engineering is: ‘How do you know when you are finished?’ Tuesday, January 01, 2019 © Robert Sabourin, 2004

Crosby on Quality “Quality is defined as conformance to requirements” “Quality is not a measure of GOODNESS” Phil B. Crosby, Quality is Free Tuesday, January 01, 2019 © Robert Sabourin, 2004

Deming Quality approach (PDCA) Plan, Do Check, and Act: Plan what you want to implement. Do the pilot implementation. Check the results of the pilot. Act on the results by tweaking the process before the next project. Tuesday, January 01, 2019 © Robert Sabourin, 2004

Pareto Principal Vilfredo Pareto, 1848 - 1923, Economist 80% of the wealth was in the hands of 20% of the population Tuesday, January 01, 2019 © Robert Sabourin, 2004

Pareto Principal Joseph Juran, 1903 - present, Quality Control Engineer 1950 Quality Control Handbook 20% of the study population accounts for 80% of the measure under consideration Tuesday, January 01, 2019 © Robert Sabourin, 2004

Edsger W. Dijkstra “Program testing can be used to show the presence of bugs, but never to show their absence” Tuesday, January 01, 2019 © Robert Sabourin, 2004

A Day in the Life of an SQA Manager Why are we here? Tuesday, January 01, 2019 © Robert Sabourin, 2004

Purpose of Testing Common definition: Broader definition: To find bugs before our customers do! Broader definition: The role of testing is to provide objective input to facilitate business decisions! Keeps stakeholders aware of all issues or concerns that relate to shipping a product! Tuesday, January 01, 2019 © Robert Sabourin, 2004

Purpose “At our organization we have a special focus on helping increase the value of our organization, we look at things from a business prospective always keeping in mind the key stakeholders, our customers, employees and shareholders” Tuesday, January 01, 2019 © Robert Sabourin, 2004

Purpose “The role of SQA in our organization is to provide objective input to facilitate business decisions (wise smart and good decisions)” “SQA keeps internal stakeholders aware of all the issues that relate to shipping a product” Tuesday, January 01, 2019 © Robert Sabourin, 2004

Purpose To be an effective SQA manager you must be an “on purpose” SQA manager On Time On Quality On Budget Are meaningless unless you are On Purpose Tuesday, January 01, 2019 © Robert Sabourin, 2004

Purpose of SQA Team The E-SQA Manager on Service Model We have a service model for SQA generalized to Software Engineering Metrics collection tracking as a service Analysis as a service Configuration management and construction as a service Integration and System testing services Formal inspection services Tuesday, January 01, 2019 © Robert Sabourin, 2004

A Day in the Life of a Tester Who do we meet? Tuesday, January 01, 2019 © Robert Sabourin, 2004

Some Roles in a Software Project Development Testing Logistics User Education Product Management Project/Program Management Architecture Build master Tuesday, January 01, 2019 © Robert Sabourin, 2004

Microsoft® SDD Team Model SDD teams at Microsoft ® small effective multidisciplinary teams interdependent roles and responsibilities clear goals and responsibilities focused and empowered teams generally work at one site Tuesday, January 01, 2019 © Robert Sabourin, 2004

Microsoft® SDD Team Model Tuesday, January 01, 2019 © Robert Sabourin, 2004

Microsoft® SDD Team Model Six Clearly Defined Roles Development Testing Logistics User Education Product Management Program Management Tuesday, January 01, 2019 © Robert Sabourin, 2004

Microsoft® SDD Team Model Product Management Role Customer advocate Manages requirements business case customer expectations marketing of product launch Drives feature, schedule tradeoffs Tuesday, January 01, 2019 © Robert Sabourin, 2004

Microsoft® SDD Team Model Program Management Role leader facilitator and coordinator of the project drives the development process make sure the product ships on time represents the team to general management Tuesday, January 01, 2019 © Robert Sabourin, 2004

Microsoft® SDD Team Model Development Management Role specifies physical design which meets requirements estimate time and effort to build features builds features prepares product for release Tuesday, January 01, 2019 © Robert Sabourin, 2004

Microsoft® SDD Team Model Testing Role defined simply and clearly in the sense of SQA not corporate QA ensure all issues are KNOWN to team develop testing strategy and plans FACILITATE BUSINESS DECISIONS PROVIDE OBJECTIVE INFORMATION Tuesday, January 01, 2019 © Robert Sabourin, 2004

Microsoft® SDD Team Model User Education Role drives usability and user performance enhancement trade-offs advocate of the end-user (as opposed to the customer!) on line help, training, user interface, documentation etc Tuesday, January 01, 2019 © Robert Sabourin, 2004

Microsoft® SDD Team Model Logistics Role operations support procurement deployment channel relationships Tuesday, January 01, 2019 © Robert Sabourin, 2004

Some Other Roles Software Architect generally part of development Head techie Decides technical solution Ensures evolution of technology Tuesday, January 01, 2019 © Robert Sabourin, 2004

Some Other Roles Software Development Coordinator Drives operational logistics Time keeper formal inspections moderator Metrics logistics, program management, development general administration Tuesday, January 01, 2019 © Robert Sabourin, 2004

A Day in the Life of an SQA Manager What if we find a Bug? Tuesday, January 01, 2019 © Robert Sabourin, 2004

Definition of a Bug To make our job more fun, whenever we have a concern with software, we call it a “bug”. Tuesday, January 01, 2019 © Robert Sabourin, 2004

Example Bug Flow What do we do when we catch a bug? Tuesday, January 01, 2019 © Robert Sabourin, 2004

Example Bug Flow Put it in a jar to look at it! Tuesday, January 01, 2019 © Robert Sabourin, 2004

About Bugs Bugs are not Good or Bad Tuesday, January 01, 2019 © Robert Sabourin, 2004

About Bugs Some bugs are important and have a high priority! Tuesday, January 01, 2019 © Robert Sabourin, 2004

About Bugs Some bugs are dangerous and have a high severity! Tuesday, January 01, 2019 © Robert Sabourin, 2004

About Bugs Setting the priority and severity of a bug is a business decision Changing business conditions impact the priority and severity of a bug! Always review previous decisions in light of changing business context Ensure staff assigning priority and severity are aware of all relevant business drivers Tuesday, January 01, 2019 © Robert Sabourin, 2004

Bug Quadrants Tuesday, January 01, 2019 © Robert Sabourin, 2004

Business Decisions SQA: Development: Product Management: Objective input Development: Technical implementation Product Management: Customer driven requirements Tuesday, January 01, 2019 © Robert Sabourin, 2004

Quadrant Changing Same technical bug can be in a different quadrant depending on the business context Monitor business drivers! Focus find and fix quadrant -1- bugs high priority/high severity Tuesday, January 01, 2019 © Robert Sabourin, 2004

Finished? How do you know you are finished? Tuesday, January 01, 2019 © Robert Sabourin, 2004

You know you are finished when … … the only bugs left are the ones that Product Management and Development agree are acceptable (based on objective SQA input) ... Tuesday, January 01, 2019 © Robert Sabourin, 2004

You know you are finished when … … the only bugs left are the ones that Product Management and Development agree are acceptable (based on objective SQA input) … At least for now! Tuesday, January 01, 2019 © Robert Sabourin, 2004

Bug Reporting Entered Reviewed Prioritized Assigned Unassigned Fixed REFUSE Entered Reviewed Prioritized Assigned CHECK TRIAGE DESIGNATE CORRECT MANDATE Unassigned Fixed Closed CONFIRM FAILURE Tuesday, January 01, 2019 © Robert Sabourin, 2004

Example Bug Flow Business Decisions Product Manager & Development Lead Based on objective input from Test Lead Tuesday, January 01, 2019 © Robert Sabourin, 2004

Example Bug Flow Business Decisions Bug prioritization is an important business decision Tuesday, January 01, 2019 © Robert Sabourin, 2004

Example Bug Flow Principal Anyone internal directly involved in a project can enter a bug Encouraged to do so! Remember never loose a bug Tuesday, January 01, 2019 © Robert Sabourin, 2004

A Day in the Life of a Tester Exactly what kind of day is this? Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Development BUG REQ FLOW FLOW Release Cycle - Who manages them? - How are they prioritized? - Where can I find them? - Are the communicated? - Do they get reprioritized? - Are business drivers known? - Are technical risks known? Getting Things Done Development BUG FLOW REQ FLOW - Are builds delivered? - Where do developers work? - Configuration management? - Source control? Baseline? - Transition? Periodic? - Smoke tests? - Owners:Dev IT DBA SQA? - Who manages them? - What are they? - Where can I find them? - When are they updated? - Why are they changing? - How are they evolving? - Do we observe turbulence? Release Cycle Tuesday, January 01, 2019 © Robert Sabourin, 2004

Rational Unified Process Iterations and Workflow m n a y I t o ( s ) . # 1 2 + c p E b C u T h W k f w A Requirements Design Implementation Test Analysis Tuesday, January 01, 2019 © Robert Sabourin, 2004

Different work takes place as part of each phase of development . Rational Unified Process (RUP) Transition Construction Inception Elaboration Core Workflow Maintenance Testing Development Design Analysis Requirements Different work takes place as part of each phase of development . Tuesday, January 01, 2019 © Robert Sabourin, 2004

Phase Rational Unified Process (RUP) Transition Construction Inception Elaboration Core Workflow Maintenance Testing Development Design Analysis Requirements Testers take part in each core workflow involved in the development organization. Tuesday, January 01, 2019 © Robert Sabourin, 2004

Elaboration, Procedures Phase Rational Unified Process (RUP) Transition Construction Inception Elaboration Core Workflow Maintenance Testing Development Design Analysis Requirements Inspections, Reviews React & adapt to change Quality Factors Planning, Strategy Testing Objectives Elaboration, Procedures Test Design Test Cases, Scripts Exploration Execution, Regression Test Support, Update Inspections, Reviews Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Trend Chart Open Bugs Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Test Lead Must Track Efforts Every Day! Number of Tests to elaborate? Number of Tests Passed? Number of Tests Failed? Number of High Priority Bugs to be fixed? Number of Bug Fixes to confirm? Count every day and plot trend graphs! Publish results. Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Trend Chart Open Bugs By Type Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Testing Schedule Tuesday, January 01, 2019 © Robert Sabourin, 2004

Philosophy We have precious little time to run tests! We must always be prepared! Tuesday, January 01, 2019 © Robert Sabourin, 2004

Time Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Testing Planning Identify Test Objectives What are we concerned about? What does the application have to do? Technical Risk Compare risk of failure from developers point of view? New code? Unknown technology? Unstable? Business Importance Relative business importance of testing objective? Any test objective more important than any other? Allocate effort Total budget effort Spread across testing objectives Define chunks Define a series of testing activities in chunks of about two hours. Include chunks for Test Elaboration, Exploration and Execution Adapt to change daily Technical and business risks change daily New test objectives and priorities are discovered Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Test Scheduling Adapt to change Revised risks? New test objectives? New chunks? Triage Testing Chunks Assign testing chunks to testing team members. Analysis and exploration to more senior team members DAILY Prioritize Bugs Relative business importance of testing objective? Any test objective more important than any other? Track Progress Total budget effort Spread across testing objectives Smoke Test Should be test new build BUILD Regression Test Does application still work as expected? Did we accidentally break something? Stress Testing How well does the application behave in harsh conditions? Treated as an experiment. Tuesday, January 01, 2019 © Robert Sabourin, 2004

Getting Things Done Testing Activities Elaboration Exploration Define test procedure and test cases. Methods, techniques, test cases. All must be repeatable. Exploration Primarily to identify areas of weakness or instability! Use exploratory testing techniques. PLATFORM Execution Follow defined test procedures or execute automated test scripts. While testing identify bugs! (use checklists!) Track Progress Track defects Open open over time. (TREND CURVE) Track test chunk status over time. (SPREED SHEET) Stress Test Experiment Execute in collaboration with development. As required report bugs and always report stress test results! Tuesday, January 01, 2019 © Robert Sabourin, 2004

"No! Try not, Do. Or do not. There is no try." Tuesday, January 01, 2019 © Robert Sabourin, 2004

So what is Exploratory Testing? Tuesday, January 01, 2019 © Robert Sabourin, 2004

Exploratory Testing An approach to “test design” that employs general-systems heuristics and specific-systems expertise to enable rapid development of test cases. Approach formalized by James Bach (www.satisfice.com) Used in General Functionality and Stability Test Procedure for Windows 2000 Application Certification Tuesday, January 01, 2019 © Robert Sabourin, 2004

Mandate to explore William Clark Meriwether Lewis The object of your mission is to explore the Missouri river, & such principal streams of it, as, by its course and communication with the waters of the Pacific ocean...may offer the most direct & practicable water communication across this continent for the purposes of commerce. - Thomas Jefferson's letter to Meriwether Lewis, June 1803 Tuesday, January 01, 2019 © Robert Sabourin, 2004

Make intelligent decisions Take notes about your decisions Map out where you have been Others can use the result Tuesday, January 01, 2019 © Robert Sabourin, 2004

Chart as you explore Further exploration yields a good idea of the state of the world! One bit at a time Tuesday, January 01, 2019 © Robert Sabourin, 2004

Exploration Notes - Tabular - Chronological - Schematic - Point form - Concise Tuesday, January 01, 2019 © Robert Sabourin, 2004

Exploratory Testing Test cases Not known in advance Defined & executed “on the fly” while you learn about the product Testers need to “hone up” their skills in making maps! Consistent note taking style Practice Tuesday, January 01, 2019 © Robert Sabourin, 2004

Exploratory Testing During test we must capture Function, options or sub-functions being explored Test cases attempted Comments, notes, images or attachments Hints, reminders and observations which may be useful to future testers Date, Platform, Build or Configuration under test Name of person running test Oracles, “strategy to assess correctness” Other relevant details Tuesday, January 01, 2019 © Robert Sabourin, 2004

Exploratory Test Process Confirm Test Objective Ensure context known Ensure HW and SW OK All tools available Kick Off Chunk of 90 to 120 min Test, Plan, Discover Prepare Run Wrap up Collect all notes data Complete Review results with Test Lead Review Follow Up Reassess goals Piece together map Tuesday, January 01, 2019 © Robert Sabourin, 2004

People “It’s all about people ... and the occasional bug.” An SQA Manager must defend the rights of the team! Tuesday, January 01, 2019 © Robert Sabourin, 2004

People SQA Bill Of Rights Right to know the business context for assigned activities. Staff must be able to answer the question: “What is the business reason for doing this assigned activity?” Right to know what it means to finish assigned activity Tuesday, January 01, 2019 © Robert Sabourin, 2004

People SQA Bill Of Rights Right to know what software being tested is supposed to do and if assumptions are to be made the right to double check with product or development management before testing activity starts Right to get software which the development team honestly believes works Tuesday, January 01, 2019 © Robert Sabourin, 2004

People SQA Bill Of Rights Right to have fun at work Right to learn new work methods, techniques and technologies Right to try out innovations which may fail Tuesday, January 01, 2019 © Robert Sabourin, 2004

People SQA Bill Of Rights Right to speak directly with developer responsible for code being tested Right to report a bug discovered even if it may already be in the bug list (never lose a bug) Right to know how much effort to spread across a testing assignment Tuesday, January 01, 2019 © Robert Sabourin, 2004

Thank You Questions? Tuesday, January 01, 2019 © Robert Sabourin, 2004

Just for Fun Tuesday, January 01, 2019 © Robert Sabourin, 2004