Just-In-Time Testing Robert Sabourin President AmiBug.Com, Inc. Montreal, Canada rsabourin@amibug.com September 19, 2018 © Robert Sabourin, 2003

Just-In-Time Testing Turbulent web development projects experience almost daily requirement changes coupled with frequent changes to user interfaces and the continual integration of new functions and features often using new technologies. This workshop teaches experienced test managers effective techniques which can help keep testing efforts on track while reacting to fast-changing priorities, technologies and requirements. There are no silver bullets – but several simple techniques can be applied to almost any development project enabling successful systematic testing efforts. September 19, 2018 © Robert Sabourin, 2003

Just-In-Time Testing After taking this workshop Test managers will be able to Set up key decision making workflows for Test Prioritization and Bug Triage Blend session based exploratory and scripted testing Implement meaningful status tracking and reporting Discover testing objectives, learning what is important to deliver, with or without written requirements Identify sources of information from developers and project stakeholders Adapt testing focus as priorities change, choosing what to test, what not to test, and in what order testing should be done. Identify technical risks and respect business importance Interact with key software engineering workflows Avoid time wasting activities Use systematic approaches to develop some Just-In-Time (JIT) Testing tools and templates and checklists September 19, 2018 © Robert Sabourin, 2003

Overview Introduction Some Philosophy Bug Flow Testing Ideas Test Triage Developers Testing Just In Time Platforms Exploratory Testing Stress Testing Checklists Scenarios Status Bibliography September 19, 2018 © Robert Sabourin, 2003

Elevator Parable Robert Sabourin , Software Evangelist President AmiBug.Com Inc. Montreal, Quebec, Canada rsabourin@amibug.com September 19, 2018 © Robert Sabourin, 2003

Just-In-Time Testing Some Philosophy September 19, 2018 © Robert Sabourin, 2003

Fundamental Question How do you know when you are finished? September 19, 2018 © Robert Sabourin, 2003

Crosby on Quality “Quality is defined as conformance to requirements” “Quality is not a measure of GOODNESS” Phil B. Crosby, Quality is Free September 19, 2018 © Robert Sabourin, 2003

Deming Quality approach (PDCA) Plan, Do Check, and Act: Plan what you want to implement. Do the pilot implementation. Check the results of the pilot. Act on the results by tweaking the process before the next project. September 19, 2018 © Robert Sabourin, 2003

Edsger W. Dijkstra “Program testing can be used to show the presence of bugs, but never to show their absence” September 19, 2018 © Robert Sabourin, 2003

Boris Beizer “Why software has bugs – the fundamental problem – Programming is a bitch.” The Frozen Keyboard - 1986 September 19, 2018 © Robert Sabourin, 2003

Ken Blanchard “Feedback is the breakfast of champions!” September 19, 2018 © Robert Sabourin, 2003

Watts S. Humphrey “… the job of the software engineer is to deliver high-quality software products at agreed cost and schedule …” “… even the most experienced software engineer injects about one defect for ten lines of code …” September 19, 2018 © Robert Sabourin, 2003

Tom DeMarco “You cannot control what you cannot measure.” September 19, 2018 © Robert Sabourin, 2003

C. Northcote Parkinson Parkinson’s Law: “…work expands so as to fill the time available for its completion…” September 19, 2018 © Robert Sabourin, 2003

Yoda "No! Try not, Do. Or do not. There is no try." September 19, 2018 © Robert Sabourin, 2003

“…begin with the end in mind … Steve Covey “…begin with the end in mind … “…first things first …" September 19, 2018 © Robert Sabourin, 2003

Purpose of Testing Common definition: Broader definition: To find bugs before our customers do! Broader definition: The role of testing is to provide objective input to facilitate business decisions! Keeps stakeholders aware of all issues or concerns that relate to shipping a product! September 19, 2018 © Robert Sabourin, 2003

Bug Defined To make our job more fun, whenever we have a concern with software, we call it a “bug”. September 19, 2018 © Robert Sabourin, 2003

Just-In-Time Testing It’s all about people! (and the occasional bug too) September 19, 2018 © Robert Sabourin, 2003

About Bugs Bugs are not Good or Bad September 19, 2018 © Robert Sabourin, 2003

About Bugs Some bugs are important and have a high priority! September 19, 2018 © Robert Sabourin, 2003

About Bugs Some bugs are dangerous and have a high severity! September 19, 2018 © Robert Sabourin, 2003

About Bugs Setting the priority and severity of a bug is a business decision Changing business conditions impact the priority and severity of a bug! Always review previous decisions in light of changing business context Ensure staff assigning priority and severity are aware of all relevant business drivers September 19, 2018 © Robert Sabourin, 2003

Bug Quadrants September 19, 2018 © Robert Sabourin, 2003

Quadrant Changing Same technical bug can be in a different quadrant depending on the business context Monitor business drivers! Focus find and fix high priority/high severity bugs September 19, 2018 © Robert Sabourin, 2003

Just for Fun September 19, 2018 © Robert Sabourin, 2003

Get Ready, Get Set, Cause here it comes Just In Time Testing Get Ready, Get Set, Cause here it comes September 19, 2018 © Robert Sabourin, 2003

Just-In-Time Testing Turbulence September 19, 2018 © Robert Sabourin, 2003

Just-In-Time Testing Unprepared September 19, 2018 © Robert Sabourin, 2003

Just-In-Time Testing So what exactly did they throw over the wall? September 19, 2018 © Robert Sabourin, 2003

Getting Things Done Metaphors Themes RUTHLESS TRIAGE RING BELL WELL OILED MACHINES Themes Parallelism wherever and always Think and test 24/7, early and always Chunking September 19, 2018 © Robert Sabourin, 2003

First Things First Begin with the end in mind Gain Consensus Goals How do we know we are finished? Purpose Why are we doing this project? How will be react to change? Meaning What is a bug? What is a test? September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Bug Flow September 19, 2018 © Robert Sabourin, 2003

Bug Flow We will be testing … imagine that we actually find a bug! What are we going to do about it? How will we decide? When should we decide how to decide? When should we change how we decide? When should we review our past decisions? September 19, 2018 © Robert Sabourin, 2003

Bug Flow Identify stakeholders Hunt them out Interview them Business, Technical, Internal, External Local, International Interview them Learn from them They should be represented but to not have to be involved in all daily aspects of project September 19, 2018 © Robert Sabourin, 2003

Keeping Informed Identify sources of information Ensure at least one person involved directly in bug triage actively reviews each source Each person may have one or more source Each source may have one or more people Information can be direct or synthesized If synthesized or summarized make sure purpose is clear September 19, 2018 © Robert Sabourin, 2003

Bug Flow Define priority scheme Define severity scheme Define workflow Define states of workflow Define states transitions of workflow Ensure all stakeholders buy in Change if context changes September 19, 2018 © Robert Sabourin, 2003

Bug Flow How did we do it last time? Did it work? How should will we do it next time? Do we need to use the same approach to prioritize all bugs? Unit Integration System September 19, 2018 © Robert Sabourin, 2003

Bug Flow Light on our feet Effective Agile Systematic Crisp and clear We have no time to muck around September 19, 2018 © Robert Sabourin, 2003

Example Bug Flow The “bug” flow is something like this bug is discovered in testing or reported from the field a bug report form is completed the bug report form is reviewed the bug report is added to the bug list a decision is made, at a bug review meeting, about whether the bug should be fixed if the bug is fixed then the software is re-tested to reconfirm that the bug has indeed been fixed if the bug is not fixed (on purpose!) then a description of the work around is published or made available to help desk staff September 19, 2018 © Robert Sabourin, 2003

Bug Flow Entered Reviewed Prioritized Assigned Unassigned Fixed Closed REFUSE Entered Reviewed Prioritized Assigned CHECK TRIAGE DESIGNATE CORRECT MANDATE Unassigned Fixed Closed CONFIRM FAILURE September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Testing Ideas September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Collect all testing ideas you can find! List Sort Organize Shuffle September 19, 2018 © Robert Sabourin, 2003

Testing Ideas How to find them? Does system do what it is suppose to do? Does the system do things it is not supposed to? How can the system break? How does the system react to it’s environment? What characteristics must the system conform to? Why have previous or similar projects failed? How have previous or similar systems failed? September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Collect testing ideas From testing ideas build a series of testing objectives (TO) Each can be assigned as work to a tester Each can include all, part of, or multiple testing ideas September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Index Cards Each card has a unique id Can be shuffled and reviewed Can be organized and reorganized Have one testing idea per card Can be sorted, grouped, prioritized and collected Concept of a prioritized heap September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Creative approaches Action verbs Mind Maps Soap Operas Lateral Thinking September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Investigative approaches We become truffle snorting pigs and try to find useful information in all evidence we discover We can even get good ideas from out of date sources September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Bug taxonomies Collections of possible bugs Appendix A of Testing Computer Software, Kaner, Falk, Nguyen Boris Biezer Taxonomy Otto Vinter manages Shopping cart taxonomy Giri Vijayaraghavan September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Requirements Use cases Functional requirements Quality factors Constraints Written requirements Implicit requirements September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Usage Scenarios Identify classes of users Identify how users will use system Describe scenarios Use Story board or similar approaches Identify variations September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Functionality Analysis Requirements, Design or Prototypes can give insights into Domain Analysis Equivalence classes Boundary analysis CRUD September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Failure Modes What can break? Reaction to invalid input? How does software behave in constrained environment? Memory Disk Space Network Bandwidth CPU capacity Shared resources Stress, Load, Volume September 19, 2018 © Robert Sabourin, 2003

Quality Factors Importance For Different Web Application Types September 19, 2018 © Robert Sabourin, 2003

Brainstorming Testing Ideas Min Max Missing Group of people Kick off meeting Individual build list Logging meeting Synergy Estimate size in sessions Identify missing or redundant ideas September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Test Data Library Inspired from reality Logs from previous or similar systems Build up SOAP OPERA life stories Build test data catalogues Organize so test data can be useful September 19, 2018 © Robert Sabourin, 2003

Testing Ideas Oracle Collection Strategies to assess correctness Similar systems Old systems Subject matter experts GURUs Standards September 19, 2018 © Robert Sabourin, 2003

Testing Tools Tools Status Dynamic analysis Static analysis Test engines Home brew with correct skills Danny Faught Open Testware ideas September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Test Triage September 19, 2018 © Robert Sabourin, 2003

Which test? Impact estimation For each test idea guesstimate: benefit of implementation consequence of implementation benefit for not implementing consequence of not implementing How credible are values? September 19, 2018 © Robert Sabourin, 2003

Understanding Complex Technology Quantitatively By Tom Gilb How to Decide? Rank Credibility 0.0 Wild guess, no credibility 0.1 We know it has been done somewhere 0.2 We have one measurement somewhere 0.3 There are several measurements in the estimated range 0.4 The measurements are relevant to our case 0.5 The method of measurement is considered reliable 0.6 We have used the method in-house 0.7 We have reliable measurements in-house 0.8 Reliable in-house measurements correlate to independent external measurements 0.9 We have used the idea on this project and measured it 1.0 Perfect credibility, we have rock solid, contract- guaranteed, long-term, credible experience with this idea on this project and, the results are unlikely to disappear September 19, 2018 © Robert Sabourin, 2003

Which test? Test Idea Rejection – What If? If the cost/benefit does not make business sense then consider implementing: part of the test, could that lead to part of the benefit at a more reasonable cost? more than the stated test, would that generate more benefit? a different test than the stated idea, could that generate more benefit for less cost? September 19, 2018 © Robert Sabourin, 2003

Test Triage Test Triage JIT Projects High Frequency Daily Test Triage Session Experience dictates Early AM (Rob Preference) Late PM (several clients) September 19, 2018 © Robert Sabourin, 2003

Test Triage Test Triage Meeting Review Context Business Technical Information since last triage Test results Bug results New testing ideas September 19, 2018 © Robert Sabourin, 2003

Test Triage Allocate Testing Assignments to Testers Make sure testers know context Best thing to test Best person to test it Best people to explore it Best lead Assign subject matter experts is required Sessions may be scripted or exploratory September 19, 2018 © Robert Sabourin, 2003

Test Triage Requirement Triage Change Control Test Triage Bug Flow Combined Equivalent to CCB Few people Fluid September 19, 2018 © Robert Sabourin, 2003

Test Triage Life of a test idea Comes into existence Clarified Prioritized Test Now (before further testing) Test before shipping Nice to have May be of interest in some future release Not of interest in current form Will never be of interest Integrate into a testing objective September 19, 2018 © Robert Sabourin, 2003

Which test is next? Magic crystal ball Ask the question Given state of project, state of business, state of technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next? How much effort are we willing to spend continuing to test this project? Can we ship yet? September 19, 2018 © Robert Sabourin, 2003

Which test is next? Magic crystal ball If it existed then how would you use it? What question would you ask it? What question would it ask you? September 19, 2018 © Robert Sabourin, 2003

Which test is next? Magic crystal ball Discover Example questions What question to ask? What information to have at hand? Example questions Given these test objectives how many sessions should I dedicate to them? Given that this part of the application is very buggy what should I test otherwise? September 19, 2018 © Robert Sabourin, 2003

Deciding what not to test? Time pressure Should we skip a test? If test failed could system still be of value to some stakeholder? If test was skipped could important bugs have been otherwise found? September 19, 2018 © Robert Sabourin, 2003

e-Commerce Search Example Domain testing Typical scenarios, valid & invalid data Scenarios touch search If scenario test passes we will have knowledge that in some typical cases with valid and invalid data search actually works (2) may be good enough for now September 19, 2018 © Robert Sabourin, 2003

Problem with formulas Units? Math? Exposure(i) = Risk * Consequence Allocation(i) = (Exposure(i)/Total Exposure) * MAX Units? Math? September 19, 2018 © Robert Sabourin, 2003

Guidelines and Decisions To each stakeholder risk of failure consequence of failure value of success how much certainty do we have is it a wild guess or an absolute truth? September 19, 2018 © Robert Sabourin, 2003

Sources of Information Version control system Monitor changes Track where work is Track where stability is Encourage finding defects earlier than system testing Inspections of code, design, requirements Unit Testing Informal code check in peer reviews September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Developers September 19, 2018 © Robert Sabourin, 2003

Testability Influence development team before software gets into your hands Can application be controlled by non-gui api for all business functions? Can application logging be controlled and logs viewed by testing? Can everything which can be set or changed be queried programmatically? Is there a static equivalent for dynamic guis? September 19, 2018 © Robert Sabourin, 2003

JIT Configuration Management Make sure that Developers and testers use different test servers Hand off to testing is based on a baseline Smoke test before and after delivery Keep previous build Know the last best build September 19, 2018 © Robert Sabourin, 2003

Testing Just In Time September 19, 2018 © Robert Sabourin, 2003

Philosophy We have precious little time to run tests! We must always be prepared! September 19, 2018 © Robert Sabourin, 2003

Time September 19, 2018 © Robert Sabourin, 2003

Getting Things Done Development BUG REQ FLOW FLOW Release Cycle - Who manages them? - How are they prioritized? - Where can I find them? - Are the communicated? - Do they get reprioritized? - Are business drivers known? - Are technical risks known? Getting Things Done Development BUG FLOW REQ FLOW - Are builds delivered? - Where do developers work? - Configuration management? - Source control? Baseline? - Transition? Periodic? - Smoke tests? - Owners:Dev IT DBA SQA? - Who manages them? - What are they? - Where can I find them? - When are they updated? - Why are they changing? - How are they evolving? - Do we observe turbulence? Release Cycle September 19, 2018 © Robert Sabourin, 2003

Getting Things Done Concern Concern Being Prepared! Being Prepared! - Information Flow - Information Flow Corporate information Key business drivers Sales Market Finance Corporate information Key business drivers Sales Market Finance - Technology Flow Architecture Technology churn Tools Techniques Training - Requirement Flow Defined Understood Interrupt Poll Prioritize Turbulence Status Truffle - Bug Flow Defined Understood Business Technical Efficient Expedient Reassess - Test Objectives Quality Factors Technical Risk Failure Modes Importance - Test Strategy Plan Analytic Exploratory Checklists Parallel Chunking Scenarios Data - Test Organization Scheduling Staffing Outsourcing Contractors Students - Testing Lab Multi-tier Server Client Platforms Swap Pattern Synchronized - Test Status Bug charts Test Plan Elaboration Status Pass Fail Execution Status September 19, 2018 © Robert Sabourin, 2003

Getting Things Done Where tests are run! Developers Desktop During development task assignments Unit Testing Development Servers During development and integration task assignments Unit and Integration testing Test Lab Servers During Integration and system test assignments Integration and System Testing Staging Servers Acceptance testing As close as possible to live technologies Live Servers Operational system Site monitoring September 19, 2018 © Robert Sabourin, 2003

Never update servers when tests are running! Getting Things Done Where tests are run! Developers Desktop Development Servers Test Lab Servers Staging Servers Live Servers - Synchronize releasing builds from the development team to the testing team! - Ensure a new builds passes the Smoke Test before it replaces the build currently being tested! Never update servers when tests are running! September 19, 2018 © Robert Sabourin, 2003

Getting Things Done Test Scheduling Adapt to change Revised risks? New test objectives? New chunks? Triage Testing Chunks Assign testing chunks to testing team members. Analysis and exploration to more senior team members DAILY Prioritize Bugs Relative business importance of testing objective? Any test objective more important than any other? Track Progress Total budget effort Spread across testing objectives Smoke Test Should the new build be tested at all? On failure continue with previous build in test. FAST Test Each area of functionality has a simple test. Is functional area stable enough to test BUILD Regression Test Does application still work as expected? Did we accidentally break something? Confirmation Test Have bugs really been fixed? Double check in test lab for each bug! Stress Testing How well does the application behave in harsh conditions? Treated as an experiment. September 19, 2018 © Robert Sabourin, 2003

Smoke Testing Smoke test is run on a new build of software to make sure all functions operate well enough to continue testing “Turn on a new appliance at the store” September 19, 2018 © Robert Sabourin, 2003

FAST Testing Functional Acceptance Simple Tests Wide in breadth, low in depth Exercise every function of the application at least once September 19, 2018 © Robert Sabourin, 2003

Regression Testing Previously executed tests are re-executed against a new version of the application have code changes broken something that used to work have we introduced new defects often automated September 19, 2018 © Robert Sabourin, 2003

Confirmation Testing Typically: Tester confirms that the fixed bug is really fixed in the appropriate software build September 19, 2018 © Robert Sabourin, 2003

Getting Things Done Testing Activities Elaboration Exploration Define test procedure and test cases. Methods, techniques, test cases. All must be repeatable. Exploration Primarily to identify areas of weakness or instability! Use exploratory testing techniques. Execution Follow defined test procedures or execute automated test scripts. While testing identify bugs! (use checklists!) PLATFORM Bug Isolation Narrow down how to repeat bug. Be practical. May need triage to determine if more isolation is needed. Track Progress Track defects Open open over time. (TREND CURVE) Track test chunk status over time. (SPREED SHEET) Stress Test Experiment Execute in collaboration with development. As required report bugs and always report stress test results! September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Platforms September 19, 2018 © Robert Sabourin, 2003

Browser Compatibility Testing www.netmechanic.com September 19, 2018 © Robert Sabourin, 2003

Platform Coverage We cannot possibly try all combinations vary platform used under testing in a manner consistent with risk associated with each platform component define a grid which indicates which platform will be used at which time vary platforms with tests Lots of parallelism!!! September 19, 2018 © Robert Sabourin, 2003

Platform Coverage Pattern Example - Multi-Layered Pattern Pattern A - Browser Resulting Pattern - Uniform Random September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Exploratory Testing September 19, 2018 © Robert Sabourin, 2003

Getting Things Done Wisdom Exploratory Testing in parallel with elaboration of chunks Elaborate higher priority chunks first Executing chunks as they are elaborated DO NOT WAIT FOR A COMPLETE SET September 19, 2018 © Robert Sabourin, 2003

Exploratory Testing Approach formalized by James Bach (www.satisfice.com) Used in General Functionality and Stability Test Procedure for Windows 2000 Application Certification September 19, 2018 © Robert Sabourin, 2003

Mandate to explore William Clark Meriwether Lewis The object of your mission is to explore the Missouri river, & such principal streams of it, as, by its course and communication with the waters of the Pacific ocean...may offer the most direct & practicable water communication across this continent for the purposes of commerce. - Thomas Jefferson's letter to Meriwether Lewis, June 1803 September 19, 2018 © Robert Sabourin, 2003

Make intelligent decisions Take notes about your decisions Map out where you have been Others can use the result September 19, 2018 © Robert Sabourin, 2003

Chart as you explore Further exploration yields a good idea of the state of the world! One bit at a time September 19, 2018 © Robert Sabourin, 2003

Exploration Notes - Tabular - Chronological - Schematic - Point form - Concise September 19, 2018 © Robert Sabourin, 2003

Exploratory Testing Test cases Not known in advance Defined & executed “on the fly” while you learn about the product Testers need to “hone up” their skills in making maps! Consistent note taking style Practice We need to have a standard template Uniform way to capture details September 19, 2018 © Robert Sabourin, 2003

Exploratory Testing During test we must capture Function, options or sub-functions being explored Test cases attempted Comments, notes, images or attachments Hints, reminders and observations which may be useful to future testers Date, Platform, Build or Configuration under test Name of person running test Oracles, “strategy to assess correctness” Other relevant details September 19, 2018 © Robert Sabourin, 2003

An Exploratory Test Process Confirm Test Objective Ensure context known Ensure HW and SW OK All tools available Kick Off Chunk of 90 to 120 min Test, Plan, Discover Prepare Run Wrap up Collect all notes data Complete Review results with Test Lead Review Follow Up Reassess goals Piece together map September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Stress Testing September 19, 2018 © Robert Sabourin, 2003

Testing operational characteristics of an application within a harshly constrained environment Limit processor speed Low memory, disk space Diminished access to shared resources Physical Environment, Static, Temperature, Humidity What is Stress Testing? September 19, 2018 © Robert Sabourin, 2003

Stress Testing September 19, 2018 © Robert Sabourin, 2003

September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Checklists September 19, 2018 © Robert Sabourin, 2003

Checklists Develop a series of one page testing check lists Aid testing user interface Train team how to use these checklists A checklist violation should be reported as a bug September 19, 2018 © Robert Sabourin, 2003

Checklists Remember the three Cs CLEAR CONCISE CONSISTENT September 19, 2018 © Robert Sabourin, 2003

Checklist Do not blindly use web checklists There are many wonderful examples These may be interesting and inspire you, but unless the same list is used by authors developing documents they can be deceptive and misleading September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Scenarios September 19, 2018 © Robert Sabourin, 2003

Scenario Based Testing Scenarios Your testing department should develop a series of typical, real, usage scenarios for each user type Can be done in parallel with development effort Could be based on Use Case Analysis Could be based on Story Boards Testing chunks can be dedicated to scenario elaboration or execution Elaboration (scripting) by testing professionals Execution by contractors, students, support staff etc Lots of parallelism! Individual scenarios can cover many functions! September 19, 2018 © Robert Sabourin, 2003

September 19, 2018 © Robert Sabourin, 2003

September 19, 2018 © Robert Sabourin, 2003

September 19, 2018 © Robert Sabourin, 2003

Scenario Based Testing User Experience Test Case Parameterize experience Walk through scenario from start to end Use pre selected input for each case Always run every test as if it were a user experience September 19, 2018 © Robert Sabourin, 2003

Just In Time Testing Status September 19, 2018 © Robert Sabourin, 2003

JIT Status Reporting Status Test Objectives Test Results Scoreboard Bug Summary What we know What is left to do September 19, 2018 © Robert Sabourin, 2003

JIT Status September 19, 2018 © Robert Sabourin, 2003

JIT Status September 19, 2018 © Robert Sabourin, 2003

JIT Status Test Lead Must Track Efforts Every Day! Number of Tests to elaborate? Number of Tests Passed? Number of Tests Failed? Number of High Priority Bugs to be fixed? Number of Bug Fixes to confirm? Count every day and plot trend graphs! Publish results. September 19, 2018 © Robert Sabourin, 2003

JIT Status Trend Chart Open Bugs September 19, 2018 © Robert Sabourin, 2003

JIT Status Trend Chart Open Bugs By Type September 19, 2018 © Robert Sabourin, 2003

JIT Status Testing Schedule September 19, 2018 © Robert Sabourin, 2003

Finished? How do you know you are finished? September 19, 2018 © Robert Sabourin, 2003

You know you are finished when … … the only bugs left are the ones are acceptable (based on your objective test team input) ... September 19, 2018 © Robert Sabourin, 2003

You know you are finished when … … the only bugs left are the ones are acceptable (based on your objective test team input) ... At least for now! September 19, 2018 © Robert Sabourin, 2003

Thank You Questions? September 19, 2018 © Robert Sabourin, 2003

Bibliography September 19, 2018 © Robert Sabourin, 2003

Books Quality Web Systems Testing Applications on the Web Dustin, Rashka, McDiarmid Testing Applications on the Web Hung Q. Nguyen The Web Testing Handbook Steven Splaine, Stefan P. Jaskiel Testing Computer Software Cem Kaner, Jack Falk, Hung Q. Nguyen September 19, 2018 © Robert Sabourin, 2003

Books Software System Testing and Quality Assurance Boris Beizer Black-Box Testing Dynamics of Software Development Jim McCarthy Software Inspection Tom Gilb, Dorothy Graham September 19, 2018 © Robert Sabourin, 2003

Books The Craft of Software Testing Brian Marick Microsoft Secrets Michael Cusumano, Richard Selby Why Does Software Cost So Much? Tom Demarco The Software Project Survival Guide Steve McConnell September 19, 2018 © Robert Sabourin, 2003

WWW www.stickyminds.com Software Test and Quality Engineering Excellent articles and forums www.satisfice.com James Bach Heuristic and Rapid testing www.qualitytree.com Elisabeth Hendrickson excellent SQA portal Bug based approaches September 19, 2018 © Robert Sabourin, 2003

WWW www.soft.com Software Research Inc. SQA Portal Software Quality Week Conferences www.construx.com Steve McConnell Software Project Survival www.testingcraft.com Brian Marrick Craft of Software Testing September 19, 2018 © Robert Sabourin, 2003

WWW www.kaner.com Cem Kaner SQA site www.amibug.com Robert Sabourin Site Presentations, SQA resources www.coyotevalley.com Brian Lawrence Site tons of check lists and life cycles tools for download September 19, 2018 © Robert Sabourin, 2003

WWW www.rational.com home of excellent UML references several white papers http://www.methods-tools.com/ This site provides various kinds of information about software development methods and tools. The site already contains information on almost 400 products from more than 160 software vendors September 19, 2018 © Robert Sabourin, 2003

WWW http://www.io.com/~wazmo/qa/ SOFTWARE TESTING HOTLIST contains tons of great pointers and SQA Test web resources. http://www.netmechanic.com Excellent testing resources related to Browser Compatibility September 19, 2018 © Robert Sabourin, 2003