Just In Time Testing Testing Turbulent Web Based Applications

Just In Time Testing Testing Turbulent Web Based Applications
Robert Sabourin President AmiBug.Com, Inc. Montreal, Canada February 4, 2019 © Robert Sabourin, 2001

Overview: Introduction and Definitions Testing Just In Time So What is Exploratory Testing? Quality Factors and Test Objectives Spreading Effort Across Test Objectives Status Some Effective Habits Patterns for Compatibility Testing Checklist Based Testing Scenario Based Testing Requirement Flow Bug Flow TRIAGE Bibliography February 4, 2019 © Robert Sabourin, 2001

Introduction & Definitions
February 4, 2019 © Robert Sabourin, 2001

Robert Sabourin , Software Evangelist President AmiBug.Com Inc. Montreal, Quebec, Canada February 4, 2019 © Robert Sabourin, 2001

AmiBug.Com, Inc. Software Development & SQA Consulting Services
Training, Coaching and Professional Development Light Effective Process Team Building and Organization We help people to get things done! February 4, 2019 © Robert Sabourin, 2001

I am a Bug In the style of a children's book.
Robert & Catherine Sabourin ISBN: In the style of a children's book. Explains elements of software development process in a fun easy to read format. February 4, 2019 © Robert Sabourin, 2001

Fundamental Question How do you know when you are finished?

Definition of a Bug To make our job more fun, whenever we have a concern with software, we call it a “bug”. February 4, 2019 © Robert Sabourin, 2001

Crosby on Quality “Quality is defined as conformance to requirements”
“Quality is not a measure of GOODNESS” Phil B. Crosby, Quality is Free February 4, 2019 © Robert Sabourin, 2001

Dr. W. Edwards Deming “Management of quality needs quality management”
“Improve quality, you automatically improve productivity.” February 4, 2019 © Robert Sabourin, 2001

Edsger W. Dijkstra “Program testing can be used to show the presence of bugs, but never to show their absence” February 4, 2019 © Robert Sabourin, 2001

Definition of a Bug To make our job more fun, whenever we have a concern with software, we call it a “bug”. February 4, 2019 © Robert Sabourin, 2001

Testing Web and E-Commerce Applications
It’s all about people! (and the occasional bug too) February 4, 2019 © Robert Sabourin, 2001

Purpose of Testing Common definition: Broader definition:
To find bugs before our customers do! Broader definition: The role of testing is to provide objective input to facilitate business decisions! Keeps stakeholders aware of all issues or concerns that relate to shipping a product! February 4, 2019 © Robert Sabourin, 2001

About Bugs Bugs are not Good or Bad February 4, 2019
© Robert Sabourin, 2001

About Bugs Some bugs are important and have a high priority!

About Bugs Some bugs are dangerous and have a high severity!

About Bugs Setting the priority and severity of a bug is a business decision Changing business conditions impact the priority and severity of a bug! Always review previous decisions in light of changing business context Ensure staff assigning priority and severity are aware of all relevant business drivers February 4, 2019 © Robert Sabourin, 2001

Bug Quadrants February 4, 2019 © Robert Sabourin, 2001

Business Decisions SQA: Development: Product Management:
Objective input Development: Technical implementation Product Management: Customer driven requirements February 4, 2019 © Robert Sabourin, 2001

Quadrant Changing Same technical bug can be in a different quadrant depending on the business context Monitor business drivers! Focus find and fix quadrant -1- bugs high priority/high severity February 4, 2019 © Robert Sabourin, 2001

Just for Fun February 4, 2019 © Robert Sabourin, 2001

Testing Just In Time February 4, 2019 © Robert Sabourin, 2001

Philosophy We have precious little time to run tests!
We must always be prepared! February 4, 2019 © Robert Sabourin, 2001

Time February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Development BUG REQ FLOW FLOW Release Cycle
- Who manages them? - How are they prioritized? - Where can I find them? - Are the communicated? - Do they get reprioritized? - Are business drivers known? - Are technical risks known? Getting Things Done Development BUG FLOW REQ FLOW - Are builds delivered? - Where do developers work? - Configuration management? - Source control? Baseline? - Transition? Periodic? - Smoke tests? - Owners:Dev IT DBA SQA? - Who manages them? - What are they? - Where can I find them? - When are they updated? - Why are they changing? - How are they evolving? - Do we observe turbulence? Release Cycle February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Concern Being Prepared! - Information Flow
Corporate information Key business drivers Sales Market Finance - Technology Flow Architecture Technology churn Tools Techniques Training - Requirement Flow Defined Understood Interrupt Poll Prioritize Turbulence Status Truffle - Bug Flow Defined Understood Business Technical Efficient Expedient Reassess - Test Objectives Quality Factors Technical Risk Failure Modes Importance - Test Strategy Plan Analytic Exploratory Checklists Parallel Chunking Scenarios Data - Test Organization Scheduling Staffing Outsourcing Contractors Students - Testing Lab Multi-tier Server Client Platforms Swap Pattern Synchronized - Test Status Bug charts Test Plan Elaboration Status Pass Fail Execution Status February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Where tests are run! Developers Desktop
During development task assignments Unit Testing Development Servers During development and integration task assignments Unit and Integration testing Test Lab Servers During Integration and system test assignments Integration and System Testing Staging Servers Acceptance testing As close as possible to live technologies Live Servers Operational system Site monitoring February 4, 2019 © Robert Sabourin, 2001

Never update servers when tests are running!
Getting Things Done Where tests are run! Developers Desktop Development Servers Test Lab Servers Staging Servers Live Servers - Synchronize releasing builds from the development team to the testing team! - Ensure a new builds passes the Smoke Test before it replaces the build currently being tested! Never update servers when tests are running! February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Testing Planning Identify Test Objectives
What are we concerned about? What does the application have to do? Technical Risk Compare risk of failure from developers point of view? New code? Unknown technology? Unstable? Business Importance Relative business importance of testing objective? Any test objective more important than any other? Allocate effort Total budget effort Spread across testing objectives Define chunks Define a series of testing activities in chunks of about two hours. Include chunks for Test Elaboration, Exploration and Execution Adapt to change daily Technical and business risks change daily New test objectives and priorities are discovered February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Test Scheduling Adapt to change
Revised risks? New test objectives? New chunks? Triage Testing Chunks Assign testing chunks to testing team members. Analysis and exploration to more senior team members DAILY Prioritize Bugs Relative business importance of testing objective? Any test objective more important than any other? Track Progress Total budget effort Spread across testing objectives Smoke Test Should be test new build BUILD Regression Test Does application still work as expected? Did we accidentally break something? Stress Testing How well does the application behave in harsh conditions? Treated as an experiment. February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Testing Activities Elaboration Exploration
Define test procedure and test cases. Methods, techniques, test cases. All must be repeatable. Exploration Primarily to identify areas of weakness or instability! Use exploratory testing techniques. PLATFORM Execution Follow defined test procedures or execute automated test scripts. While testing identify bugs! (use checklists!) Track Progress Track defects Open open over time. (TREND CURVE) Track test chunk status over time. (SPREED SHEET) Stress Test Experiment Execute in collaboration with development. As required report bugs and always report stress test results! February 4, 2019 © Robert Sabourin, 2001

Break down Test Objectives before detailed analysis
break project into high level test objectives validate priority and importance of these test objectives detail in priority order this way we can start running some important high priority tests before analysis is even started on low priority test objectives February 4, 2019 © Robert Sabourin, 2001

Risk Based Chunking Chunk method
How much testing effort to assign to each primary function is based on a risk analysis Commercial and Technical Risk are included in decision each TO is a chunk of testing which will be assigned to one tester February 4, 2019 © Robert Sabourin, 2001

Risk Based Chunking Technical risk Which functions:
Have potential instability? Have been changed recently? Look more complex? Have many controls and options? February 4, 2019 © Robert Sabourin, 2001

Risk Based Chunking Commercial risk Which functions:
Are customers going to use most? If they fail, could cause the most damage? Do our customers expect to work? Are only used by a few customers? February 4, 2019 © Robert Sabourin, 2001

Chunk Workflow Chunk method
A chunk is an amount of testing expected to take about minutes Test lead brakes project into specific test objectives (TOs) Each TO is a chunk of testing which will be assigned to one tester February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Wisdom
Exploratory Testing in parallel with elaboration of chunks Elaborate higher priority chunks first Executing chunks as they are elaborated DO NOT WAIT FOR A COMPLETE SET February 4, 2019 © Robert Sabourin, 2001

So what is Exploratory Testing?

Exploratory Testing An approach to “test design” that employs general-systems heuristics and specific-systems expertise to enable rapid development of test cases. Approach formalized by James Bach ( Used in General Functionality and Stability Test Procedure for Windows 2000 Application Certification February 4, 2019 © Robert Sabourin, 2001

Mandate to explore William Clark Meriwether Lewis The object of your mission is to explore the Missouri river, & such principal streams of it, as, by its course and communication with the waters of the Pacific ocean...may offer the most direct & practicable water communication across this continent for the purposes of commerce. - Thomas Jefferson's letter to Meriwether Lewis, June 1803 February 4, 2019 © Robert Sabourin, 2001

Make intelligent decisions
Take notes about your decisions Map out where you have been Others can use the result February 4, 2019 © Robert Sabourin, 2001

Chart as you explore Further exploration yields a good idea of the state of the world! One bit at a time February 4, 2019 © Robert Sabourin, 2001

Exploration Notes - Tabular - Chronological - Schematic - Point form
- Concise February 4, 2019 © Robert Sabourin, 2001

Exploratory Testing Test cases
Not known in advance Defined & executed “on the fly” while you learn about the product Testers need to “hone up” their skills in making maps! Consistent note taking style Practice We need to have a standard template Uniform way to capture details February 4, 2019 © Robert Sabourin, 2001

Exploratory Testing During test we must capture
Function, options or sub-functions being explored Test cases attempted Comments, notes, images or attachments Hints, reminders and observations which may be useful to future testers Date, Platform, Build or Configuration under test Name of person running test Oracles, “strategy to assess correctness” Other relevant details February 4, 2019 © Robert Sabourin, 2001

Exploratory Test Process
Confirm Test Objective Ensure context known Ensure HW and SW OK All tools available Kick Off Chunk of 90 to 120 min Test, Plan, Discover Prepare Run Wrap up Collect all notes data Complete Review results with Test Lead Review Follow Up Reassess goals Piece together map February 4, 2019 © Robert Sabourin, 2001

Quality Factors And Test Objectives

Quality Factors Definition Attributes of software
If not implemented they pose a risk to the success of the software business risk Factors that need to be considered when deciding your testing strategy February 4, 2019 © Robert Sabourin, 2001

Quality Factors Why? Purpose Focus
The purpose of testing can be viewed as being activities done to identify issues of concern to the commercial use or deployment of software Focus Quality factors help us to focus testing on the most important characteristics of the software February 4, 2019 © Robert Sabourin, 2001

Perceived Software Quality
Factor Perceived Software Quality Adaptability Can functions be enhanced without major redesign? Accessibility Can all users access your application? Auditibility Can a log be presented showing all access&transactions? Availability Can functions be accessed when needed? Continuity Can system continue operation recovering from a failure? Dependability Can users trust results provided? Expandability Can functions be added without major redesign? Functionality Does software perform as advertised? Integrity Is all data stored and returned unaltered. Interoperability Can be used with other software? February 4, 2019 © Robert Sabourin, 2001

Perceived Software Quality
Factor Perceived Software Quality Maintainability Can software be updated to new release levels? Operability Can software be easily put into operation? Portability Can software be run in different environments? Reliability How long does the software operate without failure? Re-usability Can elements of software be reused in future? Scalability Can transaction volume increase with more resources? Security Can unauthorized users access systems and data? Serviceability Can bug fixes be integrated with minimum downtime? Testability Can all functions and features be tested? Usability Can the software be easily used by target user? February 4, 2019 © Robert Sabourin, 2001

Ranking Quality Factors
It is important to determine the relative importance of each Quality Factor for your project! Testing resources are always severely limited Focus on things important for our business Use a systematic approach Ranking will be used to ensure testing effort is reasonably spread across Quality Factors February 4, 2019 © Robert Sabourin, 2001

Quality Factor Ranking Table For each Qualify Factor estimate two values Technical Risk Example Scale (1-5) 1 --> No technical risk at all 5 --> High technical risk, lot of new code Business Risk 1 --> Low importance to business 5 --> High importance to business February 4, 2019 © Robert Sabourin, 2001

Quality Factor Ranking Table For each Quality Factor compute an exposure by multiplying the Technical and Business Risk values associated with it. Then assign a Ranking to each Quality Factor in order of Exposure, for example: HIGH - >= 20 - TOP 5 MEDIUM - > 5 and < 20 LOW - low <= 5 - BOTTOM 5 February 4, 2019 © Robert Sabourin, 2001

Example Ranking February 4, 2019 © Robert Sabourin, 2001

Types of Web Systems Any Combinations Informational Delivery
Customized access User-provided content Interactive File sharing Transaction oriented Application service provider Database access Document access Workflow oriented Automatic content generator Any Combinations February 4, 2019 © Robert Sabourin, 2001

Quality Factors Importance For Different Web Application Types

Spreading Effort Across Test Objectives

Getting Things Done Effort Calculations
Overall effort to spend in system testing is company and project type dependent Effort is assigned to a project and spread across testing objectives February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Effort Guesstimation Rule of Thumb What if … ?
on first test cycle (A) we find important bugs on second test cycle (B) we find some bugs hidden by first cycle bugs one third test cycle (C) we find only minor bugs assumes CODE COMPLETE - NO REQUIREMENT CHANGES assumes MINIMAL CODE CHANGES RELATED TO BUG FIXES assumes BUILD PROCESS SOLID February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Effort Guesstimation Rule of Thumb
Then effort to run one full test cycle which passes should be approximately (1/7) the total effort budgeted for System testing. T(total) = T(A) + T(B) + T(C) T(A) = 2*T(B) T(B) = 2*T(C) T(total) = 7*T(C) A B C February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Effort Guesstimation Rule of Thumb Example:
If the development effort on a project was 14 Person Months So Integration + System Testing estimated at 14 Person Months Total Split (50/50 for example) System testing = 7 person months Test cycle = 1 person month So given 4 chunks per day our total amount of testing chunks to assign to testing objectives would be approximately 80 chunks assuming 20 working days per month! February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Example Project TRIWEBCO
Important Testing Objectives Overall Stability [system tests] Typical End User Usage Scenarios [system tests] Typical Platforms [in parallel - check lists] End User Usability [in parallel - check lists] Typical Administrator Usage Scenarios [system tests] Basic Operation of all Functions [system tests] Data Integrity [system tests] Performance [system tests] Scalability [per build stress experiment] Reliability [per build stress experiment] February 4, 2019 © Robert Sabourin, 2001

Getting Things Done February 4, 2019 © Robert Sabourin, 2001

Status February 4, 2019 © Robert Sabourin, 2001

Getting Things Done February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Test Lead Must Track Efforts Every Day!
Number of Tests to elaborate? Number of Tests Passed? Number of Tests Failed? Number of High Priority Bugs to be fixed? Number of Bug Fixes to confirm? Count every day and plot trend graphs! Publish results. February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Trend Chart Open Bugs
Monday, February 04, 2019Monday, February 04, 2019 © Robert Sabourin, 1999

Getting Things Done Trend Chart Open Bugs By Type

Getting Things Done Testing Schedule

Some Effective Habits February 4, 2019 © Robert Sabourin, 2001

Getting Things Done Metaphors Themes RUTHLESS TRIAGE RING BELL
WELL OILED MACHINES Themes Parallelism wherever and always Think and test 24/7, early and always Chunking February 4, 2019 © Robert Sabourin, 2001

Q - II E-SQA Manager explained Q - II
Steve Covey “7 Habits of Highly Effective People” A new paradigm of time management February 4, 2019 © Robert Sabourin, 2001

Quadrants What we do with our time? How do we use our time

Quadrants Urgency Importance
Things that require and demand our attention now Importance Things that have significance, meaning and value February 4, 2019 © Robert Sabourin, 2001

Time Management Matrix
II III IV February 4, 2019 © Robert Sabourin, 2001

QII This is where you have to be in order to make a significant impact of your environment To make things better and have a lasting influence To be effective To really get things done! February 4, 2019 © Robert Sabourin, 2001

QII By attending this workshop you are participating in a significant QII activity! This workshop is not urgent but obviously (hopefully!) important February 4, 2019 © Robert Sabourin, 2001

QII Information Flow is a QII activity Poll Interrupt
Tactfully trolling for information Interrupt Get in the loop Ensure others inform you But don’t depend on them alone February 4, 2019 © Robert Sabourin, 2001

Patterns for Compatibility Testing

Platform Coverage What is a Platform?
In testing we consider a “Platform” to be the “Hardware” and “Software” environment that an application under test is run on Typically required to operate on several different platforms Platform can also include other concurrently running applications February 4, 2019 © Robert Sabourin, 2001

Browser Compatibility Testing
Web application should: Operate on all popular platforms (Requirements based) Operate with all popular versions of plug-ins and third party components or viewers required Up to corporate policy how far to take this! Pareto principal February 4, 2019 © Robert Sabourin, 2001

Which Browser? Which Version? Microsoft Internet Explorer Netscape Navigator AOL Browser Lotus Notes Browser Web TV Browser Mosaic Opera PDA Browsers Cell Phone Browsers Voice Browsers February 4, 2019 © Robert Sabourin, 2001

Statistics: AmiBug.Com, August 2001

Platform Coverage Which operating system (O/S)?
International variants of all! Which Locale (of dozens)? Which Hardware? Which Co-dependent Software? Plug Ins Viewers Which Display Resolution? Which System Font Size? Which Color Depth? Cipher Strength? February 4, 2019 © Robert Sabourin, 2001

Platform Coverage Number of possible combinations are tumultuous
Cannot possible try each combination even for a short period of time each Even if we restrict out testing to the most commonly used variations over the past two years or so! February 4, 2019 © Robert Sabourin, 2001

Platform Coverage We cannot possibly try all combinations Instead
vary platform used under testing in a manner consistent with Technical Risk and Business Risk associated with each platform component define a grid which indicates which platform will be used at which time vary platforms with tests Lots of parallelism!!! February 4, 2019 © Robert Sabourin, 2001

Platform Coverage Basic principals
Each testing session will last about 90 minutes Each testing session is run in a test lab Workstations in a test lab can be quickly reconfigured (in less than 30 minutes) to a different platform image copy tools like Ghost or image restores or clever disk partitioning can be used to achieve quick reconfigurations of platforms February 4, 2019 © Robert Sabourin, 2001

Platform Coverage Basic principals
Each tester is assigned to perform a testing session on the next available workstation at the appropriate time slot Patterns are established independent of which test in being run February 4, 2019 © Robert Sabourin, 2001

Platform Coverage Single Pattern Approach List all likely combinations
For each combination Assign a projected target usage value giving the business risk Assign a technical risk value Cycle through list changing platform periodically so that different tests are run on different platforms February 4, 2019 © Robert Sabourin, 2001

Platform Coverage Example Application on
Windows or MAC Netscape or IE Five Tests exist called T1, T2 … T5 Each day we can run 5 tests February 4, 2019 © Robert Sabourin, 2001

Platform Coverage Example Technical risk Uniform
Common in Web based applications Market spread application dependent 10% - MAC/Netscape 10% - MAC/IE 30% - Windows/Netscape 50% - Windows/IE February 4, 2019 © Robert Sabourin, 2001

Platform Coverage Pattern

Example 2 - Multi-Layered Pattern For each relevant platform component a pattern is defined, in our example we care about the following three characteristics of the workstation: Pattern A - Browser Pattern B - Operating System Pattern C - Locale February 4, 2019 © Robert Sabourin, 2001

Example 2 - Multi-Layered Pattern Two testing workstations are made available to testers and are prepared well in advance according to a predefined pattern, identified as W001 and W002 Four testing sessions are run per day in different time slots know as TS01, TS02, TS03 and TS04 February 4, 2019 © Robert Sabourin, 2001

Example 2 - Multi-Layered Pattern Testing is organized into 17 different testing sessions identified as T001, T002, … , T017 80 sessions over 10 days of testing February 4, 2019 © Robert Sabourin, 2001

Example 2 - Multi-Layered Pattern Notes Railroad Testing approach is used in this example Once all test sessions are completed we restart testing with the first test session Whenever a new build is issued to testing, we continue testing at the next session we do not restart from the beginning! Testing and builds may be asynchronous February 4, 2019 © Robert Sabourin, 2001

Example 2 - Multi-Layered Pattern Pattern A - Browser Technical and Market Risk Assessment February 4, 2019 © Robert Sabourin, 2001

Example 2 - Multi-Layered Pattern Pattern A - Browser Resulting Pattern - Uniform Random February 4, 2019 © Robert Sabourin, 2001

Example 2 - Multi-Layered Pattern Pattern B - Operating System Technical and Market Risk Assessment February 4, 2019 © Robert Sabourin, 2001

Example 2 - Multi-Layered Pattern Pattern B - Operating System Resulting Pattern - Uniform Random February 4, 2019 © Robert Sabourin, 2001

Example 2 - Multi-Layered Pattern Pattern C - Locale Technical and Market Risk Assessment February 4, 2019 © Robert Sabourin, 2001

Example 2 - Multi-Layered Pattern Pattern C - Locale Resulting Pattern - Uniform Random February 4, 2019 © Robert Sabourin, 2001

Checklist Based Testing

Excellent Practice Your testing department should develop a series of one page check lists as an aid to testing the user interface of your web and e-commerce applications All members of the testing team should be trained in how to use these checklists During assigned testing activities, whenever a checklist violation is observed, it should be reported as a bug Lots of parallelism! February 4, 2019 © Robert Sabourin, 2001

Start with Checklists for: Usability Testing Checklist GUI Testing Checklist Application Style Consistency Checklist Design Mistakes Checklist Others specific to your local team and all relevant experiences! February 4, 2019 © Robert Sabourin, 2001

Web Usability Checklist
Is purpose of site evident? Are features available to user evident? Is contact information available? Are graphics optimized for quick download? Can users navigate using text only? Is there a site map? Is page layout too cluttered? Is page or site update or revision visible? Is “Title” visible, readable and consistent with intended use of the page? Does a user have to navigate through too many pages to get the information they seek? Are links and icons labeled correctly and consistently with their intended purpose? February 4, 2019 © Robert Sabourin, 2001

Web Usability Checklist
Are pages too large? Are there too many links per page? Are there too few links per page? Is help available? Overuse of flashy graphics - gimmicks - without meaningful purpose? Are there always clear and obvious links to the home page? Are printer friendly version of relevant pages available? Are Frequently Asked Questions - and associated answers available? Does you User Interface conform to any required relevant standards? ... February 4, 2019 © Robert Sabourin, 2001

GUI Check List Assure consistent look and feel on all pages
Assure the existent of proper controls which are logically required Assure GUI does not have a cluttered appearance Assure all user related system messages are presented consistently, clearly and in an obvious location on the display Navigation order with tab should be logical Default values for all scroll lists, radio buttons, text boxes or other controls should be correct Duplicate entries in scroll lists should not appear Ensure any demarcated are of the display has a meaningful name consistent with it’s intended purpose Ensure radio buttons, scroll lists and text boxes are logically grouped together When hot keys are used duplication must be avoided February 4, 2019 © Robert Sabourin, 2001

GUI Check List Ensure names or text or titles are meaningful to users of the system Ensure page titles are unique Ensure abbreviations are not used Ensure that each language version of a page has the same controls - but - which have been localized as required Ensure focus is placed, by default, in an appropriate part of the page Ensure default values are correct Ensure size and shape of buttons are consistent Ensure unavailable options are grayed out Ensure user is given relevant valid controls ... February 4, 2019 © Robert Sabourin, 2001

Style Checklist Does page have link to home or origin?
Are all “click here” buttons justified? Do all IMG tags have ALT tags? Can pages be navigated if browser image loading is disabled? Are font styles, sizes, and colors consistent? Is page layout logical for intended purpose? Does page conform to relevant guidelines? Are initial or default states of any selection fields reasonable? Is initial focus at a reasonable location? Are fonts legible? February 4, 2019 © Robert Sabourin, 2001

Style Checklist ... Does page have link to home or origin?
Are colors suitable? Background, Foreground, Font Are borders reasonable? Minimize 3-d and special effects Labels and buttons Are suitable images used? ... February 4, 2019 © Robert Sabourin, 2001

Design Mistakes Checklist
Back Button Break Slow Down Open New Window Immediate redirection Preventing caching New Browser Windows Non-Standard GUI Widgets Long scrolling pages Lack of navigation support or help Non-standard link colours Outdated information Too long download times February 4, 2019 © Robert Sabourin, 2001

Design Mistakes Checklist
Moving pages to new URLs Out of Context headlines Buzz Word Slow Server Response times Too much advertising Frames poorly used Gratuitous use of bleeding edge technologies Scrolling text, marquees, continuous animations Complex URLs Orphan Pages Send your customer away! ... February 4, 2019 © Robert Sabourin, 2001

Scenario Based Testing

Excellent Practice Your testing department should develop a series of typical, real, usage scenarios for each user type Can be done in parallel with development effort Could be based on Use Case Analysis Could be based on Story Boards Testing chunks can be dedicated to scenario elaboration or execution Elaboration (scripting) by testing professionals Execution by contractors, students, support staff etc Lots of parallelism! Individual scenarios can cover many functions! February 4, 2019 © Robert Sabourin, 2001

Story Boards E-Commerce user experience what can be done paths through the system February 4, 2019 © Robert Sabourin, 2001

User Experience Login Read message Check Out specials Buys some stuff Goes to affiliate February 4, 2019 © Robert Sabourin, 2001

User Experience Test Case Parameterize experience Walk through scenario from start to end Use pre selected input for each case Always run every test as if it were a user experience February 4, 2019 © Robert Sabourin, 2001

Best Practice Usage scenarios parameterize data driven start with valid scenarios with valid data February 4, 2019 © Robert Sabourin, 2001

Best Practice Usage scenarios make delete change Randomize order Exercise sequencing Build in parts (wrong then change to right) February 4, 2019 © Robert Sabourin, 2001

Separate test procedure from test data Procedure Instructions to follow in order to run the test Navigation instructions Invariant part of a test February 4, 2019 © Robert Sabourin, 2001

Test Data Any data typed into a form or field by the end user Variable part of a test Potential other data Any other field Any decision February 4, 2019 © Robert Sabourin, 2001

Purchase a book from Amazon.com Procedure go to URL wait for home page to load in search field type in ISBN number of book add book to shopping cart proceed to checkout log in with user name and password ship to this address place order February 4, 2019 © Robert Sabourin, 2001

Advantages Separates test code from data Leverage well defined test procedure Same procedure can be used for many different purposes depending on data selected Critical first step in design of scenario based test automation scripts Forces decisions to be deliberate - not random or ad-hoc! February 4, 2019 © Robert Sabourin, 2001

Requirement Turbulence
Software Engineering Workflow Daily change Prioritization of requirements “Very Short” development cycles We are in the react to change business! February 4, 2019 © Robert Sabourin, 2001

Defining Requirement Flow
Basic steps: Identify key roles, decision makers Identify project stakeholders Define requirement states Define all possible requirement state transitions Define steps to implement state transitions February 4, 2019 © Robert Sabourin, 2001

Key roles, decision makers: Role of requirement manager Other roles Who specifically When Minimize but ensure coverage February 4, 2019 © Robert Sabourin, 2001

Identify project stakeholders Who? Why? Business reason? Importance? Impact on prioritization? February 4, 2019 © Robert Sabourin, 2001

Define requirement states, for example: Empty Entered, Pending Review Reviewed Implement in current project Implement in future project Suspend, NEVER IMPLEMENT Already Implemented (confirmed) February 4, 2019 © Robert Sabourin, 2001

Define all possible requirement state transitions: Draw state transition diagram Write up state transition table Identify decision steps Consider new requirement Consider new business context February 4, 2019 © Robert Sabourin, 2001

Requirement State Transition Diagram Reviewed Entered Empty Current Future Never Confirmed February 4, 2019 © Robert Sabourin, 2001

Defining Requirement Work Flow
Label State Transitions L F Reviewed Entered Empty Current Future Never Confirmed K E B D H A C G I J February 4, 2019 © Robert Sabourin, 2001

Define steps to implement state transitions Identify and name each state transition Starting state Ending state Input Outcome Roles and Responsibilities When does this transition How does the transition take place How is result communicated? February 4, 2019 © Robert Sabourin, 2001

Bug Flow The “bug” flow is something like this
bug is discovered in testing or reported from the field a bug report form is completed the bug report form is reviewed the bug report is added to the bug list a decision is made, at a bug review meeting, about whether the bug should be fixed if the bug is fixed then the software is re-tested to reconfirm that the bug has indeed been fixed if the bug is not fixed (on purpose!) then a description of the work around is published or made available to help desk staff February 4, 2019 © Robert Sabourin, 2001

Bug Reporting Entered Reviewed Prioritized Assigned Unassigned Fixed
REFUSE Entered Reviewed Prioritized Assigned CHECK TRIAGE DESIGNATE CORRECT MANDATE Unassigned Fixed Closed CONFIRM FAILURE February 4, 2019 © Robert Sabourin, 2001

Bug Reporting Most important reason to report the bug
provide input to decision makers regarding status of product decision to ship is based on status of open bugs a ship decision is among the most important in the entire software development process if a decision is made to fix the bug the description had better help the developer get the job done! it is critical to have high quality bug report information! February 4, 2019 © Robert Sabourin, 2001

Example Bug Flow What do we do when we find a bug? February 4, 2019

Example Bug Flow Development Lead Test Lead Product Manager
Drive Development Technical risks Test Lead Raise concerns Provide massively objective input Product Manager Customer view Business drivers to project February 4, 2019 © Robert Sabourin, 2001

Example Bug Flow Business Decisions Product Manager & Development Lead
Based on objective input from Test Lead Bug prioritization is an important business decision February 4, 2019 © Robert Sabourin, 2001

Example Bug Flow Review Prioritization Re-Prioritization
Test lead reviews all bugs DAILY or MORE FREQUENTLY Prioritization Weekly On Demand Whenever a certain number of bugs are pending Re-Prioritization Whenever business or technical context of project changes! February 4, 2019 © Robert Sabourin, 2001

Finished? How do you know you are finished? February 4, 2019

You know you are finished when …
… the only bugs left are the ones that Product Management and Development agree are acceptable (based on objective SQA input) ... February 4, 2019 © Robert Sabourin, 2001

You know you are finished when …
… the only bugs left are the ones that Product Management and Development agree are acceptable (based on objective SQA input) … At least for now! February 4, 2019 © Robert Sabourin, 2001

Books Quality Web Systems Testing Applications on the Web
Dustin, Rashka, McDiarmid Testing Applications on the Web Hung Q. Nguyen The Web Testing Handbook Steven Splaine, Stefan P. Jaskiel Testing Computer Software Cem Kaner, Jack Falk, Hung Q. Nguyen February 4, 2019 © Robert Sabourin, 2001

Books Software System Testing and Quality Assurance Boris Beizer
Black-Box Testing Dynamics of Software Development Jim McCarthy Software Inspection Tom Gilb, Dorothy Graham February 4, 2019 © Robert Sabourin, 2001

Books The Craft of Software Testing Brian Marick Microsoft Secrets
Michael Cusumano, Richard Selby Why Does Software Cost So Much? Tom Demarco The Software Project Survival Guide Steve McConnell February 4, 2019 © Robert Sabourin, 2001

WWW www.stickyminds.com Software Test and Quality Engineering
Excellent articles and forums James Bach Heuristic and Rapid testing Elisabeth Hendrickson excellent SQA portal Bug based approaches February 4, 2019 © Robert Sabourin, 2001

WWW www.soft.com Software Research Inc. SQA Portal
Software Quality Week Conferences Steve McConnell Software Project Survival Brian Marrick Craft of Software Testing February 4, 2019 © Robert Sabourin, 2001

WWW www.kaner.com Cem Kaner SQA site www.amibug.com
Robert Sabourin Site Presentations, SQA resources Brian Lawrence Site tons of check lists and life cycles tools for download February 4, 2019 © Robert Sabourin, 2001

WWW www.rational.com home of excellent UML references
several white papers This site provides various kinds of information about software development methods and tools. The site already contains information on almost 400 products from more than 160 software vendors February 4, 2019 © Robert Sabourin, 2001

WWW http://www.io.com/~wazmo/qa/
SOFTWARE TESTING HOTLIST contains tons of great pointers and SQA Test web resources. Excellent testing resources related to Browser Compatibility February 4, 2019 © Robert Sabourin, 2001

Just In Time Testing Testing Turbulent Web Based Applications

Similar presentations

Presentation on theme: "Just In Time Testing Testing Turbulent Web Based Applications"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Just In Time Testing Testing Turbulent Web Based Applications

Similar presentations

Presentation on theme: "Just In Time Testing Testing Turbulent Web Based Applications"— Presentation transcript:

Similar presentations

About project

Feedback