Writing Test Cases Why writing and tracking test cases is important? What is a test design specification? What is a test case specification? How test procedures should be written?

YES, A CRASH IS A FAILURE I was involved in a project that used an outsourced testing company for configuration testing of a multimedia program. They weren't the best choice but were the only ones available at the time. To make sure the job went smoothly, detailed test design specs, test case specs, and test procedures were submitted to the testing company so that there would be no question as to what would and wouldn't be tested. Several weeks passed and the testing seemed to be going smoothly—too smoothly—when one day the lead tester on the project called. He reported on what his team had found for the week, which wasn't much, and just before hanging up asked if he should be reporting bugs on things that weren't listed in the documentation. When asked why, he said that since the first day they started testing his team had occasionally seen these big white boxes that said something about a “general protection fault.” They would dismiss them but eventually their PC screens would turn bright blue with another cryptic serious failure error message and they would be forced to reboot their machines. Since that specific error wasn't listed as one of the fail criteria, he wasn't sure if it was important and thought he should check. The moral of the story is that you can't assume that another tester will view the product the same way you will. Sometimes you may have to define the obvious in your test cases—even that a crash is unacceptable.

Test Documentation Test Design Specification Test Case Specification Test Procedure

Why writing and tracking test cases is important? Organization. –The cases may have been created by several testers over the course of several months or even years. Proper planning will organize them so that all the testers and other project team members can review and use them effectively. Repeatability. –run the same tests several times to look for new bugs and to make sure that old ones have been fixed. Tracking. –How many test cases did you plan to run? How many did you run on the last software release? How many passed and how many failed? Were any test cases skipped? Proof of testing –In a few high-risk industries, the software test team must prove that it did indeed run the tests that it planned to run

The Goals of Test Case Planning Ad hoc testing –No test case planning

Test Planning Overview

Test Plan Project test plan: –Written at a very high level. –Specifies strategies, schedule and risks. –It breaks out the software into specific features and testable items and assigns them to individual testers. – But it doesn't specify exactly how those features will be tested.

Test Planning Overview

Test Design Specification Test design specification –The next level of detail that defines the testing approach for individual software features –It doesn't, however, give the detailed cases or the steps to execute to perform the testing. IEEE 829 states that the test design specification “refines the test approach [defined in the test plan] and identifies the features to be covered by the design and its associated tests. It also identifies the test cases and test procedures, if any, required to accomplish the testing and specifies the feature pass/fail criteria.”

Test design specification Identifiers. –A unique identifier. Features to be tested. –A description of the software feature covered by the test design spec—for example, “the addition function of Calculator,” “font size selection and display in WordPad” –Identify features that may be indirectly tested as a side effect of testing--For example “Although not the target of this plan, the UI of the file open dialog box will be indirectly tested in the process of testing the load and save functionality.” –List features that won't be tested--For example “Because testing Calculator's addition function will be performed with automation by sending keystrokes to the software, there will be no indirect testing of the onscreen UI.”

Test design specification Approach. –More detailed approach used to test the features (than the test plan). –Describe the technique to be used, and explain how the results will be verified. –For example “A testing tool will be developed to sequentially load and save pre-built data files of various sizes. The number of data files, the sizes, and the data they contain will be determined through black-box techniques and supplemented with white- box examples from the programmer. A pass or fail will be determined by comparing the saved file bit-for-bit against the original using a file compare tool.”

Test design specification Test case identification. –A high-level description and references to the specific test cases that will be used to check the feature. –It should list the selected equivalence partitions For example, Pass/fail criteria. –Describes exactly what constitutes a pass and a fail of the tested feature. Test Case ID# Check the highest possible value Test Case ID# Check the lowest possible value Test Case ID# Check several interim powers of 2

Test Planning Overview

Test Case Specification Test Cases IEEE 829 states that the test case specification : “documents the actual values used for input along with the anticipated outputs. A test case also identifies any constraints on the test procedure resulting from use of that specific test case

Test Case Specification Test Cases –Identifiers. A unique identifier –Test item. This describes the detailed feature, code module –Input specification. –Output specification –Environmental needs (hardware, software, test tools) –Special procedural requirements (unusual things to be done to perform the test) –Intercase dependencies (If a test case depends on another test case)

Test Case Specification Example of a test case templatetest case template Or use spreadsheets OK RESULTSINPUTS Special NotesTest Cases Var Z ….Var YVar XVar 3 ….Var 2Var 1 TC0401 TC0402 TC0501 …

Test Planning Overview

Test Procedures Test Procedures or test script spec. How the test will be run. Described manually or using a tool as script. IEEE 829 states that the test procedure specification “identifies all the steps required to operate the system and exercise the specified test cases in order to implement the associated test design.”

Test Procedures Test Procedure –Identifier. A unique identifier –Purpose: & reference to the test cases –Special requirements: (special testing skills, or equipment) –Procedure steps: (step-by-step details of exactly how to perform)

Test Procedure Procedure steps. Log. Tells how and by what method the results and observations will be recorded. Setup. Explains how to prepare for the test. Start. Explains the steps used to start the test. Procedure. Describes the steps used to run the tests. Measure. Describes how the results are to be determined—for example, with a stopwatch or visual determination. Shut down. Explains the steps for suspending the test for unexpected reasons. Restart. Tells the tester how to pick up the test at a certain point if there's a failure or after shutting down. Stop. Describes the steps for an orderly halt to the test. Wrap up. Explains how to restore the environment to its pre-test condition. Contingencies. Explains what to do if things don't go as planned.

Test Procedure See Examples Test Procedure Steps

Detail Versus Reality IEEE Std Standard for Software Test Documentation –Highly detailed test case specs reduce ambiguity, make tests perfectly repeatable, and allow inexperienced testers to execute tests exactly. –But, more time and effort, can make updates difficult –As a software tester developing test cases, you need to work toward these goals—but their level is determined by your industry, your company, your project, and your team. For example: medical device or video game –Paper printouts ??? –More efficiently presented today as a spreadsheet or a database (Test Case Management).

Detail Versus Reality Condense the information into a more efficient means of communicating it Each line of the matrix is a specific test case and has its own identifier Common items to all these cases could be written once and attached to the table. (e.g. input spec, output spec, … )

Testing Activities Identify Design Build Execute Compare Test conditions (“What”): an item or event to be verified. How the “what” can be tested: realization Build test cases (imp. scripts, data) Run the system Test case outcome with expected outcome Test result

Test Process and Documentation Analysis & Design Execute & Compare

Quiz What are the four reasons for test case planning? What's the purpose of a test procedure specification? At what level of detail should test procedures be written? Other than a traditional document, what means can you use to present your test cases?