Nov 20, Fall 2006IAT 4101 Play Testing Software Testing Play Testing Team Structures

Nov 20, Fall 2006IAT 4102 Software Testing  Goal: “Prove” that the software works –Matches the functional specification –Matches the performance specification  Real Proof: NP-Hard problem –Simulate all the possible runs –Exponential growth in testing time –Each branch point multiplies possible program state by 2

Nov 20, Fall 2006IAT 4103 Testing Method  Run the program using a suite of data –Data suite exercises as much of the program’s code as possible –Each data set is accompanied by expected results –Where the results are unexpected, there is a bug!

Nov 20, Fall 2006IAT 4104 Generating Test Suites  Black Box Testing –Generate test data based purely on interpretation of the specifications –Don’t look at the code! –Code is a Black Box

Nov 20, Fall 2006IAT 4105 Generating Test Suites  White Box Testing –Use the code to generate tests with the purpose of exercising all blocks of code  Central concept: Coverage  Cover all of the parts of the programs code –Ensure that all lines of code run at least once

Nov 20, Fall 2006IAT 4106 Combine Both Approaches  Black Box + White Box test suites –Complementary roles –Hard to tell from the data what approach was used –Aims to generate that ever-elusive “proof”  Test suites: –Great for testing Compilers –Perhaps not so great for interaction and games

Nov 20, Fall 2006IAT 4107 Random Testing  Some applications are hard to test:  Random test –Give up the idea of formal coverage –Hope for stochastic coverage!  Sometimes this depends on actual human interaction –Need to get lots of people –Beta testing by customers

Nov 20, Fall 2006IAT 4108 Play Testing  Play testing follows software testing –Overlap of phases  Software Testing + More –Tuning game play –Tuning that Flow experience!  Discover Intellectual Space –Does the player Get it?? –Less important in established genres

Nov 20, Fall 2006IAT 4109 Team Structure  Fundamental issue: Willful blindness  Don’t fall in love with your code!  Having testing team members be different from your coders is vital  A Software Testing team  A Play Testing Team

Nov 20, Fall 2006IAT Play Testing Team  Internal team –Manages play-testing people –Does own play tests –Recruits outsiders for a week or two  External team –A population sample –Varying skills –Unfamiliar with product –Not in love with your product! –Not yet bored with the product

Nov 20, Fall 2006IAT External Testers  Do they like the game? –They may lie to you –Politeness effect If they can’t say why they like it, you have a problem  Do they get frustrated? –What common areas? –Often skill-based Internal team judges player’s skill, also

Nov 20, Fall 2006IAT External Testers  FAQ –Be prepared to deal with the frequent questions in the final game

Nov 20, Fall 2006IAT External Testers: Half-Life  People near Valve’s offices who sent in registration cards  Designers sit quietly while player struggles –Designer takes notes  Typical 2-hour session –100 action items  First sessions absolutely vital –Learn what was fun –200 sessions total

Nov 20, Fall 2006IAT Half-Life: Fine tuning  Add instrumentation –Player position, time, health, weapons –Activities: Game save, dying, being hurt, solving a puzzle, fighting a monster…  Graph series of sessions together –Spot too long with no encounters –Spot long periods of too much health –Spot long periods of too little health

Nov 20, Fall 2006IAT Half-Life  Most important playtesting outcome:  Clearly identified good and bad ideas  Allowed people to…

Nov 20, Fall 2006IAT Half-Life  Most important playtesting outcome:  Clearly identified good and bad ideas  Allowed people to Abandon Bad Ideas  Playtesting provides the evidence needed

Nov 20, Fall 2006IAT Advice  Don’t get defensive! –The tester opinion is important  Testers: Stick to your findings! –Respectfully point out problems  Mix up the hardware –Be honest about the specs on the box

Nov 20, Fall 2006IAT More Advice  Discourage Legacies –Don’t let bad decisions live forever! –MS-DOS!