Usability Testing Teppo Räisänen

General Information A person belonging to product’s targeting group commits testing Either finished product of prototype can be used Observations made during testing session are used in improving usability and UI

General Information Often usability tests are used during iterative stages of product development If prototypes are to be used, the appropriate moment must be chosen: Testing with too crude prototypes is not sensible Too late testing will often lead to increased development costs

General Information There are times, when an UI of earlier product can be used as a point of comparison The aim is often set at surpassing some qualities of the earlier product In testing it is important to use formal methods instead of ad hoc reasoning

Testing Vs. Heuristic Evaluation Heuristics and other methods are not exclusionary but complementary Problems that are revealed by testing or by heuristics are often of different kind To get the most out of product, several methods are often used

Challenges of Testing Conditions in usability laboratories are not completely realistic Even field tests have some differences between real usage situations When a person knows he/she is observed, it often haves an effect in person’s behaviour

Challenges of Testing A Hawthorne effect The productivity of a factory had been decreasing An observer was reqruited to find out reasons behind decrease Productivity rose back to former level without any changes to factory’s procedures

Challenges of Testing Finding suitable test person can be difficult Often knowledge about presumed end users of a product can be faulty Marketing researches are used to find out correct user groups, but results of researches may be misleading

Challenges of Testing Heuristics based on a few experts’ opinions are more cost effective than usability sessions In a long run it is still worthwhile to use both methods for gathering information

Preparation for Testing Preparation is a demanding task consisting of many subparts: Test persons are chosen Area of focus is decided Testing environment is prepared Often a pilot test is conducted to try out the arrangements

Preparation for Testing Testing should follow a precise plan (like all software testing) Test plan provides a common guideline for persons involved in testing The quality of testing process is evaluated at some later point using Test plan Test reports

Preparation for Testing Test persons are chosen based on some predefined criteria such as Age group Sex ratio Educational background (Computer user experience) Number of users

Example of User Data AGE DISTRIBUTION 15 – 255 % 25 – 3545 % 35 – 4530 % 45 – 5515 % %

Preparation for Testing If resources available for testing are limited, test persons should be representatives of central target groups With more resources the profile can be expanded Correct definition of end users is especially important when product is of a new kind

Preparation for Testing Often test persons are instructed to perform a set of tasks Those tasks must be defined for tests to be consistent Tasks must be defined in test plan Often tasks used are the most common ones during product’s intented usage

Preparation for Testing In connection with tasks numerical values can be used for defining wanted qualities, e.g. ”User must be able to remove a line from document and save edits in less than 30 seconds” Often earlier products or products of competitors can be helpful in defining limiting values

Preparation for Testing Test lab must be prepared Installation of software and hardware Other elements of infrastructure Field tests are especially demanding (electricity, batteries, lightning etc.) A pilot test should be conducted to avoid problems during actual test sessions

Completion of Testing Test situation should be as natural as possible Normally field tests are easier in this respect Usability labs should resemble the actual using environments as much as possible

Completion of Testing Some products are impossible to test out in lab conditions leaving field tests for only possibility Secretiveness should be avoided in test situation Tell the test person about test lab’s facilities Tell about usability persons’ roles

Completion of Testing After initial procedures the actual test is completed User performs a given set of tasks User’s actions are recorded User should be able to complete tasks without any outside help If, however, user gets stuck he/she must be given some advice

Completion of Testing Actions of observators should be as unnoticeable as possible When measuring discrete values it is important that test are conducted exactly in the same way for each of the test persons A thorough test plan is therefore a necessity

Completion of Testing To provide equal test conditions Test sessions should be completed within short period of time Same personnel should be used for all of the sessions Instructions should be given in written form Test plan should be carefully followed and reports written in an uniform way

Completion of Testing If something goes wrong during testing the case should be carefully documented Afterwards the value of test data collected during faulty test session must be evaluated

After completion of tasks test persons can be interviewed to gather further information Often test persons are rewarded in some way Note: Legistlation &/ bureaucracy may prohibit using money as a form of reward