Research and Analysis Methods October 5, 2006

Surveys Electronic vs. Paper Surveys –Electronic: very efficient but requires users willing to take them; possibility of technical problems; concerns about privacy –Paper: fewer concerns about confidentiality but often low response rates Issue of length > how do you motivate people to fill your survey in? –Rewards > Introduces possibility of bias –Observer bias: act of measuring changes the thing being measured Versatile: many types (user satisfaction, w usability test, etc.)

Interviews vs. Focus Groups Interview: one person at a time; captures individual differences (individual subjectivity) –Problems: hard to generalize, hard to compare with other interviews (need schedule of questions) Focus groups: many people at one time; people stimulate responses within the group; can come to some group consensus –Problems: self-censorship; lack of privacy and confidentiality

Observation Go into the work site and watch people using a website; you may see –Problems (and how people solve them) –Use of secondary information (e.g. people need to look up words > add a glossary function) –Frequency of use of different parts of the site Generally, observations are not directed –usability tests use directed scenarios – info for scenarios often comes from user observation Problems: getting access to worksites; little value if site is only used occasionally

Market Research Much research already available on general characteristics of some user groups: –Students –Yuppies –Men vs. women Good for demographic information (info about the larger population) –Can help identify characteristics of sample to recruit for surveys, interviews, usability tests Problems: often little guidance for usability decisions (navigation, choice of info on site, etc.)

Site Usage Statistics Possible to get information from the server –Who is using the website (IP address of computer) –What pages are being accessed Problems: –Can’t just count the number of times a paper is called from the server > could just be someone moving back and forth within the site –IP addresses help you identify different users but nothing about their demographics or needs

Comparative Evaluation Ask target users what sites (and what features of other sites) they like –Identify characteristics of those sites and compare them with your own Look at other sites in your segment –Assumption: you are all trying to get the attention of the same target audience –Need to be able to match the functions, text, images that they use

Usability Tests For sites that already exist Identify specific problems through task scenario testing –Pick a typical task (perhaps from observation) –Ask user to complete task and talk their way through the steps (think-aloud protocol) –Observe; may sometimes need to prompt for thoughts and responses Final survey identifies general likes | dislikes –Colours, navigation, images, etc. of this site –Accuracy, completeness, consistency –Easy to understand, emotional involvement

Additional Methods Participatory Design –Include users in the (re)design of the website –Requires an organizational commitment to actually listen to their use (problem sometimes w mgmt vs. labour situations) Paper prototyping –Use paper rather than online prototypes because they are quick, flexible, easy to change, not too finished –Tangible (touch) methods often elicit more emotional/ subjective info from users

Card Sorts Many different versions –See Lazar One use: decide what to put in | take out of a website (content analysis) –Identify content; put one item on one index card –Ask people to sort cards in 5 piles (must have, nice to have, neutral, little use, would never use) –Size of piles: specific size (forced choice) or free choice; forced choice requires people to evaluate/ make decisions Fun for users; if you have many decks of cards, can test lots of people quickly

Error Analysis (Critical Incidents) Useful to find what does not work Error logs, messages to the webmaster, phone calls for help, etc. Sites vary in terms of importance of errors: –What happens if someone doesn’t find info on the computer science dept website? –What happens if someone can’t find info on emergency contraception? May need to classify some user scenarios/ tasks as critical (must be able to complete successfully)

Heuristic (Expert) Evaluation Many problems can be found by educated usability professionals –Design, navigation, site hierarchy (too deep), performance, etc. –Can’t identify subjective likes | dislikes, etc. Usability principles (from Lazar) and design principles (from Williams’ Non-designers Guides) can be applied to improve sites One method for class project can be your group’s own heuristic evaluation of problems –Just need to be able to explain/ classify these –Why do you decide something is a problem…

More Things to Consider Population vs. Sample –Sample needs to represent the population Convenience vs. Random samples –Need to identify potential for bias in your sampling practices do students in the ASU represent all Acadia students? do students in the Wong Centre represent all groups the Wong wants to attract? How many people to survey or test –Surveys: 30+ –Usability tests: 5 (Nielsen) – 20 (statistical validity)  8-9

Types of Information Demographic info –Need to gather info about people in your surveys, interviews, focus groups, observations, usability tests to be sure they match the target users Content Questions –Create consistency by developing a schedule (set) of questions before you start your research –Watch out for leading questions (imply the answer you want to hear) We worked hard on Welcome Week; how successful was it? You had the opportunity to attend Wel Wk; how successful was it? –Probes (encourage responses – and why do you believe that – but don’t add info

Types of Scales Open questions: free answer Closed questions: fixed set of answers (e.g., multiple choice) Semantic differential: (good for emotion and subjectivity) –Warm 12345Cold –Exciting12345Boring Major distinction: –Qualitative Research (open questions, free observation, unstructured inquiry) vs. –Quantitative Research (closed questions, able to apply statistical analysis

Final Thoughts Anonymity vs. confidentiality: when you do research with human subjects, you need to protect them from harm –Anonymous responses – their identities are protected –Confidential responses – the information itself is not revealed except in statistical averages, etc. Reliability vs. validity –Research can be reliable (always gets the same kind of data) but not be valid (the data does not reflect the target population) –Rare to be valid but not reliable!