CCT 333: Imagining the Audience in a Wired World Class 8: Quantitative Methods

Quantitative Methods Unlike qualitative, involves metrics Everything reduced to numbers Not just descriptive (e.g., counting) but inferential (e.g., predictive, indication of correlation)

Experiments Controlled specific measurement of phenomenon Not just X related to Y (correlation) but X causes Y (causation) - provided overarching theoretical model is correct of course Examples of use in design?

Benefits and Limitations Controlled environment Precise responses Causality can be inferred Must constrain outside influence Can be hard to measure and determine causation in complex multivariate phenomena Creates fake tasks in fake contexts - validity issues

Data Mining Observation without being there - quantitative artifacts - e.g., Web access logs, click regions, turnstile counts, etc. Records consequences of actual action Examples of use?

Benefits and Limitations Results based in actual action vs. perception or intent Potential for complete sample – not easily achieved in other methods Once infrastructure set, data collection can be automatic Privacy concerns - data could be used for less than noble purposes Evidence of actions might not reflect motivations for why actions are done

Surveys/Questionnaires Common method of obtaining information from broad cross-section of people Quality of information directly depends on who is surveyed and the quality of questions asked Online tools help - e.g.,

Sampling Who is included in sample? How are they reached? Responders vs. non-responders - are they qualitatively different groups? Sampling errors and confidence intervals (or, what +/- 3.5%, 19 times out of 20 means…) Examples of sampling error

Question Guidelines Understandable Unambiguous Data is relevant to research goals Data can be easily analyzed Limited in scope - take respondent’s attention span and willingness to help into account!

Question Guidelines (2) Specific better than general Open/closed-ended questions - benefits and challenges Opening and closing preamble and instructions important - especially if you’re not there to supervise Test it before you use it!

Variable Data Types Nominal (categorical differences; student, staff, faculty, etc.) Ordinal (rank order implied; e.g., year of study) Interval (ranks w/ equivalent intervals - e.g., grade point averages) Ratio (intervals amenable to mathematical manipulations; e.g, age in years Pick the right one for the question at hand!

Likert Scale Questions 1-5, 1-7, 1-9 scales Midpoint - what does it mean? If no opinion, give that option Can force decisions with even number of options (1-6) Take care in too many consecutive items with same polarity of options - leads to patterned responses

Scales Set of related questions measuring attitudes, beliefs, orientations etc. Ex: multiple intelligences, personality tests (others?) Scales must actually measure what they claim, not be redundant Scale reliability - consistency in response patterns Verify authenticity (esp. in web searches) - many scales are meaningless, published scales generally stronger

Data analysis Simple description (55% of respondents are male, 75% under 25 said yes, etc.) Cross-tabulation (males answered this, females answered that) Correlation (direction and strength of link between two variables - slope and r-values) Causation (inferential tests, confidence of variation)

Descriptive (

Cross Tabulation primaries/results/epolls/index.html#WIDEM

Correlation angereport.com/sh owDiary.do?diaryId =5106www.burntor angereport.com/

Causation? Assuming causation can be dangerous in previous examples Hard to prove causation from correlation - must tie to logical explanation of correlation (which can itself be debatable) Ex: “mean-world” syndrome - does TV watching cause higher perceived dangers in the world, or do paranoid people seek shelter from the world in their houses and end up watching more TV?

Org. research example Social network questionnaire - who trusted whom in six domains Correlated with three scales - interdependence, independence and proactivity Correlational study - what relations existed between scales and position in network? Causation - not easy to prove Relations verified by respondent reflection and personal experience Organizational redesign implications

Implications for Final Project Experiments and data mining – probably not useful given circumstances, although latter might be used to gauge interest in topic Surveys – perhaps – depends on nature of population targeted

Next week Complex User Interaction issues and scenarios/requirements planning