1. Summarized by
Chaky
For more details, please read,
Really good book.

2. Outline Quantitative Controlled experiments Questionnaires
Eye-tracking
Cognitive modeling
Analysis: statistics
Qualitative
Interviews
Focus groups
Analysis: Thematic approach, grounded theory approach

3. Quantitative

4. Controlled experiments
Purpose: to find whether there is a effect of X on Y
Independent and dependent variables
Within-subject and between subject
Within-subject – one participant all conditions
Reduce individual differences
Use Latin squares for ordering tasks
Between subject – one participant one condition
When there is noticeable learning or exposure effect, medicines or teaching methods
Common measures are time, errors, and user satisfaction ratings.
Good experiments should be replicable by others.

5. Questionnaires Useful for quantifying subjective ratings
Avoid ambiguity in words
Avoid very lengthy questionnaires
Grouping similar questions help
Likert-scale
Strongly agree – disagree

6. Eye-tracking Useful for capturing visual attention
Identifies region of interest
Often used in website, games, or interface
Design
Prepare stimuli
Prepare different tasks
Record eye-tracking
Triangulate the data with self-reports

7. Cognitive modeling Measuring performance based on execution time
Goals-Operators-Methods-Selections (GOMS)
Keystroke-Level Model (KLM)
Predict the execution time of tasks based on keystrokes or moving hands
Cognitive-Perceptual-Motor (CPM)
Predict execution time based on Model Human Processor (MHP)

8. Quantitative analysis
Inferential analysis
Make sure your data does not occur by chance
Methods
t-test for comparing two groups when data are normally distributed, if not, use Wilcoxon or Mann-Withney
ANOVA for comparing more than two groups when data are normally distribute. If not, try Friedman or Sheirer-Ray-Hare test.
X2 (Chi-squared) test for comparing actual and expected
Use Pearson correlation to show association

9. QuaLitative

10. Interviews Provide in-depth insights Ask open-ended questions
Don’t influence the participants
Always reaffirm participants’ meaning
Reaffirm vague words such as “good”, “nice”
Record textual and audio data for latter analysis

11. Focus groups Similar to interviews
But change from one-to-one to a focus-group
Should not exceed six or seven participants
Avoid leader in the focus-group
Allow everyone in the group to talk
Privacy and trust can hinder effectiveness in focus group

12. Qualitative analysis Thematic approach Grounded theory approach
Find common occurrences in the data and identify themes
Grounded theory approach
Use data to form a theory
Steps
Data are coded into concepts
Concepts are grouped into categories
Categories are coded into theory

13. Conclusion Experiment should be designed as simple as possible
Designing experiment includes consideration of the research question, the data you want, how to eliminate noise, and the way to analyze the data
If your research question is complex, break them into series of simple experiments, rather one complex experiment
Best combine quantitative and qualitative measures
Conduct pilot study if possible