1 Instructor: Vincent Duffy, Ph.D. Associate Professor of IE Lab 1 – Part 1 Review - Lifelong Learning Fri. Feb. 2, 2007 IE 486 Work Analysis & Design II

2 10 ways to look at a research paper systematically for critical analysis v 1. research ideas/question -what is the purpose? v 2. what background support? - literature review v 3. theoretical basis for analyzing question/hypothesis? v 4. applicability-practical contribution? v 5. theoretical contribution? v 6. appropriate methodologies for carrying out study? (determining variables, data collection, method used to test hypotheses, validity of measures and reliability) v 7. appropriate statistical analyses and assumptions? v 8. presentation of results: what do they really mean? v 9. conclusions drawn: are they reasonable? v 10. future work/research directions: any possibilities?

3 10 ways to look at a research paper systematically for critical analysis v 1. research ideas/question -what is the purpose? v Determine if computers are social actors across cultures v Possible because previous studies showed…humans interact with computers the way they behave with humans - U.S. v Questions – is this generalizable across cultures…maybe, sometimes…(conditional) v Differences in culture; individualistic vs. collectivist behavior; reciprocity

4 10 ways to look at a research paper systematically for critical analysis v 2. what background support? - literature review v Tested repeatedly in U.S. – social-psychology literature, some HCI studies demonstrated resuts that contributed to the Computers as Social Actors paradigm v Nass, 1996; 1999….; colleagues also…including Reeves (1996), Moon (1999), Fogg (1997)… v Reciprocity? (supported?)…literature back to 1960.

5 10 ways to look at a research paper systematically for critical analysis v 3. theoretical basis for analyzing question/hypothesis? Reciprocity? (supported?)…literature back to Computers as Social Actors Paradigm v Method for identifying new hypotheses… v Go to social-psychology literature v Re-write rule for human-human interaction to see If it applies to HCI v Design an experiment to test v Used on 100 social rules in the past

6 10 ways to look at a research paper systematically for critical analysis v 4. applicability-practical contribution? v Computer user interface design… v More specifically…products designed for multiple markets/countries v (marketing and branding and perception that influences behavior in different culture can impact, ultimately, market share). v Impacting how designers consider potential users (more broad perspective needed for ‘who is the user’.) v People informed in the scientific method can contribute to future product development…

7 10 ways to look at a research paper systematically for critical analysis v 5. theoretical contribution? v Could broaden scope of where existing HCI studies can be applied v Also could highlight limitations in relation to past expected generalizability v Could challenge us to re-think how broadly we should (or shouldn’t generalize) how to apply the results v See discussion… especially paragraph 2… v This research extends the Computers as Social Actors in two ways… v Verify previous studies, or extend generalizability of previous findings…(relating to understanding of human-human interaction differences across culture)

8 10 ways to look at a research paper systematically for critical analysis v 6. appropriate methodologies for carrying out study? (determining variables, data collection, method used to test hypotheses, validity of measures and reliability) v 2x2 between subjects test… (vs. within subjects test). v Variables – reliable? Valid? Measures? (is that the way?)

9 10 ways to look at a research paper systematically for critical analysis v 7. appropriate statistical analyses and assumptions? v Confidence to ignore uncontrolled factors?? v Results are clear (differences shown in graph) results of F test are significant… meaning…p value significant and below some alpha value v (likely.05 since many are showing <.01 and words describing the relationships as significant. v Threshold is the level at which …alpha level is likelihood of (type I error) making an error – 5% chance that the reported (significant) relationship is reported incorrectly/in error. v 2 graphs…one for each experiment v Values of hypothesis testing…

10 10 ways to look at a research paper systematically for critical analysis v 8. presentation of results: what do they really mean? T test, F-test. Post hoc test…why? To interpret the ‘signficant differences’ reported from the F tests. Additional items were mentioned earlier in the ‘elevator’ description.

11 10 ways to look at a research paper systematically for critical analysis v 9. conclusions drawn: are they reasonable?

12 10 ways to look at a research paper systematically for critical analysis v 10. future work/research directions: any possibilities?