1. Experimental design ITS class December 2, 2004 ITS class December 2, 2004

2. Quantitative research  Quantify relationships between variables  Measured on a sample of subjects  Relationships expressed through effect statistics  Correlations, relative frequencies, diff. between means  Quantify relationships between variables  Measured on a sample of subjects  Relationships expressed through effect statistics  Correlations, relative frequencies, diff. between means

3. Types of Study  Descriptive  Measure things as they are  Experimental  Take measurements  Try an intervention (treatment)  Take measurements  Descriptive  Measure things as they are  Experimental  Take measurements  Try an intervention (treatment)  Take measurements

4. Descriptive studies  Case  Case series  Cross-sectional  Cohort or prospective or longitudinal  Case-control or retrospective  Case  Case series  Cross-sectional  Cohort or prospective or longitudinal  Case-control or retrospective

5. Experimental studies  AKA longitudinal, repeated- measures, interventions  Without a control group  Time series  Crossover  With a control group  AKA longitudinal, repeated- measures, interventions  Without a control group  Time series  Crossover  With a control group

6. Control group studies  Necessary if treatment effect cannot be “washed out”  Random assignment: min. chance that either group is not typical of population  Single-blind: subjects  Double-blind: experimenter too  Best studies: data analyzed blind  Necessary if treatment effect cannot be “washed out”  Random assignment: min. chance that either group is not typical of population  Single-blind: subjects  Double-blind: experimenter too  Best studies: data analyzed blind

7. Ethical issues  Randomized controlled study may not be ethical  E.g.: Heart disease study over 10 years  Access to treatment  Inform subjects of randomization  Randomized controlled study may not be ethical  E.g.: Heart disease study over 10 years  Access to treatment  Inform subjects of randomization

8. Quality of designs  Quality of evidence for causal relationships  Least for case, case-series  Cross-sectional, case-control  Prospective  Experimental best evidence  Quality of evidence for causal relationships  Least for case, case-series  Cross-sectional, case-control  Prospective  Experimental best evidence

9. Confounding  Two variables may be related for other reasons  Control for potential confounding factors  Two variables may be related for other reasons  Control for potential confounding factors

10. Samples  Generalizability  Representative  Safest: Random sample  Balance:  Stratified random (%s)  Balance on pretest!  Selection bias:  Not representative  Sources: age, socioeconomic status  Self-selection into groups  High compliance  Generalizability  Representative  Safest: Random sample  Balance:  Stratified random (%s)  Balance on pretest!  Selection bias:  Not representative  Sources: age, socioeconomic status  Self-selection into groups  High compliance

11. Sample size  Statistical significance  Big enough to be sure you will detect the smallest worthwhile effect  To be sure =  detect 80% of time  Detect =  95% of time, expect a smaller value if there is no effect, p<0.05  Smallest worthwhile effect =  Smallest to make a diff. to subjects’ lives  Statistical significance  Big enough to be sure you will detect the smallest worthwhile effect  To be sure =  detect 80% of time  Detect =  95% of time, expect a smaller value if there is no effect, p<0.05  Smallest worthwhile effect =  Smallest to make a diff. to subjects’ lives

12. Statistical significance  P-value:  the probability of getting something more extreme than your result, when there is no effect in the population  P-value:  the probability of getting something more extreme than your result, when there is no effect in the population

13. Sources  http://www.sportsci.org/jour/0001/wghd esign.html: Good resource for basic stats http://www.sportsci.org/jour/0001/wghd esign.html:  http://www.softwareevaluation.de/ : Website for EASy-D: a database of evaluation of adaptive systems, also links to research-based usability guidelines http://www.softwareevaluation.de/  http://www.sportsci.org/jour/0001/wghd esign.html: Good resource for basic stats http://www.sportsci.org/jour/0001/wghd esign.html:  http://www.softwareevaluation.de/ : Website for EASy-D: a database of evaluation of adaptive systems, also links to research-based usability guidelines http://www.softwareevaluation.de/