1. TEACHING STUDY DESIGN PRINCIPLES VS. DATA ANALYSIS Tisha Hooks and April Kerby Winona State University The authors grant permission for the ASA Section on Statistical Education to post these slides.

2. TEACHING STUDY DESIGN PRINCIPLES VS. DATA ANALYSIS Tisha Hooks and April Kerby Winona State University thooks@winona.edu | akerby@winona.edu 2015 Joint Statistical Meetings Seattle, WA

3. Motivation Curriculum Guidelines for Undergraduate Programs in Statistical Science - Recommended Curriculum for Statistics Majors  Statistical Methods and Theory  Statistical theory  Exploratory data analysis  Design of studies and issues of bias, confounding, etc.  Statistical models  Data Manipulation and Computation  Mathematical Foundations  Statistical Practice  Pedagogical Considerations (which includes the integration of training in professional conduct and ethics)

4. Motivation Curriculum Guidelines for Undergraduate Programs in Statistical Science - Recommended Curriculum for Statistics Majors  Statistical Methods and Theory  Statistical theory  Exploratory data analysis  Design of studies and issues of bias, confounding, etc.  Statistical models  Data Manipulation and Computation  Mathematical Foundations  Statistical Practice  Pedagogical Considerations (which includes the integration of training in professional conduct and ethics)

5. Motivation Though these topics were introduced early in our curriculum and reinforced throughout, we were motivated to create a new course: Principles of Study Design. Why?  Concern that our students saw only “data analysis” as the statistician’s primary responsibility. Other important concepts (proper procedures for collecting data, bias, confounding, ethical considerations) were too often secondary thoughts to students.  Concern that our new Data Science majors wouldn’t see enough of these concepts in their upper-level courses.

6. Motivation To students: Why are you taking this course?  “Statistics is not just a collection of computational techniques. It is a way of thinking about the world. Anyone can take a set of numbers and apply formulas to them... There is no point to analyzing data from a study that was not properly designed to answer the research question under investigation. In fact, there's a real point in refusing to analyze such data lest faulty results be responsible for implementing a program or policy contrary to what's really needed.” -- Gerard E. Dallal  "The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.” -- John Tukey

7. Course Catalog Description Prerequisite: one year of introductory statistics This course will provide an introduction to study design and proper methods of data collection. Students will gain an overview of… the research cycle, basic principles of experimental design, observational studies, effective design of survey instruments, examples of study bias, and ethical considerations in the conduct of research.

8. Course Outline I.Overview of Research Cycle a.Formulating a research question b.Study design c.Data collection d.Descriptive and inferential analysis e.Drawing appropriate conclusions II.Design of Experiments a.Basic definitions (factors, treatments, exp. units) b.Confounding variables c.Fundamental concepts (control, randomization, replication) d.Placebos, blinding e.CRDs, RCBDs, crossover studies, longitudinal studies

9. Course Outline III.Observational Studies a.Prospective, retrospective, and cross-sectional studies b.Effects of confounding variables c.Matching methods IV.Surveys a.Sampling methods (probability sampling methods, non-probability sampling methods, comparison of sampling error estimates) b.Questionnaire design c.Reliability and Validity d.Bias

10. Course Outline V.Ethical Considerations in the Design of Studies a.Ethical treatment of research subjects b.Responsibility to apply sampling and analysis procedures scientifically, without pre-determining the outcome c.Responsibility to clearly report the intent of a study, how it was performed, and any limitations on its validity

11. Possible Learning Activities  Reading assignments  Journal articles (students critique study designs / data collection procedures)  Study results as reported by the media (students critique)  Journal articles on the design of survey questionnaires  Readings on Ethics (e.g., ASA ethical guidelines, “Ethical issues in oncology statistics” by Peter Thall, Statistical Methods in Medical Research, 2002)  Case studies  Small-group and classroom discussions  Writing assignments  Presentations

12. Possible Learning Activities  Case study for discussing confounding during class Questions for students: 1.Is this conclusion warranted? Why or why not? 2.If not, what would you do? “The results of this study indicate that an increased number of ECT treatments leads to a larger gain in functional status (p =.004).”

13. Possible Learning Activities  Design studies before introduction of design concepts; critique their own studies after formal instruction.  Give students frequent opportunities to plan data collection. Evaluating each other’s work is also an excellent learning opportunity! Example: Your task is to design an experiment to compare the wear characteristics of four competing brands of tires. You have four cars of a standard model and four tires of each brand. The tires will be placed on the cars, and the cars will be driven 30,000 miles on a 2-mile racetrack. Source: An Introduction to Statistical Methods and Data Analysis by Ott and Longnecker

14. Possible Learning Activities  Critique existing questionnaire designs  Give several examples of bad questions and task the students with making improvements  Give several good examples, as well  Revise and/or create new questionnaires for clients (e.g., Student Senate, departments on campus)

15. Possible Learning Activities  Introduction to ethical issues encountered by statisticians  Introduce students to infamous studies Milgram’s behavior study of obedience Stanford prison experiment Willowbrook hepatitis study Tuskegee syphilis experiment Students discuss how these studies violate their own ethics and then the Belmont Report and the Nuremberg Code.  Describe a time when you found yourself in an ethical quandary as a professional statistician. Have students describe how they would react.  The ASA Committee on Professional Ethics has several case studies on their website that can also be used.

16. Possible Learning Activities  Large-scale group project that gives students an opportunity to demonstrate what they have learned  Devise a research question  Write and present a research proposal that clearly outlines the study design and data collection plan  Collect and analyze data  Present project to peers and provide written summary

17. Student Feedback The thing I liked most about this course was…  “I liked learning about the different ethical dilemmas that we may run into as statisticians.”  “I enjoyed the class discussions and presentations.”  “I liked the casual discussion type atmosphere.”  “Actually getting to construct a few legitimate surveys”  “The real life studies.”  “The topic itself. I think this side of stats is so interesting. I really enjoyed this course.”  “It was a nice change of pace from quantitative classes.”

18. Final Thoughts  The course emphasizes that a statistician’s job is more than data analysis and enhances student understanding of these other important concepts.  This course can be challenging to teach.  It is also very enjoyable, rewarding, and worthwhile.

19. Thank you! Tisha Hooks (thooks@winona.edu) April Kerby (akerby@winona.edu) We anticipate an open position starting Fall 2016!