1. Evaluating the Impact of Change in Curriculum and Teaching Bob delMas, Elizabeth Fry, Laura Le and Anelise Sabbag Funded by NSF DUE-1043141

2. Session Overview Introduce three assessment instruments Participants examine subsets of items Background on assessment instruments Participants explore possible use of the assessment instruments in their curriculums Wrap-Up: recommendations for using the instruments

3. But First, Some Light Entertainment

4. Three Assessment Instruments GOALS: 27 forced-choice items –Study design –Reasoning about variability –Sampling and sampling variability –Interpreting confidence intervals and p-values –Statistical inference –Modeling and simulation MOST: Measure of statistical thinking –4 real-world contexts –open-ended and forced-choice items Affect survey: attitudes and perceptions

5. Review Assessment Items GOALS items 15, 16, 17 and 18 MOST items 7, 9, 10 and 11 Affect items 1, 3, 4, 5 and 9 Answer these questions for each item –What do you think the item is trying to assess? –How do you think students would answer? –Are there options that you think students would choose/not choose?

6. Small Groups Form a group with one or two other people and discuss your perspectives about the items What do you think the item is trying to assess? How do you think students would answer? Are there options that you think students would choose/not choose?

7. GOALS Items Interpret Confidence Interval Context High School Class Wants to estimate average wt. of chocolate in Chocolate Chip cookies Rinsed away dough & weighed chocolate Estimated 95% Confidence Interval

8. GOALS Items 15 through 18 Interpreting Confidence Intervals For questions 15 to 18, indicate whether the interpretation of the interval provided is valid or invalid. 15.With 95% confidence, we can infer that each cookie for this generic brand has approximately 5.65 to 6.35 grams of chocolate. ValidInvalid 16.With 95% confidence, we can infer that the average weight of chocolate per cookie in this generic brand is 6 grams plus or minus 0.35 grams. ValidInvalid 17.We can infer that 95% of all cookies from this generic brand will have between 5.65 to 6.35 grams of chocolate. ValidInvalid 18.With 95% confidence, we can infer that the interval of 5.65 to 6.35 grams includes the true average weight of chocolate per cookie. ValidInvalid

9. MOST Item: Comparing Groups Context 20 students Random assignment 10 students: Strategy A 10 students: Strategy B Mean for A = 5 points higher than mean for B

10. MOST Item: Comparing Groups Context 7.Explain how the researcher could determine whether the difference in means of 5 points is large enough to claim that one preparation strategy is better than the other. (Be sure to give enough detail that someone else could easily follow your explanation in order to implement your proposed analysis and draw an appropriate inference (conclusion).)

11. 100 students: Strategy A 100 students: Strategy B MOST Item: 2 nd Comparing Groups

12. 9. Imagine that a p-value was produced for the situation described in Questions 7 and 8 where there were 10 students in each group. Would you expect to get the same p-value in this situation where there are 100 students in each group, or would you expect a larger or smaller p-value? Circle the letter of the best response to the question. a)Smaller p-value b)Same p-value c)Larger p-value MOST Item: 2 nd Comparing Groups

13. 10.Circle the letter of the best explanation for your answer in Question 9. a)Because a larger sample size produces a more accurate p- value. b)Because a larger sample size decreases the amount of variation in the distribution of statistics. c)Because a larger sample size allows us to generalize better to a whole population. d)Because a larger sample size will give us more variation in the results. e)Because the observed mean difference and null model didn’t change. f)Other_______________________________________ MOST Item: 2 nd Comparing Groups

14. MOST Item: Estimation Context

15. 11.Explain how you could help them come up with a good estimate of how many hours a student at that school typically works at outside jobs based on their sample data. (Be sure to give enough detail that someone else could easily follow your explanation in order to implement your proposed analysis and draw an appropriate inference (conclusion).) MOST Item: Estimation

16. Affect Items 1.This course helped me understand statistical information I hear or read about in the media. 3.This course helped me learn how to make better decisions when faced with uncertain outcomes. 4.This course helped me realize that how data are produced or collected has an impact on the scope of conclusions that can be made. 8.Learning to (use software/create models in TinkerPlots) helped me learn to think statistically. 9.I believe I am well prepared for future classes that require an understanding of statistics. RESPONSE SCALE Strongly DisagreeDisagreeAgree Strongly Agree

17. Instrument Background General background on CATALST For each instrument –Assessment focus –Design features Use of instruments in evaluation of CATALST curriculum

18. CATALST Project 4-year NSF-funded project (DUE-0814433) Radically different intro statistics course –No t-tests; Use of probability for simulation and modeling –Coherent curriculum that builds ideas of models, chance, simulated data, inference from first day –Immersion of students in statistical thinking –Activities based on real problems, real data

19. GOALS (Goals and Outcomes Associated with Learning Statistics) Focus on types of reasoning to be developed in a first statistics class 27 forced-choice items 2 versions, same except for items 19-22 –One for students in course that teaches randomization methods for inference –One for students in a course that teaches traditional methods of inference

20. MOST (Models of Statistical Thinking) Focus on how students think about problems that involve statistical inference 4 real-life contexts designed to elicit statistical thinking 11 items –4 open-ended questions, one for each context –7 forced-choice questions to provide more detail 1 version –Can be used in traditional or randomization-based courses

21. Affect Survey Focus on attitudes and beliefs at the end of an introductory statistics class 12 items, each using 4-point scale: Strongly Disagree – Disagree – Agree – Strongly Agree –4 items about the course –4 items about using software –4 items about statistics 2 versions –One for classes (e.g, CATALST) that use TinkerPlots™ software –One for classes that use any other software

22. General Purpose and Use of Instruments Use of instruments –To evaluate important student outcomes: reasoning, thinking and attitudes/beliefs In CATALST, 3 groups were compared: –CATALST class at U of M (Spring 2012) 3 sections all taught the same way, with final version of materials Similar students enrolled in all sections –CATALST classes at 4 other schools (Spring 2012) Different types of students, requirements –Non-CATALST: from 6 schools (Fall 2011 and Spring 2012) Served as a type of control group

23. GOALS: Bootstrap Confidence Intervals of difference in Mean Proportion Correct for Each Item (CATALST: n = 289; non-CATALST: n = 440)

24. Comparison: Confidence Intervals Items Mean Percent Correct

25. Comparison: Confidence Intervals Items Mean Total Number Correct NMEANSD UofM CATALST 1382.670.80 Non-UofM CATALST 1512.310.85 Non- CATALST 4402.100.97

26. Analyzing MOST Responses Statistical Thinking Checklist –Wild and Pfannkuch’s (1999) framework of statistical thinking –Thinking like an expert from expert-novice literature Students in Randomization- based Courses Students in Traditional Course Modeling Integration of statistical and contextual Domain-specific knowledge Statistical Models

27. MOST Results: Exam Strategy, Description Item CATALST UofM students (n = 138) CATALST non-UofM students (n = 120) Modeling Students described a Null Model 39.49%21.25% Integration of Statistical and Contextual Students described the Context within Model 48.19%32.92% Modeling Students provided a Description of the Simulation 43.84%35.42% Domain-Specific Knowledge Students knew how to Make Conclusions based off of the (possible) statistical results 38.77%24.58% Integration of Statistical and Contextual Students described the Context within Conclusion 28.62%14.17%

28. MOST Results: Exam Strategy, Description Item non-CATALST students (n = 187) Modeling Students described a Null Hypothesis 4.55% Modeling Students described an Alternative Hypothesis 1.87% Integration of Statistical and Contextual Students described the Context within Hypotheses 3.21% Domain-Specific Knowledge Students described needing to Check Assumptions about the Data 0.80% Statistical Models Students Stated type of test (z, t, etc.) that would be appropriate for the problem 17.11% Domain-Specific Knowledge Students knew how to Make Conclusions based off of the (possible) statistical results 17.11% Integration of Statistical and Contextual Students described the Context within Conclusion 5.35%

29. MOST Results: Exam Strategy, M-C items Group Students who chose “smaller p-value” and correct reason CATALST UofM students (n= 138) 38.41% CATALST non-UofM students (n = 120) 15.83% non-CATALST students (n = 187) 18.18%

30. MOST Results: Study hours, interval estimate item CATALST UofM students (n = 138) CATALST non-UofM students (n = 120) Modeling Students described using the Data to find an Interval Estimate 30.07%15.42% Integration of Statistical and Contextual Students described the Context within Model 23.19%12.92% Modeling Students provided a Description of the Simulation 36.23%17.92% Variation Students mentioned the Need for a variability measure for the CI calculation 48.91%17.92% Statistical Models Students knew the CI calculation 40.22%12.50% Domain-Specific Knowledge Students knew how to Make Conclusions based off of the (possible) statistical results and Interpret the interval 22.83%7.92% Integration of Statistical and Contextual Students described the Context within Conclusion 28.99%13.33%

31. MOST Results: Study hours, interval estimate item non-CATALST students (n = 187) Statistical Models Students Stated type of test (z, t, etc.) that would be appropriate for the problem 3.74% Domain-Specific Knowledge Students described needing to Check Assumptions about the Data 0.27% Variation Students mentioned the Need for a variability measure for the CI calculation 4.81% Statistical Models Students knew the CI calculation 1.60% Domain-Specific Knowledge Students knew how to Make Conclusions based off of the (possible) statistical results and Interpret the interval 3.48% Integration of Statistical and Contextual Students described the Context within Conclusion 7.49%

32. Affect Survey Results

33. Evaluating the Impact of Change Look at the full instruments! Questions to think about: –Would these provide useful information to you about your students if you knew how they performed on these assessments? –What could you learn from data that you would gather from your students using these assessments? –What kind of data would you want to compare your student data to? (e.g., national sample, other statistics courses at your institution, other future statistics courses)

34. Summary and Final Thoughts Project Goal: Develop good assessment instruments that provide useful information for curriculum change and development Purpose of these instruments: research and evaluation projects

35. Summary and Final Thoughts Instrument development is time consuming, taking several years of testing and revision Still under development

36. Summary and Final Thoughts Not designed to replace other kinds of assessments in course (e.g., quizzes, minute papers, projects) We want people to use the instruments Or, you can use some of the items from an instrument BUT…

37. If you use only some items from an instrument DON'T use them for program evaluations or for research projects DO use them as part of assessments in your own course

38. Thank You for your Participation If you have comments, issues, ideas of what is missing in the instruments, please send one of the PIs an email: Joan Garfield: jbg@umn.edu Bob delMas: delma001@umn.edu Andy Zieffler: zief0002@umn.edu