Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement11-1 examples Supplement 11: Testing hypothesis examples

Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement11-2 Requiring a Math Skills Unit Pozo, Susan and Charles A. Stull (2006): “Requiring a Math Skills Unit: Results of a Randomized Experiment,” American Economic Review, 96(2):

Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement11-3 Mathematics matters Research spanning three decades supports what many experienced instructors of economics have long concluded—math matters. Students with greater mathematics preparation attain higher test scores in introductory Economics Ballard and Johnson (2004) find “mastery of extremely basic quantitative skills is among the most important factors for success in introductory microeconomics.” Mathematical competency reduces anxiety in economics classes (Mary Ellen Benedict and John Hoag, 2002). To the extent that anxiety may interfere with the cognitive process, an effective mechanism to correct for math deficiencies is desirable.

Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement11-4 In this paper reports on the results of a controlled experiment with random assignment, which tests whether giving a grade incentive to complete a math skills unit results in higher overall achievement in introductory economics.

Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement11-5 Experimental design The experiment was performed with students enrolled in two sections of principles of macroeconomics taught by the same professor during the spring 2004 semester at Western Michigan University, a large regional university located in Kalamazoo. The assignment to treatment or control group was determined by randomization and not by class section. The experiment included 273 students.

Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement11-6 Treatment vs. control Students in both the experimental and control groups were asked to complete Aplia’s on-line diagnostic math test during the first week of class. The test consisted of 28 questions covering numerical calculations, graphs, units of measurement, area, and simple algebra. Following the due date, students received their scores and information about math deficiencies. Students in the experimental group were reminded that this test score would serve as their math grade, but they could improve their grade by working through the appropriate on-line tutorials and taking Aplia’s post- review test. The students were reminded that the higher of the two test scores would be used in computing final grades. Hence, students with math deficiencies were provided an incentive to improve math skills, while proficient students need bear no further costs. Students in the control group were strongly encouraged to complete the math diagnostic test, to work through the appropriate tutorials, and to take the post-review test to gauge their comprehension of the material. They were informed that neither the pre-review nor postreview score would be used in their final grade.

Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement11-7 The performance in the math test

Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement11-8 The results H 0 :  E –  C  0 H 1 :  E –  C > 0 Can we assume independence of the two populations? Standard deviation of the sample mean. Yes. Because of the randomization procedure in putting students into treatment and control group.

Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement11-9 Can we conclude that Math matters? Due to the random allocation of students into the control and experiment group, there should not be systematic reason to see students’ background (e.g., IQ, training in economics, and math, etc.) before taking the course has any impact on the final result. So, the result is likely due to the “treatment”. What is the treatment? Giving incentive to students to work hard in the additional math test. Two possible channels for the treatment group to obtain better exam results. Treatment group has improved their math skills. Better math skills leads to better performance. Treatment group had acquire the taste of working harder in the course after working hard in the math test.

Ka-fu Wong © 2007 ECON1003: Analysis of Economic Data Supplement END - Supplement 11: examples Testing hypothesis examples