1. Validity of web-based placement testing outcomes: Some recent findings Hoi Suen Professor of Educational Psychology Pennsylvania State University University Park, PA 16802-3108, U.S.A Office: 814-865-2235 Fax: 814-863-1002 Email: HoiSuen@psu.edu Website: http://suen.ed.psu.edu

2. Distance learning and assessment We deliver and facilitate instructional activities over the Web or other media. However, we require students to take high- stakes exams at designated centers where the tests are secured and where the identities of the examinees can be authenticated, and the examinees can be proctored and observed.

3. Unproctored web-based Exam Security and Authentication Without the physical presence of a human proctor, there is no known easy way of ensuring that the responses to exam questions in fact originated from the examinee; nor that the examinee did not obtain assistance from other people, from printed reference books, from his/her own computer hard drive or even from the web itself, while taking the exam.

4. Assumption regarding unproctored web- based placement testing Students should not and would not cheat when taking a placement test, because cheating in this situation would only hurt the student himself/herself. If a student did better due to cheating and were thus placed into a course more advanced than his/her capabilities, the student would be the one who would suffer.

5. We theorized that: 1.When given convenient opportunities to improve test scores, students will tend to take advantage of these opportunities whether these opportunities are appropriate or not and whether the end result will hurt the student or not. This is because students today have been brought up in a high-stakes testing environment and that they generally believe that it is important to score as high as possible in any exam. Therefore, even for a placement test, when administered through an unproctored web- based setting, students will cheat.

6. We theorized that: 2.As a result of cheating in these placement tests, a substantial portion of students will be placed into courses beyond their capabilities. 3.When confronted with a substantial portion of students being placed beyond their levels of abilities, instructors would change instructional method and/or contents to adjust to the students’ lower level of abilities; i.e., as an indirect result of unproctored web-based placement testing, instructors will tend to “water down” instruction.

7. Study to test the theory designed and executed by: Dawn Zimmaro, Ph.D. Research Associate University Testing Center Penn State University University Park, PA 16802 Email: dmz115@psu.edu

8. SAMPLE Penn State freshmen placement testing program Focused on math testing: The mathematics placement exam was a multiple-choice format. The test contained 72 five-response option items based on a set of 29 mathematical topics ranging from arithmetic of integers to inverse trigonometric functions. Matched samples: Matched samples of web-based and paper testers based on SAT Math and high school grade point average. A total of 1,010 exactly identical matches on these two variables were identified. The final sample included 1,010 Web testers and 1,010 paper testers with an average high school grade point average of 3.79 (s.d. 0.25) and an average SAT Math score of 628 (s.d. 66).

9. Major Hypotheses: Focus of this presentation I.After controlling for differences in mathematics ability students who take an unproctored Web mathematics placement test will score higher on the placement test than students who take an equivalent proctored paper- and-pencil mathematics placement test. II.After controlling for differences in mathematics ability, there will be an interaction between type of placement test and the type of math course in which the student enrolled with lower first exam scores in calculus courses for students placed based on Web test results. (The testing of a 3 rd “watered-down” hypothesis was inconclusive.)

10. Results: Math placement test scores of matched samples TEST TYPEMeanStd. DeviationN Paper testers48.8911.891010 Web testers51.3412.161010 Total50.1112.082020 t (2018) = 4.591, p <.001. The correlation between math placement test score and placement test type was found to be r = 0.102. The amount of variation in total math placement test score that can be attributed to the type of test the student took was found to be 1% (r 2 =.010).

11. Results: First exam scores by course by placement test type COURSE TYPETEST TYPEMeanStd. DeviationN Non-calculus Paper testers73.4316.37234 Web testers70.1318.04182 Total71.9817.18416 Calculus coursesPaper testers67.7717.47202 Web testers61.2019.26231 Total64.2718.72433 TotalPaper testers70.8117.10436 Web testers65.1319.23413 Total68.0518.38849

12. Results: First exam scores by course by placement test type

13. Tests of Between-Subjects Effects Dependent Variable: First Math Exam Score Source SSdfMean SquareFSig. Model18408.39236136.13119.346.000 Intercept3899194.84513899194.84512293.290.000 COURSE TYPE11163.866111163.86635.197.000 TEST TYPE5118.68715118.68716.138.000 COURSE TYPE * TEST TYPE562.1251562.1251.772.183 Error268017.723845317.181 Total4217644.000849 *R Squared =.064 (Adjusted R Squared =.061)