1. University of Georgia – Chemistry Department JExam - A Method to Measure Outcomes Assessment Charles H. Atwood, Kimberly D. Schurmeier, and Carrie G. Shepler Chemistry Department University of Georgia Athens, GA 30602 Presented to Academic Affairs Faculty Symposium Unicoi State Park, Helen GA March 31, 2007

2. University of Georgia – Chemistry Department Outline of Presentation JExam – Computerized Testing Program Item Response Theory (IRT) Using IRT to measure testing effectiveness Outcomes Assessment Conclusion

3. University of Georgia – Chemistry Department Item Response Theory (IRT) –More modern psychometric analysis tool than CTT –Works well for large sample sizes (> 200) –Iterative process which fits the students responses to a mathematical formula

4. University of Georgia – Chemistry Department Item Response Theory (IRT) We use the program Bilog-MG3 to fit our data. Bilog-MG3 assigns to every test item values for a, b, and c and displays them on an item characteristic curve –Plot of the probability of a student with a given IRT ability vs. probability that the student answered the question

5. University of Georgia – Chemistry Department Item Response Theory (IRT)

6. University of Georgia – Chemistry Department Item Response Theory (IRT) Less than 5 minutes later an item response list is generated. –Our raw data 00000000 991099919919919919919909919909919 00000001 919919199199199199199199199199199 00000002 919919199199199199199199199199199 00000003 919909199199199199199199199199199 00000004 919919199099099099199199199199099 00000005 919919199199099099199199199199199 00000006 919919199199199199199099199199199 00000007 919909199199099199199199199199099 00000008 919919199199099199199199199099199 00000009 919919199199099099199099099199199 00000010 919919199199199099199199199199199 00000011 919919199199199099199199199199099 00000012 919919199199099199199199199199199 00000013 919919199199199199199199199099199 00000014 919919199199199199199199199199199 00000015 919909199199199199199099199099099 00000016 990099909919909909919909919909919 00000017 990099919919909909919919919919919 00000018 991099919909909909909909909909909 00000019 991099919919909919909919919919909 00000020 991099919919909909919909919919919 00000021 991099919919919919919919919919919 00000022 991199909919919919919919919919919 00000023 990099909909919909909909909909909 00000024 991199919909909919919919919909919

7. University of Georgia – Chemistry Department Item Response Theory (IRT) We enter the raw data into Bilog-MG 3.0 along with a few parameters. Less then 5 minutes later we have our Item Characteristic Curves for every question on the test.

8. University of Georgia – Chemistry Department Item Response Theory (IRT) Item Information Curve for every test question

9. University of Georgia – Chemistry Department Item Response Theory (IRT) C-D discriminator example:

10. University of Georgia – Chemistry Department Item Response Theory (IRT) B-C discriminator example:

11. University of Georgia – Chemistry Department Item Response Theory (IRT) A-B discriminator example:

12. University of Georgia – Chemistry Department Using IRT to measure testing effectiveness Can tell us whether or not our tests are effectively assessing students at every ability level. Test information curve –Prior to IRT

13. University of Georgia – Chemistry Department Using IRT to measure testing effectiveness Test information curve –After IRT

14. University of Georgia – Chemistry Department Using IRT to measure testing effectiveness Generate a plot of student ability versus test score for every test during the semester. Plot for 1 st test of the second semester of General Chemistry.

15. University of Georgia – Chemistry Department Using IRT to measure testing effectiveness When student ability vs. test scores is averaged over the last 6 years –Grading scale that appears to be consistent for the entire year and independent of student population –Possible absolute grading scale

16. University of Georgia – Chemistry Department Using IRT to measure testing effectiveness If we generate a plot of the number of students that have given ability on a test: –Get the regular bell-shaped curve –Plot for Spring semester 2006, 1 st test

17. University of Georgia – Chemistry Department Using IRT to measure testing effectiveness Compare that to this plot: –Spring Semester 2006, 1 st test

18. University of Georgia – Chemistry Department Outcomes Assessment After Spring Semester 2006, we went and looked at the test questions which had the highest discrimination and ability levels. From this analysis we discovered that there are several topics which cause the students the most difficulty for the entire year.

19. University of Georgia – Chemistry Department Outcomes Assessment 1.Understanding the structure of ionic compounds 2.Unit conversion problems Particularly converting from volume to area or height 3.Molecular polarity 4.Intermolecular forces 5.Understanding quantum numbers 6.Distinguishing the terms strong, weak, concentrated and dilute 7.Inorganic nomenclature

20. University of Georgia – Chemistry Department Outcomes Assessment Understanding the structure of ionic compounds is crucial to student performance in 1211/1212 This concept is essentially a gatekeeper.

21. University of Georgia – Chemistry Department Outcomes Assessment Ability at 0.336 Discriminates between C and D students Fall 2005 Test 1 Question: “What is the correct name of this ionic compound?” Al(NO 3 ) 3 aluminum nitrate “How many ions are present in one formula unit of the compound shown above?” 4

22. University of Georgia – Chemistry Department Outcomes Assessment Since understanding ions is so difficult and so crucial to success, we decided to tackle it first. Prior to start of Fall semester 2006 –Teaching faculty meeting –Discussed difficult topics –Stressed that we must emphasize ions in lecture

23. University of Georgia – Chemistry Department Outcomes Assessment Fall 2006 Test 1 Question: “What is the correct name of this ionic compound?” Al(NO 3 ) 3 aluminum nitrate “How many ions are present in one formula unit of the compound shown above?” 4

24. University of Georgia – Chemistry Department Outcomes Assessment Look at the Gaussian Fit to Ability Scores for several years –Plot for Fall Semester 2004, 1 st test

25. University of Georgia – Chemistry Department Outcomes Assessment Compare that to this plot: –Fall Semester 2005, 1 st test

26. University of Georgia – Chemistry Department Outcomes Assessment Compare that to this plot: –Fall Semester 2006, 1 st test

27. University of Georgia – Chemistry Department Outcomes Assessment Second Test Comparisons –Fall Semester 2004, 2 nd test

28. University of Georgia – Chemistry Department Outcomes Assessment Second Test Comparisons –Fall Semester 2005, 2 nd test

29. University of Georgia – Chemistry Department Outcomes Assessment Second test comparisons –Fall Semester 2006, 2 nd test

30. University of Georgia – Chemistry Department Outcomes Assessment Third test comparisons –Fall Semester 2004, 3 rd test

31. University of Georgia – Chemistry Department Outcomes Assessment Third test comparisons –Fall Semester 2005, 3 rd test

32. University of Georgia – Chemistry Department Outcomes Assessment Third test comparisons –Fall Semester 2006, 3 rd test

33. University of Georgia – Chemistry Department Outcomes Assessment Final Exam comparisons –Fall Semester 2004, Final Exam

34. University of Georgia – Chemistry Department Outcomes Assessment Final Exam comparisons –Fall Semester 2005, Final Exam

35. University of Georgia – Chemistry Department Outcomes Assessment Final Exam comparisons –Fall Semester 2006, Final Exam

36. University of Georgia – Chemistry Department Conclusion Item Response Theory is a more effective method of assessing student ability. We can design and implement test questions that discriminate student abilities at all levels. Appropriate interventions can measurably improve overall student abilities.

37. University of Georgia – Chemistry Department Acknowledgements Bob Scott and John Stickney –Department Chairs Gary Lautenschlager –Professor of Psychology and IRT expert UGA’s PRISM project. –This material is based on work supported by the National Science Foundation under Grant No. EHR-0314953.