Assessing the Assessments: Physics An Initial Analysis of the June 2003 NYS Regents Physics Exam J. Zawicki, SUNY Buffalo State College M. Jabot, SUNY Fredonia

J. Zawicki, M. Jabot2 July 2004 Assessment Purposes Measure knowledge Measure gain in knowledge Measure preparation (predict success) Sorting (Grading) Degree requirements (benchmarks) …

J. Zawicki, M. Jabot3 July 2004 Curriculum, Assessment and Instruction Frameworks Syllabi Guides Blueprints Benchmarks Objective tests Performance assessments Portfolios Teacher Observations Group Activities Program Evaluations Curriculum Standards Assessment/Evaluation SystemInstructional Program alignment validity correlation Instructional styles Print materials Equipment Facilities Technology Community

J. Zawicki, M. Jabot4 July 2004 Concepts (Continued) Difficulty – (Percentage or proportion that are successful on an item)  Facility  Difficulty Discrimination – (How well does the item differentiate between students who understand the subject and those who do not?)

J. Zawicki, M. Jabot5 July 2004 Concepts Validity – how well the item measures match the target construct. May be qualified as:  Construct  Content (Face)  Criterion Related Typically determined by a panel of experts

J. Zawicki, M. Jabot6 July 2004 Concepts (Continued) Reliability – can the results be replicated?  Inter-rater (Do two or more raters agree on the score for an item?)  Test/Re-test (Will a student earn similar scores on different administrations?)  Internal Consistency Criterion referenced tests – have the students met the “standard”

J. Zawicki, M. Jabot7 July 2004 Concepts (Continued) Latency – (How long do students take to complete the test?) Equitable (Fair) Timed tests (Power tests)

J. Zawicki, M. Jabot8 July 2004 Types of Analysis Traditional (difficulty, discrimination) Rasch Analysis (item difficulty is equated to student ability) Cognitive Level (Bloom’s taxonomy simplified: knowing, using, integrating)

J. Zawicki, M. Jabot9 July 2004 Initial Analysis Physics 2004FormatCreditKeyR/C 0R/C 1R/C 2R/C 3R/C 4R/C (%)FacilityDifficultyDEP Item # 01MC Item # 39MC Item # 46MC Item # 20MC Item # 60CR Item # 63CR Item # 68CR Item # 55CR

J. Zawicki, M. Jabot10 July 2004 Difficulty Rankings Easier MC: 1, 39 More Difficult MC: 46, 20 Easier CR: 60, 63 More Difficult CR 68, 55

J. Zawicki, M. Jabot11 July 2004 Easier Multiple Choice

J. Zawicki, M. Jabot12 July 2004 Easier Multiple Choice

J. Zawicki, M. Jabot13 July 2004 More Difficult Multiple Choice

J. Zawicki, M. Jabot14 July 2004 More Difficult Multiple Choice

J. Zawicki, M. Jabot15 July 2004 Easier Constructed Response

J. Zawicki, M. Jabot16 July 2004 Easier Constructed Response

J. Zawicki, M. Jabot17 July 2004 More Difficult Constructed Response

J. Zawicki, M. Jabot18 July 2004 More Difficult Constructed Response

J. Zawicki, M. Jabot19 July 2004 Common Threads – Easier Items Graphing Using charts, tables, graphs Classification (vectors, scalars, …)

J. Zawicki, M. Jabot20 July 2004 Common Threads – More Difficult Items Mathematical relationships (Modeling)  Direct  Inverse  Linear  Exponential Electric, Magnetic, and Gravitational Interactions; Fields (CSEM, Modeling, Castle, Knight) Energy vs. Force (FCI, PET, Modeling)

J. Zawicki, M. Jabot21 July 2004 Next Steps Questions Implications for classrooms (program review) Additional Resources  BSC: SCI685, Evaluation in Science Education  SUNY Fredonia: EDU503, Evaluation in the Schools  SUNY Buffalo: LAI534, Measurement & Evaluation Of Science Instruction  STANYS 2004 Annual Conference, November 7-9, 2004  Rochester Science Educator’s Conference  NYSSELA Perspectives