Computation-based versus concept-based test questions: High school teachers’ perceptions Computation-based versus concept-based test questions: High school teachers’ perceptions Luanna Gomez and Daniel MacIsaac Physics Joseph Zawicki Earth Sciences & Science Education SUNY College at Buffalo Buffalo, NY

Abstract The New York State Regents examination in physics is a standardized assessment of high school students' competencies after completing a year of introductory physics. The analysis of select items taken from 1500 students will be provided. We have compared traditional computational problems to less traditional conceptual problems to examine the extent to which the response pattern provides insight to the difficulty of the two types of questions. This discussion will form a context in which teachers' perceptions of the nature of numeric and concept questions, of their relation to physics understanding, and of their implications to physics instruction. 2

False dichotomies We examined select items from recent New York State Regents examinations, including response analysis from about 4500 high school student papers. Student response data revealed the conceptual items were amongst the most difficult, which was surprising to several vociferous high school physics teachers interviewed. 3

Student population and context Urban, suburban, and rural school districts NYS Regents exam in physics The secondary physics exam is administered in June to Grade 11 & 12 (N = students) 4

Regents’ examination background The exam ‡ : is aligned with New York State core curriculum ¶ ; has been administered since 2007; and contains 3 sections (~ 60 items): multiple choice, constructed response (e.g. requires a short answer or calculation), and extended constructed (e.g. requires a written passage or multi-step calculation) ‡ See, for example, ¶ The NYS core curriculum may be viewed at 5

June 2008 secondary-level student data (Example item 1) Item 1 Major learning goal: To be able to discern that measured quantities can be classified as either a vector or a scalar. Discrimination index: D = 0.80 Concept multiple-choice question The speedometer in a car does not measure the car’s velocity because velocity is a (1) vector quantity and has magnitude and direction (correct). (2) vector quantity and does not have a direction associated with it. (3) scalar quantity and has a direction associated with it. (4) scalar quantity and does not have and does not have a direction associated with it. 6

June 2008 secondary-level student data (Example item 2) Item 2 Major learning goals: To recognize that an object or system has a kinetic energy associated with its velocity, and that its gravitational potential energy depends solely on the relative positions of the objects in that system. Discrimination index: D = 0.48 Concept multiple-choice question A car travels from point A to point B at constant speed up a hill. As the car travels its gravitation potential energy (1) increases and its kinetic energy decreases. (2) increases and its kinetic energy remains the same (correct). (3) remains the same and its kinetic energy decreases. (4) remains the same and its kinetic energy remains the same. 7

June 2008 secondary-level student data (Example item 3) Item 3 Major learning goals: To recognize that an object or system has a kinetic energy associated with its velocity, and that its gravitational potential energy depends solely on the relative positions of the objects in that system. Discrimination index: D = 0.75 Short calculation item A 65kg pole vaulter wishes to vault at a height of 5.5m. Calculate the minimum amount of kinetic energy the vaulter needs to reach the height if air friction is neglected and all the vaulting energy is derived from kinetic energy. [Show all work, including the equation and substitution with units.] Correct answer: KE = 3500 J (rounded to 2 significant figures) 8

June 2008 secondary-level student data (Example items 4-6) Major learning goal: Interpretation of graphs Item 4 Discrimination index: D = 0.97 Item 5 Discrimination index: D = 0.93 Item 6 Discrimination index: D = 0.45 Multi-step calculation item Item 4: Plot the data points for the dart’s maximum vertical displacement versus spring compression. [Use the information in the data table (not shown).] Item 5: Draw the line or curve of best fit. Item 6: Using information from your graph, calculate the energy provided by the compressed spring that causes the dart to achieve a maximum vertical displacement of 3.50m. [Show all your work, including equation and substitution with units.] (Solution not shown) 9

Teacher data collection (preliminary) Individual teacher interviews (N ~ 3) Teachers were asked whether they believed computation-based (i.e., formula-driven) problems were more or less challenging for students than concept-based (i.e., qualitative) problems on the Regents’ exam in physics/physical science—they believed that the calculation problems were more difficult than conceptual ones! 10

Tentative conclusion Conceptual items may be more difficult than most calculation-driven ones Teachers may not appreciate the value of concept questions and dismiss the results because they are perceived as “trick” questions. On the basis of these results, teachers may be spending more time preparing students for the Regents exam in physics with equations rather than concepts. 11

Implications Some skill sets, such as inscription, remain more difficult for students. Mundane skill sets, such as plotting points and solving for commonly rehearsed variables, appear to be readily achieved 12

Acknowledgements SUNY Buffalo State College Kathleen Falconer, Elementary Education and Reading Western New York Regional Information Center Timothy Johnson, Erie 1 BOCES 13