1. Developing a Useful Instrument to Assess Student Problem Solving Jennifer L. Docktor Ken Heller Physics Education Research & Development Group http://groups.physics.umn.edu/physed DUE-0715615

2. 4/3/2009 Jennifer Docktor, University of Minnesota2 Problem Solving Measure Problem solving is an important part of learning physics. Unfortunately, there is no standard way to measure problem solving so that student progress can be assessed. The goal is to develop a robust instrument to assess students’ written solutions to physics problems, and obtain evidence for reliability, validity, and utility of scores. The instrument should be general not specific to instructor practices or techniques applicable to a range of problem topics and types

3. 4/3/2009 Jennifer Docktor, University of Minnesota3 Reliability, Validity, & Utility Reliability – score agreement Validity evidence from multiple sources Content Response processes Internal & external structure Generalizability Consequences of testing Utility - usefulness of scores AERA, APA, NCME (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Messick, S. (1995). Validity of psychological assessment. American Psychologist, 50(9), 741-749.

4. 4/3/2009 Jennifer Docktor, University of Minnesota4 Overview of Study 1. Drafting the instrument (rubric) 2. Preliminary tests with two raters (final exams and instructor solutions) 3. Training exercise with graduate students 4. Analysis of tests from an introductory mechanics course 5. Student problem-solving interviews (in progress)

5. 4/3/2009 Jennifer Docktor, University of Minnesota5 What is problem solving? Problem solving “Problem solving is the process of moving toward a goal when the path to that goal is uncertain” (Martinez, 1998, p. 605) problem What is a problem for one person might not be a problem for another person. Problem solving involves decision-making. exercise If the steps to reach a solution are immediately known, this is an exercise for the solver. Martinez, M. E. (1998). What is Problem Solving? Phi Delta Kappan, 79, 605-609. Hayes, J.R. (1989). The complete problem solver (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates. Schoenfeld, A.H. (1985). Mathematical problem solving. Orlando, FL: Academic Press, Inc.

6. 4/3/2009 Jennifer Docktor, University of Minnesota6 Problem Solving Process Understand / Describe the Problem Devise a Plan Carry Out the Plan Look Back Organize problem information Introduce symbolic notation Identify key concepts Use concepts to relate target to known information appropriate math procedures check answer Pόlya, G. (1957). How to solve it (2nd ed.). Princeton, NJ: Princeton University Press. Reif, F. & Heller, J.I. (1982). Knowledge structure and problem solving in physics. Educational Psychologist, 17(2), 102-127.

7. 4/3/2009 Jennifer Docktor, University of Minnesota7 Problem Solver Characteristics Inexperienced solvers: Knowledge disconnected Little representation (jump to equations) Inefficient approaches (formula-seeking & solution pattern matching) Early number crunching Do not evaluate solution Chi, M. T., Feltovich, P. J., & Glaser, R. (1980). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152. Larkin, J., McDermott, J., Simon, D.P., & Simon, H.A. (1980). Expert and novice performance in solving physics problems. Science, 208(4450), 1335-1342. Experienced solvers: chunks Hierarchical knowledge organization or chunks Low-detail overview / description of the problem before equations qualitative analysis qualitative analysis Principle-based approaches Solve in symbols first Monitor progress, evaluate the solution

8. 4/3/2009 Jennifer Docktor, University of Minnesota8 Instrument at a glance (Rubric) 543210NA (P) NA (S) Physics Approach Specific Application Math Procedures Logical Progression Useful Description SCORE CATEGORY: (based on literature)  Minimum number of categories that include relevant aspects of problem solving  Minimum number of scores that give enough information to improve instruction Want

9. 4/3/2009 Jennifer Docktor, University of Minnesota9 Rubric Category Descriptions Useful Description Useful Description organize information from the problem statement symbolically, visually, and/or in writing. Physics Approach Physics Approach select appropriate physics concepts and principles Specific Application of Physics Specific Application of Physics apply physics approach to the specific conditions in problem Mathematical Procedures Mathematical Procedures follow appropriate & correct math rules/procedures Logical Progression Logical Progression (overall) solution progresses logically; it is coherent, focused toward a goal, and consistent

10. 4/3/2009 Jennifer Docktor, University of Minnesota10 Rubric Scores (in general) 543210 Complete & appro- priate Minor omission or errors Parts missing and/or contain errors Most missing and/or contain errors All inappro- priate No evidence of category NA - ProblemNA - Solver Not necessary for this problem (i.e. visualization or physics principles given) Not necessary for this solver (i.e. able to solve without explicit statement) NOT APPLICABLE (NA):

11. 4/3/2009 Jennifer Docktor, University of Minnesota11

12. 4/3/2009 Jennifer Docktor, University of Minnesota12 Early Tests of the Rubric Preliminary testing (two raters) Distinguishes instructor & student solutions Score agreement between two raters – good Training Exercise (8 Graduate Students) Half scored a mechanics problem, half E&M Scored 8 student solutions with the rubric, received example scores & rationale for first 3, then re-scored 5 and scored 5 new solutions Answered survey questions about the rubric

13. 4/3/2009 Jennifer Docktor, University of Minnesota13 Grad Student Comments Influenced by traditional grading experiences Unwilling to score math & logic if physics incorrect Desire to weight categories “I don't think credit should be given for a clear, focused, consistent solution with correct math that uses a totally wrong physics approach” (GS#1) “[The student] didn't do any math that was wrong, but it seems like too many points for such simple math…I would weigh the points for math depending on how difficult it was. In this problem the math was very simple” (GS#8)

14. 4/3/2009 Jennifer Docktor, University of Minnesota14 Grad Student Comments Difficulty distinguishing categories Physics approach & application Description & logical progression “Specific application of physics was most difficult. I find this difficult to untangle from physics approach. Also, how should I score it when the approach is wrong?” (GS#1) “I think description & organization are in some respect very correlated, & could perhaps be combined” (GS#5)

15. 4/3/2009 Jennifer Docktor, University of Minnesota15 Inter-rater Agreement BEFORE TRAINING AFTER TRAINING Perfect Agreement Agreement Within One Perfect Agreement Agreement Within One Useful Description 38%75%38%80% Physics Approach 37%82%47%90% Specific Application 45%95%48%93% Math Procedures 20%63%39%76% Logical Progression 28%70%50%88% OVERALL34%77%44%85% Weighted kappa0.27±0.030.42±0.03* Fair agreement Moderate agreement *Weighted kappa difference significant at 0.01 level

16. 4/3/2009 Jennifer Docktor, University of Minnesota16 Written Training Exercise Minimal written training was insufficient Confusion about NA scores (want more examples) Score agreement improved significantly with training, but is not yet at an optimal level Difficulty distinguishing physics approach & application Math & Logical progression most affected by training Multi-part problems more difficult to score Grad students influenced by traditional grading experience

17. 4/3/2009 Jennifer Docktor, University of Minnesota17 Analysis of Tests Calculus-based introductory physics course for Science & Engineering students (mechanics) Fall >900 students split into 4 lecture sections 4 Tests during the semester 5 MC, 2 individual problems, 1 group problem Problems graded in the usual way by teaching assistants After they were graded, I used the rubric to evaluate 8 individual problems spaced throughout the semester Approximately 300 student solutions per problem (copies made by TAs from 2 sections)

18. 4/3/2009 Jennifer Docktor, University of Minnesota18 Exam 3 Question 1 Show all work! The system of three blocks shown is released from rest. The connecting strings are massless, the pulleys ideal and massless, and there is no friction between the 3 kg block and the table. (A) At the instant M 3 is moving at speed v, how far (d) has it moved from the point where it was released from rest? (answer in terms of M 1, M 2, M 3, g and v.) [10 points] (B) At the instant the 3 kg block is moving with a speed of 0.8 m/s, how far, d, has it moved from the point where it was released from rest? [5 pts] (C)…. (D)…. SYMBOLICCUES ON MASS 3 How would you solve part A?

19. 4/3/2009 Jennifer Docktor, University of Minnesota19 Grader Scores Excludes part c) multiple choice question. Average score the same (9 points or ~ half).

20. 4/3/2009 Jennifer Docktor, University of Minnesota20 Rubric Scores Useful Description: Free-body diagram (not necessary for energy approach) Physics Approach: Deciding to use Newton’s 2 nd Law or Energy Conservation Specific Application: Correctly using Newton’s 2 nd Law or Energy Cons. Math Procedures: solving for target Logical Progression: clear, focused, consistent

21. 4/3/2009 Jennifer Docktor, University of Minnesota21 Common Responses Statements in red suggest students focused on M3, which was cued in the problem statement

22. 4/3/2009 Jennifer Docktor, University of Minnesota22 Example Student Solution

23. 4/3/2009 Jennifer Docktor, University of Minnesota23 Example Student Solution Only consider kinetic energy of mass M3. ? Was cued in problem statement.

24. 4/3/2009 Jennifer Docktor, University of Minnesota24 Example Student Solution (E1=E2=E3)

25. 4/3/2009 Jennifer Docktor, University of Minnesota25 Example Student Solutions Considers forces on M3, and uses T=mg (incorrect)

26. 4/3/2009 Jennifer Docktor, University of Minnesota26 Answer is correct, but reasoning for “F” unclear Example Student Solution

27. 4/3/2009 Jennifer Docktor, University of Minnesota27 Findings The rubric indicates areas of student difficulty for a given problem i.e. the most common difficulty is specific application of physics whereas other categories are adequate Focus instruction to coach physics, math, clear and logical reasoning processes, etc. The rubric responds to different problem features For example, in this problem visualization skills were not generally measured. Modify problems to elicit / practice processes

28. 4/3/2009 Jennifer Docktor, University of Minnesota28 Problem Characteristics that could Bias Problem Solving Description: Picture given Familiarity of context Prompts symbols for quantities Prompt procedures (i.e. Draw a FBD) Physics: Prompts physics Cue focuses on a specific objects Math: Symbolic vs. numeric question Mathematics too simple (i.e. one-step problem) Excessively lengthy or detailed math

29. 4/3/2009 Jennifer Docktor, University of Minnesota29 Summary A rubric has been developed from descriptions of problem solving process, expert-novice research studies, and past studies at UMN Focus on written solutions to physics problems Training revised to improve score agreement Rubric provides useful information that can be used for research & instruction Rubric works for standard range of physics topics in an introductory mechanics course There are some problem characteristics that make score interpretation difficult Interviews will provide information about response processes

30. Additional Slides (if time permits) docktor@physics.umn.edu http://groups.physics.umn.edu/physed

31. 4/3/2009 Jennifer Docktor, University of Minnesota31 Exam 2 Question (Different) A block of known mass m and a block of unknown mass M are connected by a massless rope over a frictionless pulley, as shown. The kinetic frictional coefficient between the block m and the inclined plane is μk. The acceleration, a, of the block M points downward. (A) If the block M drops by a distance h, how much work, W, is done on the block m by the tension in the rope? Answer in terms of known quantities. [15 points] A block of mass m = 3 kg and a block of unknown mass M are connected by a massless rope over a frictionless pulley, as shown below. The kinetic frictional coefficient between the block m and the inclined plane is μ k = 0.17. The plane makes an angle 30 o with horizontal. The acceleration, a, of the block M is 1 m/s 2 downward. (A) Draw free-body diagrams for both masses. [5 points] (B) Find the tension in the rope. [5 points] (C) If the block M drops by 0.5 m, how much work, W, is done on the block m by the tension in the rope? [15 points] NUMERIC SYMBOLIC

32. 4/3/2009 Jennifer Docktor, University of Minnesota32 Grader Scores Symbolic: Fewer students could follow through to get the correct answer. Numeric, prompted: Several people received the full number of points, some about half. AVERAGE: 15 points AVERAGE: 16 points

33. 4/3/2009 Jennifer Docktor, University of Minnesota33 Rubric Scores Useful Description: Free-body diagram Physics Approach: Deciding to use Newton’s 2 nd Law Specific Application: Correctly using Newton’s 2 nd Law Math Procedures: solving for target Logical Progression: clear, focused, consistent prompted

34. 4/3/2009 Jennifer Docktor, University of Minnesota34 Solution Examples (numeric question w/FBD prompted) Could draw FBD, but didn’t seem to use it to solve the problem

35. 4/3/2009 Jennifer Docktor, University of Minnesota35 Solution Example (numeric question w/FBD prompted) NOTE: received full credit from the grader NUMBERS

36. 4/3/2009 Jennifer Docktor, University of Minnesota36 (numeric question w/FBD prompted)

37. 4/3/2009 Jennifer Docktor, University of Minnesota37 Symbolic form of question

38. 4/3/2009 Jennifer Docktor, University of Minnesota38 Symbolic form of question Left answer in terms of unknown mass “M”

39. 4/3/2009 Jennifer Docktor, University of Minnesota39 Findings about the Problem Statement Both questions exhibited similar problem solving characteristics shown by the rubric. However prompting appears to mask a student’s inclination to draw a free-body diagram the symbolic problem statement might interfere with the student’s ability to construct a logical path to a solution the numerical problem statement might interfere with the student’s ability to correctly apply Newton’s second law In addition, the numerical problem statement causes students to manipulate numbers rather than symbols

40. 4/3/2009 Jennifer Docktor, University of Minnesota40 Findings about the Rubric The rubric provides significantly more information than grading that can be used for coaching students Focus instruction on physics, math, clear and logical reasoning processes, etc. The rubric provides instructors information about how the problem statement affects students’ problem solving performance Could be used to modify problems

41. 4/3/2009 Jennifer Docktor, University of Minnesota41 References P. Heller, R. Keith, and S. Anderson, “Teaching problem solving through cooperative grouping. Part 1: Group versus individual problem solving,” Am. J. Phys., 60(7), 627-636 (1992). J.M. Blue, Sex differences in physics learning and evaluations in an introductory course. Unpublished doctoral dissertation, University of Minnesota, Twin Cities (1997). T. Foster, The development of students' problem-solving skills from instruction emphasizing qualitative problem-solving. Unpublished doctoral dissertation, University of Minnesota, Twin Cities (2000). J.H. Larkin, J. McDermott, D.P. Simon, and H.A. Simon, “Expert and novice performance in solving physics problems,” Science 208 (4450), 1335-1342. F. Reif and J.I. Heller, “Knowledge structure and problem solving in physics,” Educational Psychologist, 17(2), 102-127 (1982). http://groups.physics.umn.edu/physed docktor@physics.umn.edu

42. Additional Slides

43. 4/3/2009 Jennifer Docktor, University of Minnesota43 Independent scoring of student solutions by a PER graduate student and a high school physics teacher (N=160) Category % agree (exact) % agree (within 1) Cohen’s kappa Physics Approach71.397.10.62 Useful Description75.099.20.63 Specific Application61.396.90.48 Math Procedures65.699.40.51 Logical Progression63.196.90.49 OVERALL67.398.50.55 Inter-rater Reliability Kappa: <0 No agreement 0-0.19 Poor 0.20-0.39 Fair 0.40-0.59 Moderate 0.60-0.79 Substantial 0.80-1 Almost perfect

44. 4/3/2009 Jennifer Docktor, University of Minnesota44 Inter-rater Agreement BEFORE TRAINING AFTER TRAINING Perfect Agreement Agreement Within One Perfect Agreement Agreement Within One Useful Description 38%75%38%80% Physics Approach 37%82%47%90% Specific Application 45%95%48%93% Math Procedures 20%63%39%76% Logical Progression 28%70%50%88% OVERALL 34%77%44%85% Weighted kappa0.27±0.030.42±0.03* Fair agreement Moderate agreement *Weighted kappa difference significant at 0.01 level

45. 4/3/2009 Jennifer Docktor, University of Minnesota45 All Training in Writing: Example CATEGORY SCORE RATIONALE Training includes the actual student solution

46. 4/3/2009 Jennifer Docktor, University of Minnesota46 Ratings Before & After Training Math & Logic most affected by training

47. 4/3/2009 Jennifer Docktor, University of Minnesota47 Kappa fo: observed frequencies of exact agreement (diagonal of pivot table) fe: expected frequencies of exact agreement by chance N: number of subjects rated 95% confidence limit: 1.960 99% confidence limit: 2.576 99.9% confidence limit: 3.291

48. 4/3/2009 Jennifer Docktor, University of Minnesota48 Weighted Kappa 95% confidence limit: 1.960 99% confidence limit: 2.576 99.9% confidence limit: 3.291 fo: observed frequencies of exact agreement (diagonal of pivot table) fe: expected frequencies of exact agreement by chance N: number of subjects rated

49. 4/3/2009 Jennifer Docktor, University of Minnesota49 Exam 1 Question 1 A block of mass m=2.5 kg starts from rest and slides down a frictionless ramp that makes an angle of θ=25 o with respect to the horizontal floor. The block slides a distance d down the ramp to reach the bottom. At the bottom of the ramp, the speed of the block is measured to be v=12 m/s. a)Draw a diagram, labeling θ and d. [5 points] b) What is the acceleration of the block, in terms of g? [5 points] c) What is the distance, d, in meters? [15 points] INSTRUCTOR SOLUTION

50. 4/3/2009 Jennifer Docktor, University of Minnesota50 Grader Scores >40% of students received the full points on this question Was this an exercise or a problem?

51. 4/3/2009 Jennifer Docktor, University of Minnesota51 Rubric Scores Scores shifted to high end (5’s) or NA

52. 4/3/2009 Jennifer Docktor, University of Minnesota52 Problem Solving Process 1. Identify & define the problem 2. Analyze the situation 3. Generate possible solutions/approaches 4. Select approach & devise a plan 5. Carry out the plan 6. Evaluate the solution http://www.hc-sc.gc.ca/fniah-spnia/images/fnihb- dgspni/pubs/services/toolbox-outils/78-eng.gif 123 456

53. 4/3/2009 Jennifer Docktor, University of Minnesota53 Developing & Testing the Rubric Spring 2007 Summer 2007 Fall 2007 Spring 2008 Spring 2009 1. Draft instrument based on literature & archived exam data 3. Pilot with graduate students (brief training) 2. Test with two raters (consistency of scores) 5. Revise rubric and training materials. Retest. 8. Final data analysis & reporting 6. Collect & score exam problems from fall semester of 1301 course. Fall 2008 Summer 2008 4. Analyze pilot data (feedback & scores) 7. (Interviews) Video & audio recordings of students solving problems. Summer 2009