1. Zdeslav Hrepic Dean A. Zollman N. Sanjay Rebello Supported by NSF ROLE Grant # REC-0087788 Fort Hays State University Kansas State University AAPT, Sacramento

2. Outline 1.Rationale: Why use in-class, real-time assessment? 2.Previous research:  Mental models of sound propagation.  Hybrid mental models and their role. 3.Test construction and validation 4.Results 5.Using the test 6.Further study

3. Goal of the study To create a multiple choice test… …that can elicit students’ mental models of sound propagation… …during the lecture… …using a class response system and appropriate software.

4. Real time, in class assessment Uses some form of Class Response System Enables quick collection and immediate analysis of students responses in the classroom.

5. Benefits of class assessment  Engages students.  Facilitates interactive learning and peer instruction (especially in large enrolment classes).  Gives immediate feedback to the teacher.  Enables the teacher to adjust the teaching before the exam rather than after it and according to specific needs of his/her students.  Allows a post lecture detailed analysis.

6. Research questions  Main question: What is the optimal multiple choice test that can elicit students’ mental models of sound propagation in a real time, during the instruction?  Sub questions (Addressed in the presentation): What is the optimal analytical tool for analysis of students’ responses in this test? How do we represent data so the display provides a variety of instruction guiding information?

7. Starting point in test creation: Identifying mental models of sound propagation  Wave Model - Scientifically accepted model  Independent Entity Model - Dominant alternative model: Sound is a self-standing, independent entity different from the medium through which it propagates.  Hybrid models - Composed of entity and wave model features and at the same time they are incompatible with both the entity and the wave models. (E.g.) Hrepic, Z., Zollman, D., & Rebello, S. (2002). Identifying students' models of sound propagation. Paper presented at the 2002 Physics Education Research Conference, Boise ID.

8. Mental models of Earth Mixed Model State Hybrid Models Target model Initial model

9. Metaphor Mental Models and Model states A mule = hybrid of a donkey + a horse. A horse – 64 chromosomes A donkey – 62 chromosomes A mule – 63 chromosomes http://www.luckythreeranch.com/muletrainer/mulefact.asp Donkey Horse Hybrid = Mule

10. Model States (In terms of children’s mental models of Earth; Vosniadou, 1994) Instance 1 Instance 2 Mixed Model State Hybrid Model State Pure Model 2 State Pure Model 1 State

11. Hybrid mental models identified in domains of…  Earth science (Vosniadou, 1994)  Electrostatics (Otero, 2001)  Newtonian mechanics (Hrepic, 2002; Itza-Ortiz, Rebello, & Zollman, 2004)  Sound (Hrepic, 2002; Hrepic, Zollman, & Rebello, 2002).  Optics (Galili, Bendall, & Goldberg, 1993) (“hybridized knowledge”)  Inertia and gravity (Brown & Clement, 1992) (“intermediate concepts”)

12. Implications of hybrid mental models Implications for teaching  A student can give a variety of correct answers on standard questions using a hybrid (wrong) model. Implications for analysis of our test  Hybrid models cause overlaps in multiple choice questionnaires – more than one model corresponds to the same choice.  Complexity: 3 questions define a model  Model analysis requires that each answer choice is uniquely associated with a model.

13. No Model State Model States Mixed Model State x x x x x x x Knowledge elements related to Model 1 only Knowledge elements related to both models or neither one Hybrid Model State Instance 1 Instance 2 Knowledge elements related to Model 2 only x x x x Pure Model 2 State x x x x x x x x Pure Model 1 State x xx x x

14. 4 basic models - mechanisms of propagation Human characters = Air particles Footballs = Sound entities

15. Wave Model Scientifically Accepted Model Independent Entity Dominant Alternative Model 4 basic models - mechanisms of propagation (+) Ear Born Sound Propagating Air Dependent Entity Hybrid Models

16. Pilot testing  Did we miss anything in terms of mental models? Open-ended questionnaire on a large sample  Did we miss anything in terms of productive questions to determine students mental models? Battery of semi-structured conceptual questions related to sound as a wave phenomena in variety of situations

17. Test Contexts 1. Air How does sound propagate in this situation?

18. Test Contexts 2. Wall How does sound propagate in this situation?

19. Test Contexts 1a, 2a - Vacuum What happens without the medium (air or wall)?

20. Test questions - paraphrased 1.What is the mechanism of sound propagation in the air/wall? 2.How do particles of the medium vibrate, if at all, while the sound propagates? 3.How do particles of the medium travel, if at all, while the sound propagates? 4.What does this motion have to do with sound propagation – cause and effect relationship? 5.What does this motion have to do with sound propagation – time relationship? 6.What happens with sound propagation in the vacuum?

21. Displaying the test results  Several representations of students’ state of understanding  Available in real time and in post instruction analysis  Consistency: Consistent – a student uses one model (Pure model state) Inconsistent – a student uses more than one model (Mixed model state)

22. Using a particular model Pre Instruction; Calculus based; University; NY N = 100 Inconsistently Consistently

23. Using a particular model at least once Pre Instruction; Calculus based; University; NY N = 100 Inconsistently Consistently

24. Movements of particles of the medium Pre Instruction; Calculus based; University; NY N = 100 (+) Random Travel (+) Travel Away From The source Vibration on the Spot

25. Model states Pre Instruction; Calculus based; University; NY N = 100 Pure Other Pure Wave Mixed Any Mixed Entity Mixed Ear-Wave

26. Correctness Pre Instruction; Calculus based; University; NY N = 100

27. Using a particular model Pre Instruction; Calculus based; University; NY N = 100 Inconsistently Consistently

28. Using a particular model Post Instruction; Calculus based; University; NY N = 95 Inconsistently Consistently

29. Movements of particles of the medium Pre Instruction; Calculus based; University; NY N = 100 (+) Random Travel (+) Travel Away From The source Vibration on the Spot

30. Movements of particles of the medium Post Instruction; Calculus based; University; NY N = 95 (+) Random Travel (+) Travel Away From The source Vibration on the Spot

31. Correctness Pre Instruction; Calculus based; University; NY N = 100

32. Correctness Post Instruction; Calculus based; University; NY N = 95

33. Test validity Built and shown through:  Interviews with students  Expert reviews  Role playing validation with experts  Validity strengthening test development procedures  Tables of content and construct specifications  Meaningful correlations between all answer choices  Instructional sensitivity of the test  Stability (reliability) of results obtained in the large scale survey… across different educational levels across different institutions at equivalent educational levels across different course levels at same institutions

34. Constructing the test Four steps of test construction and validation: 1.Pilot testing large open ended survey – settling on models, choosing contexts 2.Pre-survey testing expert validation, 7 choice survey (with none of the above, more than one of the above), correlation analysis of answer choices, refinement through interviews 3.Survey testing large scale survey –correlation analysis, comparisons between levels, pre-post results; interview validation 4.Post Survey testing moderately large scale survey, role playing, expert validation

35. Survey participants

36. Survey phase - Validity interviews  17 x 4 probes in the interviewed sample.  The invalid display of a model would have occurred in 6 instances (out of 68).  8.8% of the probes  3 instances because of 5a (+ another 3 that did not cause invalid probe)

37. Correlation analysis of answer choices

38. Post-Survey Testing Expert review:  To validate post survey version Few minor items improved  Surveying:  To determine correlations between response items and see if changes made the desired effect. Problems fixed  Role playing validation:  To validate new test version in an additional way Perfect score

39. Comparing model distribution Different educational levels

40. Comparing model distribution Grouped models; Different Educational Levels

41. Comparing model distribution Different course levels

42. Comparing differences in model distribution Variability within different educational levels

43. Pre-Post instruction difference *Gain (G) = (post-test) – (pre-test) **Normalized gain (h) = gain / (maximum possible gain) (Hake, 1997).

44. Test package Prospective uses of test, test questions Online package related to test and analysis of data available at: http://web.phys.ksu.edu/role/sound/ http://web.phys.ksu.edu/role/sound/ Formative assessment combined with any instructional method/approach  “traditional”  “progressive”  “misconception oriented” Model – cause Misconception – symptom As peer instruction questions (not model defining) Not recommended as a summative assessment

45. Limitations  Common to multiple choice tests Answer options do affect students understanding / models Test taking strategies may obscure results Test projects no model state as mixed model state and possibly pure model state.

46. Future research Unique approach - Wide themes opened  Applicability of the approach in other domains of physics: Is the approach “hybrid model-(in)dependent”? Applicability in domains of other natural sciences?  How effectively teachers can implement the real-time aspect of this testing approach?  Instructional utility of this type of testing: Will addressing of the underlying models in real time help students learn?  Possibility of individualized addressing of student’s models in real time?  Applicability of the testing approach in eliciting non-cognitive psychological constructs: Personality tests: Would it provide information that current tests in that field do not? Reduction of items when compared to Likert scale

47. Future research Specific issues opened  Optimal using of the test in combination with online homework Saving of time Any classroom benefit counterbalance?  How applicable is this test at the middle school level?  How would a branched version of the test look, and would it have any advantages with respect to this one?  Improved simplicity and validity of the test

48. More Information / Feedback zhrepic@fhsu.edu www.fhsu.edu/~zhrepic (www.hrepic.com)

49. Literature  Brown, D., & Clement, J. (1992). Clasroom teaching experiments in mechanics. In R. Duit, Goldberg, F., Niedderer, H. (Ed.), Research in physics learning: Theoretical issues and empirical studies (pp. 380-389). Kiel: IPN.  Galili, I., Bendall, S., & Goldberg, F. M. (1993). The effects of prior knowledge and instruction on understanding image formation. Journal of Research in Science Teaching, 30(3), 271-301.  Hrepic, Z. (2002). Identifying students' mental models of sound propagation. Unpublished Master's thesis, Kansas State University, Manhattan.  Hrepic, Z., Zollman, D., & Rebello, S. (2002). Identifying students' models of sound propagation. Paper presented at the 2002 Physics Education Research Conference, Boise ID.  Itza-Ortiz, S. F., Rebello, S., & Zollman, D. A. (2004). Students’ models of Newton’s second law in mechanics and electromagnetism. European Journal of Physics, 25, 81–89.  Otero, V. K. (2001). The process of learning about static electricity and the role of the computer simulator. Unpublished Ph.D. Dissertation, University of California, San Diego, CA.  Vosniadou, S. (1994). Capturing and modeling the process of conceptual change. Learning & Instruction, 4, 45-69.  Greca, I. M., & Moreira, M. A. (2002). Mental, physical, and mathematical models in the teaching and learning of physics. Science Education, 86(1), 106-121.  diSessa, A. A. (2002). Why "conceptual ecology" is a good idea. In M. Limon & L. Mason (Eds.), Reconsidering conceptual change: Issues in theory and practice (pp. 29-60). Dordrecht, Netherlands: Kluwer Academic Publishers.  Hrepic, Z., Zollman, D., & Rebello, S. (2002). Identifying students' models of sound propagation. Paper presented at the 2002 Physics Education Research Conference, Boise ID.  Vosniadou, S. (1994). Capturing and modeling the process of conceptual change. Learning & Instruction, 4, 45-69.  Physics Education Group at Arizona State University. (2000). Modeling Instruction Program [www]. Arizona State University. Retrieved 24. Aug. 2003, 2003, from the World Wide Web: http://modeling.la.asu.edu/http://modeling.la.asu.edu/  Clement, J. M. (2003). Re: testing to discriminate between students vs other approaches: PhysLrnR post of 18 Apr 2003 10:35:46 -0500; online at.  Hanna, G. S. (1993). Better teaching through better measurement. Orlando, Florida: Harcourt Brace Jovanovic, Inc.  Oosterhof, A. (2001). Classroom applications of educational measurement. Upper Saddle River, New Jersey: Prentice Hall, Inc.