1. Assessing Student Understanding of Physical Hydrology (#0691) Adam J. Castillo a,c ; Jill Marshall a ; Meinhard B. Cardenas b a Department of Curriculum and Instruction, The University of Texas at Austin, 1912 Speedway Stop D5700, Austin, Texas, 78712, USA b Department of Geological Sciences, The University of Texas at Austin, 2275 Speedway Stop C9000, Austin, Texas, 78712, USA c corresponding author: adamj.castillo@austin.utexas.eduadamj.castillo@austin.utexas.edu Assessing Student Understanding of Physical Hydrology (#0691) Adam J. Castillo a,c ; Jill Marshall a ; Meinhard B. Cardenas b a Department of Curriculum and Instruction, The University of Texas at Austin, 1912 Speedway Stop D5700, Austin, Texas, 78712, USA b Department of Geological Sciences, The University of Texas at Austin, 2275 Speedway Stop C9000, Austin, Texas, 78712, USA c corresponding author: adamj.castillo@austin.utexas.eduadamj.castillo@austin.utexas.edu Objectives Contribute to the effort to articulate a common knowledge base Identify effective instructional strategies Identify desired learning outcomes of a physical hydrology course Evaluate curriculum reform in physical hydrology Determine a means of assessing whether those learning goals have been met – prototype an instrument Identify “misconceptions, preconceptions, and difficulties” prior to the curriculum development effort 1 Objectives Contribute to the effort to articulate a common knowledge base Identify effective instructional strategies Identify desired learning outcomes of a physical hydrology course Evaluate curriculum reform in physical hydrology Determine a means of assessing whether those learning goals have been met – prototype an instrument Identify “misconceptions, preconceptions, and difficulties” prior to the curriculum development effort 1 Methodology Setting & Participants A physical hydrology course at a large research university Two sections: one upper-division undergraduate & one graduate section No lab or field component Course components include: in-class lectures, homework sets, exams, course project Informed Consent (15 undergraduates & 10 graduates consented; participants equally split between M/F) Learning Goals for Physical Hydrology Course 1.Quantitative process-based understanding of hydrologic processes 2.Experience with different methods in hydrology 3.Learning, problem-solving, communication skills Goals Translated into Questions for the Assessment Tool Q1. What are the important physical processes involved in hydrology? Describe how they affect hydrologic systems in as much detail as you can. Q2. What are the relevant physical laws that govern hydrology and how do these laws determine hydrological processes? Describe them in as much detail as you can. Q3. You have been hired as a consultant by __ to (1) assess how urbanization and the current drought have affected a local spring and (2) predict what the effects will be in the future if the drought continues. What information would you need to gather? What measurements would you make? What analyses would you perform? Be as specific as you can. Methodology Setting & Participants A physical hydrology course at a large research university Two sections: one upper-division undergraduate & one graduate section No lab or field component Course components include: in-class lectures, homework sets, exams, course project Informed Consent (15 undergraduates & 10 graduates consented; participants equally split between M/F) Learning Goals for Physical Hydrology Course 1.Quantitative process-based understanding of hydrologic processes 2.Experience with different methods in hydrology 3.Learning, problem-solving, communication skills Goals Translated into Questions for the Assessment Tool Q1. What are the important physical processes involved in hydrology? Describe how they affect hydrologic systems in as much detail as you can. Q2. What are the relevant physical laws that govern hydrology and how do these laws determine hydrological processes? Describe them in as much detail as you can. Q3. You have been hired as a consultant by __ to (1) assess how urbanization and the current drought have affected a local spring and (2) predict what the effects will be in the future if the drought continues. What information would you need to gather? What measurements would you make? What analyses would you perform? Be as specific as you can. Acknowledgements This research was supported by a U.S. National Science Foundation CAREER Grant (EAR-0955750). We thank the graduate students and researchers who applied the rubric, namely Kevin Befus, Kuldeep Chaudhary, Wen Deng, Alec Norman, Lichun Wang, and Peter Zamora, and the students in the classes where the assessment tool was applied. Acknowledgements This research was supported by a U.S. National Science Foundation CAREER Grant (EAR-0955750). We thank the graduate students and researchers who applied the rubric, namely Kevin Befus, Kuldeep Chaudhary, Wen Deng, Alec Norman, Lichun Wang, and Peter Zamora, and the students in the classes where the assessment tool was applied. Discussion Question 1 Most students entered course with rudimentary understanding of processes involved in hydrology Understanding of the water cycle, especially its physical drivers, proved challenging Difficulties included incorporating groundwater, surface-groundwater interactions, perceptions of the hydrological system as static, and a non- systems approach Students often either neglected or exaggerated biological & human interactions Question 2 Most students entered course with rudimentary understanding of laws that govern hydrology Question 3 Understanding was somewhat higher in the ‘methods’ dimension Still not approaching the ‘full’ level of understanding Post-test Results Student understanding increased as assessed by the drafted rubric 5 Majority of students fell into the ‘basic’ or ‘full’ understanding categories Limitations Small sample – participants were all from one institution Rubric was being piloted and still under development Students are not able to show full range of what they know & are able to do in a limited time and on only one type of assessment 4 Final Comments Had substantial inter-rater agreement among a group of experts from the discipline of hydrology Experts not responsible for the development of the rubric Instrument should serve as a ‘strawman’ proposal to be critiqued Results give indication of ways in which instruction might be redesigned to target specific difficulties & highlight important themes in coherent manner A well-designed pre and post assessment can be used to conclude whether a given instructional intervention has caused a change in understanding in a given group of students Results are not necessarily generalizable Discussion Question 1 Most students entered course with rudimentary understanding of processes involved in hydrology Understanding of the water cycle, especially its physical drivers, proved challenging Difficulties included incorporating groundwater, surface-groundwater interactions, perceptions of the hydrological system as static, and a non- systems approach Students often either neglected or exaggerated biological & human interactions Question 2 Most students entered course with rudimentary understanding of laws that govern hydrology Question 3 Understanding was somewhat higher in the ‘methods’ dimension Still not approaching the ‘full’ level of understanding Post-test Results Student understanding increased as assessed by the drafted rubric 5 Majority of students fell into the ‘basic’ or ‘full’ understanding categories Limitations Small sample – participants were all from one institution Rubric was being piloted and still under development Students are not able to show full range of what they know & are able to do in a limited time and on only one type of assessment 4 Final Comments Had substantial inter-rater agreement among a group of experts from the discipline of hydrology Experts not responsible for the development of the rubric Instrument should serve as a ‘strawman’ proposal to be critiqued Results give indication of ways in which instruction might be redesigned to target specific difficulties & highlight important themes in coherent manner A well-designed pre and post assessment can be used to conclude whether a given instructional intervention has caused a change in understanding in a given group of students Results are not necessarily generalizable Results Background Hydrology has become increasingly interdisciplinary & technologically complex Limited number of studies of student understanding of hydrology & how it evolves over the course of schooling 5 Calls for examining, evaluating & enhancing hydrology education at both the upper-division and graduate level 2,6,7 Early effort – extensive survey of topics covered in college hydrology courses provided as resource to the community 3 Recent calls to “analyze, synthesize, and unite hydrology education” 8 Recent effort – University hydrology educators surveyed about current teaching methods & ways that curriculum and instructor preparation could be improved 8 Major challenge within hydrology community – identifying “common principles, core knowledge, and approaches” 8 Background Hydrology has become increasingly interdisciplinary & technologically complex Limited number of studies of student understanding of hydrology & how it evolves over the course of schooling 5 Calls for examining, evaluating & enhancing hydrology education at both the upper-division and graduate level 2,6,7 Early effort – extensive survey of topics covered in college hydrology courses provided as resource to the community 3 Recent calls to “analyze, synthesize, and unite hydrology education” 8 Recent effort – University hydrology educators surveyed about current teaching methods & ways that curriculum and instructor preparation could be improved 8 Major challenge within hydrology community – identifying “common principles, core knowledge, and approaches” 8 Categories Research Question How can we characterize and assess upper division and graduate student thinking about physical hydrology? Research Question How can we characterize and assess upper division and graduate student thinking about physical hydrology? Rubric for Assessing Student Understanding of Physical Hydrology Rubric Development Process Implementation Obtain informed consent Administration of pre-/post- test Analysis of Student Artifacts R1 – Open-coding R2 – Rubric used to code sample of artifacts R3 – Team used rubric to code set of pre-/post- tests Met to negotiate consensus Development/Revision of Rubric R1 – Codes grouped by theme Four categories established Rubric drafted R2 – Revisions made to rubric R3 – Guidelines created Revisions made to both assessment & rubric