Assessment and Evaluation Feedback to Teams David Steer and Kim Kastens

Data Collected GLE Multiple Choice Pre/Post Essays Post Attitude and Career Interest Pre/Post Comments from initial reviews Student work

GLE Pre/Post No surprises Pre 6.0 +/- 2.0 Post 6.7 +/- 1.9

Interdisciplinary Essay Knowledge of Earth system interactions can influence how people make decisions about global challenges. Identify and describe a global challenge that society will likely face in the next 50 years. Explain how the science related to that challenge informs economic, social, and/or political decision making related to the global challenge you described.

Interdisciplinary Essay Consistent grading Range of topics

Interdisciplinary Essay Non Integrate similar Answers link to content

Interdisciplinary Essay

Complex System Earth consists of interacting systems that exchange energy and/or mass. Describe how two components of the Earth System interact with one another, the rate at which they interact and how a change in one system can lead to change in the other system.

Complex System Essay Single Grader Needs Attention

Evaluation of assessments/student work Needed a way to document learning across the project Assessments are diverse Relate to discrete content related goals Relate to InTeGrate Guiding Principles Methodology differed from year 1 to year 2 Year 1, feedback may be helpful during revisions Year 2, feedback should be addressed

Rubric Evaluated extent to which collected assessments addressed module/unit goals Can the goal be assessed with this artifact? Evaluated extent to which student work reflected progress toward meeting those goals To what extent does the student work reflect progress toward goal? Same for InTeGrate guiding principles

Module Goal Rubric Elements 3 = Assessment/work clearly addresses major elements of goal 2 = Assessment/work lacks one or more major elements 1 = Assessment/work missing multiple key elements 0 = Assessment/work does not appear to address goals

Introduction to the InTeGrate Evaluation Program Inputs Activities Outcomes Outputs Increased quality, quantity and diversity of Geoscience professionals in workforce Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges Progress on nationally- urgent problems: • Resources: - Minerals - Energy - Water - Food • Environment: - Climate change - Natural hazards - Waste disposal - Environmental degradation Increased Geo- understanding of other professionals in workforce Increased Geo- understanding of the public (voters, consumers) Teachers with ability to tie Geo concepts to sustainability challenges

The Evaluation Team Carol Baldassari Kim Kastens Jackie DeLisi here here Carol Baldassari Program Evaluation Research Group Endicott College Kim Kastens Education Development Center The Evaluation Team here Jackie DeLisi Education Development Center Ellen Iverson SERC Carlton College Frances Lawrenz Educ. Psychology U. Of Minnesota Increased quality, quantity and diversity of Geoscience professionals in workforce Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges Progress on nationally- urgent problems: • Resources: - Minerals - Energy - Water - Food • Environment: - Climate change - Natural hazards - Waste disposal - Environmental degradation Increased Geo- understanding of other professionals in workforce Increased Geo- understanding of the public (voters, consumers) Teachers with ability to tie Geo concepts to sustainability challenges

What we told NSF we would do (1) Evaluate the project’s effectiveness at “expanding the number of students who enroll….” (2) Evaluate the project’s effectiveness at achieving “…enhanced learning…” (3) Evaluate the project’s effectiveness at achieving “… significant progress towards addressing the national challenge” of environmental sustainability (4) Formatively evaluate the partnership (5) Evaluate the effectiveness of the InTeGrate implementation program model Increased quality, quantity and diversity of Geoscience professionals in workforce Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges Progress on nationally- urgent problems: • Resources: - Minerals - Energy - Water - Food • Environment: - Climate change - Natural hazards - Waste disposal - Environmental degradation Increased Geo- understanding of other professionals in workforce Increased Geo- understanding of the public (voters, consumers) Teachers with ability to tie Geo concepts to sustainability challenges Wormhole Feeds into, or causes, or leads to, or influences, or produces

Tools and approaches Database of participants in all workshops, meetings, teams, courses, etc. Pre-/post-instruction measures of students, probing attitudes, geoscience literacy, and ability to tackle grand challenges Reflection surveys of materials developers and testers at key points throughout development/testing process Interviews with selected leaders and participants Document examination: websites, instructional materials, etc. Observation of selected meetings, workshops, etc. Mapping of flows of influence and information (logic model) Increased quality, quantity and diversity of Geoscience professionals in workforce Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges Progress on nationally- urgent problems: • Resources: - Minerals - Energy - Water - Food • Environment: - Climate change - Natural hazards - Waste disposal - Environmental degradation Increased Geo- understanding of other professionals in workforce Increased Geo- understanding of the public (voters, consumers) Teachers with ability to tie Geo concepts to sustainability challenges Wormhole Feeds into, or causes, or leads to, or influences, or produces

Evaluation examines the effectiveness of InTeGrate in the aggregate, and does not monitor individual students, teachers, or modules. We focus on the flock, not the birds. http://www.pbs.org/wgbh/nova/sciencenow/3410/03-ever-01.html

What you can expect Spreadsheet of GLE Pre- Post- answers by team Spreadsheet of Attitude Pre- Post- answers by team Evaluation of assessments and student work

How to use this feedback GLE/Attitude – analyze as you wish. Since numbers are small, analysis has limited utility Essays – Consider overall quality of answers and how they do or do not relate to module Evaluation of assessments Year 1: Use caution. Data and assessments were not designed to meet these criteria Year 2: Address as warranted during revisions