1. Assessment and Evaluation Feedback to Teams Summer 2015

2. Data Collected GLE Multiple Choice Pre/Post Essays (Interdisciplinary, Systems) Summative Assessments Attitude and Career Interest Pre/Post Demographic data

3. GLE Pre/Post Composite Results Pre 6.0 +/- 2.0 Post 6.7 +/- 1.9 Norm Gain = 11% n=424 Pre 6.8 +/- 1.9 Post 7.0 +/- 2.0 Norm Gain = 3% n=890 GLE Data F14-S15 Higher start All in Geoscience Courses ~50% Geoscience Courses

4. GLE Pre/Post n=65n=39n=72 n=21 n=35 (n=72; Norm Gain = 15%) (n= 65, Norm Gain = 11%)

5. Interdisciplinary Essay Knowledge of Earth system interactions can influence how people make decisions about global challenges. Identify and describe a global challenge that society will likely face in the next 50 years. Explain how the science related to that challenge informs economic, social, and/or political decision making related to the global challenge you described.

6. Interdisciplinary Essay Geoscience Courses F12-S13 Range of topics

7. Interdisciplinary Essay Non Integrate similar F12-S13 Answers link to content

8. Interdisciplinary Essay

9. Complex System A systems thinker can identify a system (a natural system, a human system, a linked human/environment system), understand how that system can be divided into interacting parts, and recognize that changes in one part of the system will affect other parts of the system. 1.Give an example of a real-world system and describe its parts. 2.Explain how parts of the system interact. Use systems concepts in your explanation (e.g., positive and negative feedbacks, equilibrium, rates, etc.) 3.Using your example system, discuss how an effect in one part of that system can be influenced by multiple causal factors.

10. (Very) Preliminary Systems Essay Results (2014-15) Overview (n=168 answers) – Most students could identify a system with its components – 57% identified a “Geo” system; 43% “Non-Geo” Scores – 65% score 0 or 1 – 33% score 2 (meant they wrote something about feedback) – 1 score 3 Big Picture = Students are not getting this concept although the question does not appear to be the problem 10

11. Evaluation of assessments/student work Needed a way to document learning across the project Assessments are diverse – Relate to discrete content related goals – Relate to InTeGrate Guiding Principles – Methodology differed from year 1 to year 2 – Year 1, feedback may be helpful during revisions – Year 2, feedback should be addressed

12. Rubric Evaluated extent to which collected assessments addressed module/unit goals – Can the goal be assessed with this artifact? Evaluated extent to which student work reflected progress toward meeting those goals – To what extent does the student work reflect progress toward goal? Same for InTeGrate guiding principles

13. Module Goal Rubric Elements 3 = Assessment/work clearly addresses major elements of goal 2 = Assessment/work lacks one or more major elements 1 = Assessment/work missing multiple key elements 0 = Assessment/work does not appear to address goals

14. Increased quality, quantity and diversity of Geoscience professionals in workforce Progress on nationally- urgent problems: Resources: - Minerals - Energy - Water - Food Environment: - Climate change - Natural hazards - Waste disposal - Environmental degradation Increased Geo- understanding of other professionals in workforce Increased Geo- understanding of the public (voters, consumers) Teachers with ability to tie Geo concepts to sustainability challenges Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges Introduction to the InTeGrate Evaluation Program InputsActivitiesOutcomesOutputs

15. Increased quality, quantity and diversity of Geoscience professionals in workforce Progress on nationally- urgent problems: Resources: - Minerals - Energy - Water - Food Environment: - Climate change - Natural hazards - Waste disposal - Environmental degradation Increased Geo- understanding of other professionals in workforce Increased Geo- understanding of the public (voters, consumers) Teachers with ability to tie Geo concepts to sustainability challenges Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges The Evaluation Team Frances Lawrenz Educ. Psychology U. Of Minnesota Ellen Iverson SERC Carlton College Carol Baldassari Program Evaluation Research Group Endicott College Kim Kastens Columbia University here Vali Mara Center for International Earth Science Information Network 15

16. Wormhole Feeds into, or causes, or leads to, or influences, or produces Increased quality, quantity and diversity of Geoscience professionals in workforce Progress on nationally- urgent problems: Resources: - Minerals - Energy - Water - Food Environment: - Climate change - Natural hazards - Waste disposal - Environmental degradation Increased Geo- understanding of other professionals in workforce Increased Geo- understanding of the public (voters, consumers) Teachers with ability to tie Geo concepts to sustainability challenges Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges What we told NSF we would do (1) Evaluate the project’s effectiveness at “expanding the number of students who enroll….” (2) Evaluate the project’s effectiveness at achieving “…enhanced learning…” (3) Evaluate the project’s effectiveness at achieving “… significant progress towards addressing the national challenge” of environmental sustainability (4) Formatively evaluate the partnership (5) Evaluate the effectiveness of the InTeGrate implementation program model 16

17. Wormhole Feeds into, or causes, or leads to, or influences, or produces Increased quality, quantity and diversity of Geoscience professionals in workforce Progress on nationally- urgent problems: Resources: - Minerals - Energy - Water - Food Environment: - Climate change - Natural hazards - Waste disposal - Environmental degradation Increased Geo- understanding of other professionals in workforce Increased Geo- understanding of the public (voters, consumers) Teachers with ability to tie Geo concepts to sustainability challenges Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges Tools and approaches Database of participants in all workshops, meetings, teams, courses, etc. Pre-/post-instruction measures of students, probing attitudes, geoscience literacy, and ability to tackle grand challenges Reflection surveys of materials developers and testers at key points throughout development/testing process Interviews with selected leaders and participants Document examination: websites, instructional materials, etc. Observation of selected meetings, workshops, etc. Mapping of flows of influence and information (logic model) 17

18. http://www.pbs.org/wgbh/nova/sciencenow/3410/03-ever-01.html Evaluation examines the effectiveness of InTeGrate in the aggregate, and does not monitor individual students, teachers, or modules. We focus on the flock, not the birds. 18

19. What you can expect Spreadsheet of GLE Pre- Post- answers by team (Emailed at a later date) Spreadsheet of Attitude Pre- Post- answers by team (Emailed at a later date) Evaluation of assessments and student work 19

20. How to use this feedback GLE/Attitude – analyze as you wish. Since numbers are small, analysis has limited utility Essays – Consider overall quality of answers and how they do or do not relate to module Evaluation of assessments – Address as warranted during revisions 20