1. Assessment and Evaluation Feedback to Teams
David Steer and Kim Kastens

2. Data Collected GLE Multiple Choice Pre/Post Essays Post
Attitude and Career Interest Pre/Post
Comments from initial reviews
Student work

3. GLE Pre/Post
No surprises
Pre 6.0 +/- 2.0
Post 6.7 +/- 1.9

4. Interdisciplinary Essay
Knowledge of Earth system interactions can influence how people make decisions about global challenges. Identify and describe a global challenge that society will likely face in the next 50 years. Explain how the science related to that challenge informs economic, social, and/or political decision making related to the global challenge you described.

5. Interdisciplinary Essay
Consistent grading
Range of topics

6. Interdisciplinary Essay
Non Integrate similar
Answers link to content

7. Interdisciplinary Essay

8. Complex System
Earth consists of interacting systems that exchange energy and/or mass. Describe how two components of the Earth System interact with one another, the rate at which they interact and how a change in one system can lead to change in the other system.

9. Complex System Essay
Single Grader
Needs Attention

10. Evaluation of assessments/student work
Needed a way to document learning across the project
Assessments are diverse
Relate to discrete content related goals
Relate to InTeGrate Guiding Principles
Methodology differed from year 1 to year 2
Year 1, feedback may be helpful during revisions
Year 2, feedback should be addressed

11. Rubric
Evaluated extent to which collected assessments addressed module/unit goals
Can the goal be assessed with this artifact?
Evaluated extent to which student work reflected progress toward meeting those goals
To what extent does the student work reflect progress toward goal?
Same for InTeGrate guiding principles

12. Module Goal Rubric Elements
3 = Assessment/work clearly addresses major elements of goal
2 = Assessment/work lacks one or more major elements
1 = Assessment/work missing multiple key elements
0 = Assessment/work does not appear to address goals

13. Introduction to the InTeGrate Evaluation Program
Inputs
Activities
Outcomes
Outputs
Increased quality, quantity and diversity of Geoscience professionals in workforce
Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges
Progress on nationally- urgent problems:
• Resources:
- Minerals
- Energy
- Water
- Food
• Environment:
- Climate change
- Natural hazards
- Waste disposal
- Environmental
degradation
Increased Geo- understanding of other professionals in workforce
Increased Geo- understanding of the public (voters, consumers)
Teachers with ability to tie Geo concepts to sustainability challenges

14. The Evaluation Team Carol Baldassari Kim Kastens Jackie DeLisi
here
here
Carol Baldassari
Program Evaluation Research Group
Endicott College
Kim Kastens
Education Development Center
The Evaluation Team
here
Jackie DeLisi
Education Development Center
Ellen Iverson
SERC
Carlton College
Frances Lawrenz
Educ. Psychology
U. Of Minnesota
Increased quality, quantity and diversity of Geoscience professionals in workforce
Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges
Progress on nationally- urgent problems:
• Resources:
- Minerals
- Energy
- Water
- Food
• Environment:
- Climate change
- Natural hazards
- Waste disposal
- Environmental
degradation
Increased Geo- understanding of other professionals in workforce
Increased Geo- understanding of the public (voters, consumers)
Teachers with ability to tie Geo concepts to sustainability challenges

15. What we told NSF we would do
(1) Evaluate the project’s effectiveness at “expanding the number of students who enroll….”
(2) Evaluate the project’s effectiveness at achieving “…enhanced learning…”
(3) Evaluate the project’s effectiveness at achieving “… significant progress towards addressing the national challenge” of environmental sustainability
(4) Formatively evaluate the partnership
(5) Evaluate the effectiveness of the InTeGrate implementation program model
Increased quality, quantity and diversity of Geoscience professionals in workforce
Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges
Progress on nationally- urgent problems:
• Resources:
- Minerals
- Energy
- Water
- Food
• Environment:
- Climate change
- Natural hazards
- Waste disposal
- Environmental
degradation
Increased Geo- understanding of other professionals in workforce
Increased Geo- understanding of the public (voters, consumers)
Teachers with ability to tie Geo concepts to sustainability challenges
Wormhole
Feeds into, or causes, or leads to, or influences, or produces

16. Tools and approaches
Database of participants in all workshops, meetings, teams, courses, etc.
Pre-/post-instruction measures of students, probing attitudes, geoscience literacy, and ability to tackle grand challenges
Reflection surveys of materials developers and testers at key points throughout development/testing process
Interviews with selected leaders and participants
Document examination: websites, instructional materials, etc.
Observation of selected meetings, workshops, etc.
Mapping of flows of influence and information (logic model)
Increased quality, quantity and diversity of Geoscience professionals in workforce
Increased motivation and ability of Geoscience professionals to address resource and environmental Grand Challenges
Progress on nationally- urgent problems:
• Resources:
- Minerals
- Energy
- Water
- Food
• Environment:
- Climate change
- Natural hazards
- Waste disposal
- Environmental
degradation
Increased Geo- understanding of other professionals in workforce
Increased Geo- understanding of the public (voters, consumers)
Teachers with ability to tie Geo concepts to sustainability challenges
Wormhole
Feeds into, or causes, or leads to, or influences, or produces

17. Evaluation examines the effectiveness of InTeGrate in the aggregate, and does not monitor individual students, teachers, or modules. We focus on the flock, not the birds.

18. What you can expect Spreadsheet of GLE Pre- Post- answers by team
Spreadsheet of Attitude Pre- Post- answers by team
Evaluation of assessments and student work

19. How to use this feedback
GLE/Attitude – analyze as you wish. Since numbers are small, analysis has limited utility
Essays – Consider overall quality of answers and how they do or do not relate to module
Evaluation of assessments
Year 1: Use caution. Data and assessments were not designed to meet these criteria
Year 2: Address as warranted during revisions