Assessment: Measuring the Impact of our Programs on Students Cathy Manduca and Ellen Iverson SERC at Carleton College

Why Assess? “I don’t know what we are doing right” “We only have 120 hours”

Performance Evaluation Does the program address an important challenge? What is the quality of the implementation? What is the utility--are participants using it? Is it being used in different situations? Is it being shared by participants? John McLaughlin

Why Assess? “I don’t know what we are doing right” “We only have 120 hours” “We will lose majors if we do X” “Change is accomplished one funeral at a time” “That would really help me with the dean” “It must work because other institutions are doing it.”

Adoption of a Practice What results does it yield? How do you do it? Where is it flexible? How much does it cost? What aspects of context influence use? John McLaughlin

Powers of Observation Applied to Complex Systems

Experimental Design

Assessment and Geoscience Research Experimental design: What do I want to know? How can I obtain this information? What do I think is going on? How can I test this idea? Analysis: Is the experimental apparatus working? Is the experiment yielding the desired information? How can I verify my results? Interpretation: How do I determine causality? Is this the only interpretation?

Observing Complex Systems Why are undergraduate research experiences valuable? What do I think is going on? What can I observe to test my theory? How can I quantify my observations?

Geologic Strategies in Assessment Multiple working hypotheses Observations Describing, classifying, coding Inferring process and cause Probing complex systems

Understanding your model through concept maps Articulate and map change process Go from your assumptions to desired goal Present plausible rationale for how program works Analyze likelihood of success Clarify what should be measured when, how, and by whom Depict the links between the resources, activities, or services and desired outcomes Is a Critical thinking exercise to help you with your action plan. It lets you… goal-based, plausible, someone on one of the road checks said assessing is easy but if it fails what is plan B…this thinking helps you go through the what ifs and measurments when, how, by whom and then does it hold together

Internship logic model example Students gain from program: Who then impact: Which impact students’: Employers perception of program Success in finding a job in geoscience at graduation Longevity at first job Knowledge Skills Attitudes Networks Resources Students participate in internship initiative Other students’ perception of program So I’d like to take you through a simplified example for adding a student internship component to a program. KSAs- knowledge of industry S: communication, ability to work with team, balance competing demands, A greater confidence, greater awareness of relevance of education

Are you implementing your model? Is your model correct? Is what you are mapping what you think should happen? What are the impacts? Are the impacts caused by your planned actions?

Instruments to measure interventions Qualitative – self report and artifacts Open-ended survey questions Embedded assessment Artifacts such as reports or online discussion Interviews and focus groups Observations Quantitative - counting Data sets Surveys Counting – how many students, how many employers, how many internships, % partcipating and % satisfied (likert scale) Open-ended survey or interview – end of internship survey student /employer Embedded assessment – journal student keeps or could be online discussion artifacts or the assessment can be its own intervention by students giving oral report of internship back to intro geo class Focus groups to get a sense of their expectations/structure ahead of time

Why Assess? “I don’t know what we are doing right” “We only have 120 hours” “We will lose majors if we do X” “Change is accomplished one funeral at a time” “That would really help me with the dean” “It must work because other institutions are doing it.” What is the purpose? Who is the audience? What will make a compelling argument?