Gwynn Mettetal

 Discuss different ways to assess outcomes  Help you decide which methods would be best for your project

 Who are you?  What sort of Vision 2020 project are you planning? (the two sentence version)

 Assessment—evidence that your project is making a difference  You MUST assess the effectiveness of your Vision 2020 grant to get continued funding!  Lots of strategies possible  Depends on your goals  Depends on your situation

 Quantitative (numbers)  Grades, attendance, ratings on a scale, retention rate  Qualitative (words)  Interviews, essays, open ended survey questions  Both are fine, just different

 Existing data (easiest, already there) ◦ Student records ◦ Archival data ◦ Student work in course  Conventional sources (easy, but must generate) ◦ Behavioral data—journals, library usage ◦ Perceptual data—surveys, focus groups, interviews  Inventive sources (difficult) ◦ Products or performances

 Must treat students respectfully  Must protect privacy  Must “do no harm”  Collecting new data (not coursework) from your own students? ◦ Have someone else collect and hold until grades are in ◦ Can’t force them to participate ◦ Can’t take up too much instruction time  Institutional Review Board (IRB) ◦ If planning to publish

 Add power--compare groups! ◦ Before and after ◦ Different course units ◦ This semester and last ◦ Two sections with different methods ◦ Your class to that of another instructor  Be realistic--start small

 Validity—does your evidence (data) mean what you think it means?  Example  test scores = deep learning?  What if just rote memory?  What if students cheated?

 Reliability—would you get the same evidence if you collected it again? Or was this just a fluke?  Example:  Test scores = deep learning?  What if you gave again next week and scores were very different?

 In general, hard to have both.  Real life is messy (valid, not as reliable)  Experiments are controlled (reliable, not as valid)  Solution is...

 Get several different types of data  Different sources: ◦ Instructors, students, advisors, records  Different methods: ◦ Surveys, observations, student work samples  Different times: ◦ Start and end of semester, two different classes, two different semesters

Course evaluations final project rubric Comparison to last semester’s class

What data could YOU collect?

 Qualitative analyses: look for themes in words and behaviors Theme 1: Students understood more abstract concepts after group discussion. (Follow with quotes from student exams, other evidence.)

 Quantitative analyses: simple graphs, tables  Simple statistics: means, correlations, t-tests

Focus on practical significance, more than statistical significance

What would convince YOU?

 If evidence was good, keep your old strategy  If evidence was weak, tinker to improve your strategy  Plan to assess again, after working with a new group of students  You will need to show how you used your data to get continued Vision 2020 funding!