1 An Evaluation Plan and Tool Kit for the Archibald Bush Innovative Teaching and Technology Strategies Grant Valerie Ruhe and J.D. Walker, Center for Teaching and Learning Services and Digital Media Center University of Minnesota

Evaluation Plan  Evaluation: A formal appraisal of the quality of an educational phenomenon (Popham, 1993)  Gathering evidence to determine how well the innovation is “working”.  What does the grant say about the evaluation plan?

Foci of the Bush Grant Course-level Evaluation  Student Outcomes  Learning Process: Student Engagement Reflective Learning Responsible Learning

The Bush grant re: Evidence  Grades—ABC and DFW rates (outcomes).  Faculty Reflection Logs and conversations with consultants (process)  Surveys e.g. compare engagement across groups: with/without innovation and/or:  Surveys to compare across time, e.g. before/after the innovation.

Survey Items: Closed-ended (Examples)  Student Engagement  Mostly, I come to class because I want the certificate/degree.  Most of the time, I enjoy this class.  Reflective Learning  I enjoy applying theories or concepts from this class to new situations.  I sometimes find myself thinking about the lecture after it has ended.  Responsible Learning  I usually come to class prepared.  I usually review my lecture notes after class.

Survey Items: Open-ended  What are the most important benefits of this innovation for you?  What are the drawbacks?  What kind of problems have you had with the technology? Be specific.  What suggestions do you have for improvement?

Tailoring the Evaluation Plan to Meet Your Needs  What do you want to know?  Collect the kind of evidence to find out what you want to know, for example:  Business: experimental design with two treatment groups and long pre- and post- intervention surveys.  Architecture: a series of short surveys, interviews and observations, then analyze them for recurring themes.

Optional Evaluation Methods  Scoring Rubrics for student assignments  Class Observation: take notes, analyze them for themes, can inform surveys.  Narrative: To learn about learners’ reflective practices. Triangulation is important.  Interviews: Open-ended items, can deviate, tape-record and analyze for themes.  Focus groups: Validity checks.

Implementing the Plan  Decide what you want to know  Choose your preferred methods/tools  Write surveys and/or interview protocols  Collect data  Data entry: training?  Data analysis

Tools: Scoring Rubrics  “Scoring rubrics…guide the analysis of the products and/or processes of students' efforts…[and] provide a description of what is expected at each score level.” (Moskal, 2000)  a systematic way of evaluating qualitative data; a way of reducing subjectivity

Tools: Scoring Rubrics Development process:  break general concepts down into evaluation criteria  develop descriptions of degrees to which, and ways in which, the criteria can be satisfied  conduct pilot testing on sample data

Tools: Surveys Survey questions should:  be interpreted by respondents in the same way;  ask for information that respondents are able to provide;  provide information that is interpretable by you.

Tools: Questionnaires Time estimates: How many hours per day do you typically study?  Less than 1 hour  1 – 1.5 hours  1.5 – 2 hours  hours  More than 2.5 hours  Less than 2.5 hours  2.5 – 3 hours  3 – 3.5 hours  3.5 – 4 hours  More than 4 hours

Tools: Questionnaires Scale issues: How would you rate your instructor’s knowledge of the subject matter of this class?  poor  fair  adequate  good  very good  exceptional

Tools: Questionnaires Scale issues: How effective were the small group discussions in helping you to learn the course material?  very effective  effective  don’t know  ineffective  very ineffective

Tools: Questionnaires Item order effects: Which of the following activities helped you to learn the course material?  Lectures  Large group discussions  Small group discussions  Group assignments  Course readings  Studying for quizzes

Tools: Questionnaires Open versus closed-ended questions: How easy or difficult to use did you find each of the components of our course website?

Tools: Questionnaires Open versus closed-ended questions: How easy or difficult to use did you find these components of our course website? Discussions  very easy Quizzes  easy Content modules  difficult Audio files  very difficult PowerPoint files etc.

Tools: Questionnaires Interpretability: Which of these statements best describes your professor?  very good teacher, but not very approachable  about the same on approachability and teaching quality  very approachable, but not a very good teacher  a born teacher  extremely approachable