1. IDEA Student Ratings of Instruction Shelley A. Chapman, PhD Insight Improvement Impact ® University of Alabama Birmingham September 11, 2012

2. Plan for this Session What Makes IDEA Unique Conditions for Good Use Reflective Practice Framework Student Learning Framework Faculty Information Form Interpreting Reports Questions and Answers

3. Individual Development and Educational Assessment Kellogg Grant in 1975 Non-profit Organization 2000 Mission Insight Improvement Impact ® To help colleges and universities as they seek to improve teaching, learning, and leadership

4. What makes IDEA unique? 1. Focus on Student Learning 2. Focus on Instructor’s Purpose 3. Adjustments for Extraneous Influences 4. Validity and Reliability 5. Comparison Data 6. Flexibility

5. Conditions for Good Use The instrument Focuses on learning Provides suggested action steps for teaching improvement

6. Conditions for Good Use The Faculty Trust the process Value student feedback Are motivated to make improvements

7. Conditions for Good Use Campus Culture Teaching excellence - high priority Resources to improve - provided Student ratings - appropriate weight

8. Conditions for Good Use The Evaluation Process 30-50% of evaluation of teaching 6-8 classes, more if small (<10) Not over-interpreted (3-5 performance categories)

9. Reflective Practice using Individual Reports Collect Feedback Interpret Results Read & Learn Reflect & Discuss Improve IDEA resources that are keyed to reports Talk with colleagues Try new ideas Online, Paper What the reports say and what they mean

10. Student Learning Framework: 2 Assumptions Assumption 1: Types of learning must reflect the instructor’s purpose.

11. Student Diagnostic Form Assumption 2: Effectiveness determined by students’ progress on objectives stressed by instructor

12. Student Learning Model Specific teaching behaviors are associated with certain types of student progress under certain circumstances. Student Learning Teaching Behaviors Circumstances

13. Student Learning Model: Diagnostic Form Student Learning Items 21-32 Teaching Behaviors Items 1-20 Circumstances Students: Items 36-39, 43 Course: Items 33-35 Summary Items: 40-42 Research Items: 44-47 Up to 20 extra items

14. Student Learning Model: Short Form Summary Measures: Items 16-18 Experimental Questions: Items 14 20 Additional Questions Student Learning Items 1-12 Teaching Behaviors Circumstances Students: Items 13-15

15. Faculty Information Form (FIF)

16. FIF: Selecting Objectives 3-5 as “Essential” or “Important” Is it a significant part of the course? Do you do something specific to help students accomplish the objective? Does the student’s progress on the objective influence his or her grade? Be true to your course.

17. The Average Number of Objectives Selected by UAB: Spring 2012

18. Common Misconception #1 Students are expected to make significant progress on all 12 learning objectives in a given course.

19. Common Misconception #2 Effective instructors need to successfully employ all 20 teaching methods in a given course.

20. Relationship of Learning Objectives to Teaching Methods

21. Common Misconception #3 The 20 teaching methods items should be used to make an overall judgment about teaching effectiveness. Faculty Evaluation

22. Course Description Items (FIF) Used for research Best answered toward end of term Do NOT influence your results Bottom of Page 1 Top of page 2

23. IDEA Online

24. Email delivery/reminders Start/end dates determined by Institution Access is unlimited while available Questions can be added to student survey Objectives can be copied from previously completed FIFs IDEA Online: FIF Delivery

25. Copying Objectives

26. Email/Course embedded URL Blackboard Building Block Email reminders Start/end dates Determined by Institution Submission is confidential and restricted to one IDEA Online: Student Survey Delivery

27. Online Response Rates – Best Practices Create value for student feedback Monitor and Communicate through multiple modalities: Twitter Facebook Other Prepare Students Talk about it Syllabus

28. Example: Course Syllabus Objective 3: Learning to apply course material (to improve thinking, problem solving, and decisions) Students will be able to apply the methods, processes, and principles of earth science to understanding natural phenomena Students will think more critically about the earth and environment Objective 8: Developing skill in expressing myself orally or in writing Students will be able to present scientific results in written and oral forms IDEA Center Learning Objective Course Learning Outcomes

29. Reflective Practice with IDEA Collect Feedback Interpret Results Read & Learn Reflect & Discuss Improve IDEA Student Ratings of Instruction Individual Reports Group Summary Reports Benchmarking Reports

30. Diagnostic Report Overview  Page 1 – Big Picture  How did I do? Page 3 – Diagnostic  What can I do differently? Page 2 – Learning Details  What did students learn? Page 4 – Statistical Detail  Any additional insights?

31. Your Average (5-point Scale) RawAdj. A.Progress on Relevant Objectives 1 Four objectives were selected as relevant (Important or Essential—see page 2) 4.14.3 1 If you are comparing Progress on Relevant Objectives from one instructor to another, use the converted average. The Big Picture

32. Progress On Relevant Objectives 4 4.3 + 4.3 4.1 4.2 3.6 5

33. Summary Evaluation: Five-Point Scale Report Page 1 Your Average Score (5-point scale) RawAdj. A.Progress on Relevant Objectives Four objectives were selected as relevant (Important or Essential—see page 2) 4.14.3 Overall Ratings B. Excellent Teacher4.74.9 C. Excellent Course4.14.4 D. Average of B & C4.44.7 Summary Evaluation (Average of A & D)4.34.5 50% 25%

34. Adjusted Scores

35. Student Work Habits (#43DF, #13SF) Student Motivation (#39DF, #15SF) Class Size (Enrollment, FIF)

36. Understanding Adjusted Scores

37. Work Habits (Item 43) Student Motivation (Item 39) Hig h Avg. Avg. Low Avg. Low High 4.484.384.284.134.04 High Avg. 4.384.294.143.963.76 Average 4.284.144.013.833.64 Low Avg. 4.154.053.883.703.51 Low 4.113.963.783.583.38 Impact of Extraneous Factors Gaining Factual Knowledge – Average Progress Ratings Technical Report 12, page 40

38. Work Habits (Item 43) Student Motivation (Item 39) Hig h Avg. Avg. Low Avg. Low High 4.484.38 High Avg. 4.384.29 Average 4.01 Low Avg. 3.703.51 Low 3.583.38 Impact of Extraneous Factors Gaining Factual Knowledge – Average Progress Ratings Technical Report 12, page 40

39. Do raw scores meet or exceed expectations? * Are adjusted scores lower or higher than raw scores? Use adjusted scores Use raw scores Lower Yes Higher When to Use Adjusted Scores for Personnel Decisions *Expectations defined by your unit. No

40. Comparisons (Norms): Converted Averages Able to compare scores on the same scale Use T Scores Average = 50 Standard Deviation = 10 They are not percentiles or percentages

41. Comparisons (Norms): Converted Averages

42. Comparison Scores Distribution 40% Similar Lower 20% 10% Much Lower 10% Much Higher 20% Higher Gray Area

43. Comparison Scores

44. Using the Report to Improve Course Planning and Teaching

45. Page 2: What did students learn?

46. Suggested Action Steps #16 #18 #19

47. POD-IDEA Notes IDEA Website

48. POD-IDEA Notes Background Helpful Hints Application for online learning Assessment Issues References and Resources

49. References and Links to Helpful Resources are Provided

50. IDEA Papers Resources for Faculty Evaluation Faculty Development

51. Reflective Practice Collect Feedback Interpret Results Read & Learn Reflect & Discuss Improve POD-IDEA Notes IDEA Papers Meet with colleagues to reflect Interpret Reports Paper or Online Try something new

52. Questions ? www.theideacenter.org Visit our IDEA Help Community!IDEA Help