1. MAINTAINING QUALITY IN BLENDED LEARNING: FROM CLASSROOM ASSESSMENT TO IMPACT EVALUATION PART II: IMPACT EVALUATION Patsy Moskal (407) 823-0283 pdmoskal@mail.ucf.edu http://rite.ucf.edu

2. DEFINING SOTL

3. SCHOLARSHIP OF TEACHING AND LEARNING (SOTL)  Scholarly research on effective teaching and student learning  Ernest Boyar, 1990, Scholarship Reconsidered  Carnegie Academy for the Scholarship of Teaching and Learning  Scholarship Assessed (1997) Charles Glassick, Mary Taylor Huber, and Gene Maeroff

4. NATIONAL ORGANIZATIONS DEVOTED TO THE SCHOLARSHIP OF TEACHING & LEARNING  International Society for the Scholarship of Teaching & Learning  American Association for Higher Education & Accreditation  Carnegie Academy for the Scholarship of Teaching & Learning  The Carnegie Foundation for the Advancement of Teaching

5. MOTIVATION FOR SOTL  Research new instructional methods or classroom changes for improvement  Provides opportunities for publication and presentation  Tenure and promotion  Supporting data for accreditation, grant proposals, etc…

6. JOURNALS DEVOTED TO SOTL  Journal of Scholarship of Teaching & Learning  Teaching in Higher Education  New Directions for Teaching & Learning  Journal on Excellence for Teaching and Learning  Achieving Learning in Higher Education

7. DISCIPLINE SPECIFIC JOURNALS  Journal of Education for Business  American Biology Teacher  Journal of Research in Science Teaching  Studies in Art Education  Teaching and Learning in Medicine  Journal of Nursing Information  Arts and Humanities in Higher Education

8. HOW TO ACCOMPLISH SOTL

9. CHALLENGES IN COMPLETING SOTL RESEARCH  Faculty lack of expertise in research/stats  Lack of time, resources  Minimize class disruption  Challenges to designing research

10. THE ALICE IN WONDERLAND APPROACH TO ASSESSMENT “Would you tell me, please, which way I ought to go from here?” said Alice. “That depends a good deal on where you want to get to,” said the Cat. “I don’t much care where—” said Alice. “Then it doesn’t matter which way you go,” said the Cat. “—so long as I get somewhere,” Alice added “Oh, you’re sure to do that,” said the Cat, “if you only walk around long enough.” --Lewis Carroll

11. PRINCIPLES THAT GUIDE OUR EVALUATION  Evaluation must be objective.  Evaluation must conform to the culture  Uncollected data cannot be analyzed.  Data do not equal information.  Qualitative and quantitative approaches must complement each other.  Evaluation must show an impact.  Results may not be generalized

12. THE KEY TO SUCCESSFULLY ACCOMPLISHING SOTL… 1. Clear Goals 2. Adequate Preparation 3. Appropriate Methods 4. Significant Results 5. Effective Presentation 6. Reflective Critique (Glassick, Huber, Maeroff, 1997)

13. CLEAR GOALS  Do you have a clear goal?  Is your goal doable?

14. THE K.I.S.S. PRINCIPLE OF ASSESSMENT DESIGN Keep It Simple and Straightforward! A simple, doable design is better than a complex, impossible design that is never completed!

15. HITTING THE TARGET…  What do you want to know?  You must clearly define your questions.  And, if your data doesn’t answer your questions… what’s the point?!

16. ADEQUATE PREPARATION  Have you looked at the literature?  Can you do your study?  If you need help, can you get support?

17. FINDING SOURCES OF HELP  Faculty development center  Institutional research  Office of Assessment  Statistics or research folks  Content experts  Other researchers

18. APPROPRIATE METHODS  Do you have data or can you get it?  Do your methods “fit” your goal and objectives?  Be prepared to rewind and repeat!

19. SOME ASSESSMENT/EVALUATION TOOLS  Surveys  Focus groups  Course-based performance  Observations  Tests/exams  Pre-collected data  E-portfolios  Rubrics

20. SIGNIFICANT RESULTS  Did you achieve your objectives?  Does this work inform and add to the field?

21. STATISTICALLY OR PRACTICALLY SIGNIFICANT??  Don’t let the statistics run your design!  Statistically significant may not be practically significant.  What about the random sample?  Quantitative and qualitative approaches must complement each other

22. SOME EXAMPLES SURVEYS

23. SURVEY PROS AND CONS PROS  Easy to administer  Electronic is possible  Researcher’s questions  Can tell you “what”  Open ended can tell you more  Can look at demographics CONS  Student opinions  Low response rate  Timing can impact results  Wording of questions is important  “Over surveyed” students

24. STUDENT SATISFACTION IN BLENDED COURSES N = 36,801 49% 17% Very Satisfied Unsatisfied SatisfiedNeutral 28% Very Unsatisfied 6% 2% Percent 24

25. STUDENTS’ POSITIVE PERCEPTIONS ABOUT BLENDED LEARNING  Convenience  Reduced Logistic Demands  Increased Learning Flexibility  Technology Enhanced Learning Reduced Opportunity Costs for Education

26. LESS POSITIVES WITH BLENDED LEARNING  Reduced Face-to-Face Time  Technology Problems  Reduced Instructor Assistance  Overwhelming  Increased Workload Increased Opportunity Costs for Education 26

27. STUDENT SATISFACTION IN FULLY ONLINE AND BLENDED COURSES 47% Fully online (N = 67,433) Blended (N = 36,801) 49% 17% 6% Very Satisfied UnsatisfiedSatisfied Neutral 28% 16% Very Unsatisfied 6% 3% 2% Percent

28. SOME EXAMPLES PRE-EXISTING DATA

29. PRE-EXISTING DATA PROS AND CONS PROS  Already collected  May have longitudinal data  Often in electronic spreadsheet CONS  Someone else decided what to collect  May not be in a form of your choosing  Requires permission and obtaining from others

30. STUDENT SUCCESS AND WITHDRAWAL

31. SUCCESS RATES BY MODALITY SPRING 09 THROUGH SPRING 10 F2F n=456,125 Blended n=30,361 Fully Online n=83,274 Percent

32. WITHDRAWAL RATES BY MODALITY SPRING 09 THROUGH SPRING 10 Percent F2F n=456,125 Blended n=30,361 Fully Online n=83,274

33. STUDENT EVALUATION OF INSTRUCTION SEI

34. Facilitation of learning Communication of ideas Excellent Very Good Fair Poor Then... The probability of an overall rating of Excellent =.97 & The probability of an overall rating of Fair or Poor =.00 If... A decision rule for the probability of faculty member receiving an overall rating of Excellent (n=1,280,890) Respect and concern for students Good

35. A COMPARISON OF EXCELLENT RATINGS BY COURSE MODALITY--UNADJUSTED AND ADJUSTED FOR INSTRUCTORS SATISFYING RULE 1 (N=1,171,664) Blended48.997.2 Online47.697.3 Enhanced46.897.5 F2F45.797.2 ITV34.296.6 CourseOverallIf Rule 1 Modality % Excellent % Excellent

36. EFFECTIVE PRESENTATION  Did you remember your objectives?  Did you remember your audience?  Did you present your message in a clear, understandable manner?

37. DATA DO NOT EQUAL INFORMATION…  Data, by itself, answers no questions and is nothing more than a bunch of numbers and/or letters.  How you interpret the data for others can determine how well they understand.  Visuals are good!  Ongoing assessment is best.

38. REFLECTIVE CRITIQUE  Did you critique your own work?  What worked well and what didn’t work?  Where do you go from here?

39. EVALUATION CYCLE 1. Define your question(s). 2. Determine methods that can answer question(s). 3. Implement methods, gather, and analyze data. 4. Interpret results – did you answer your question? 5. Make decisions based on results.

40. ADDITIONAL THINGS TO CONSIDER

41. SOME ISSUES TO PONDER WITH DATA  Uncollected data cannot be analyzed!  Data is not always “clean” and can require work  Look for data you already have

42. TACKLING IRB  What is it?  Training may be required  A MUST if you want to publish or present  Don’t be intimidated!

43. UNEXPECTED ISSUES  Technology challenges  People challenges  No response  Dirty data

44. MAKE AN IMPACT WITH YOUR ASSESSMENT…  The final step in assessment (or evaluation) should be determining how your results can impact decisions for the future.

45. EXTENDING SOTL TO PROGRAM RESEARCH AND BEYOND

46. IMPACT EVALUATION AS A MATTER OF SCALE Level FewMany Individual Program Department College Institution There is added value at every level 46 A little A lot Opportunity Costs

47. RESEARCH INITIATIVE FOR TEACHING EFFECTIVENESS What services do we provide?  Research design  Survey construction & administration  Data analysis & interpretation  Results provided in “charts and graphs” format  Publication and presentation assistance

48. CONTACT INFORMATION Patsy Moskal, Ed.D. (407) 823-0283 pdmoskal@mail.ucf.edu http://rite.ucf.edu