1. Evaluating Educational Technology Brian McNurlen & Chris Migotsky University of Illinois at Urbana-Champaign

2. Perspective & Context “Like every shining new gift, sensory devices may dazzle one into all kinds of wishful thinking…Audio-visual materials must be understood in their relationship to teaching as a whole and to the learning process as a whole.” Edgar Dale, 1946

3. Hold Your Horses!

4. Who Should Evaluate? Instructors who want to try something new with educational technology Instructors who want to decide if what they currently do is effective Classroom or online (or both) Others?

5. Why Evaluate? Systematic documentation of your teaching What is working? How can it be improved? Other reasons?

6. What is the Focus of the Evaluation? What is the product or process that needs to be resolved, improved? Is it something that can be addressed as an instructor, with limited resources? What counts as “success”?

7. Activity #1 Select a program, product, or service to be evaluated Define how the program/product/service fits into your course (goals/objectives) Discuss with partner Share with group

8. Special Issues Associated with Educational Technology Continuity with course objectives Access issues Technical support Time Costs Ongoing technology development

9. Defining Key Evaluation Terms Evaluand Formative vs. summative Evaluation vs. assessment Evaluation vs research Logic model Resources for further study –Evaluation Thesaurus (Scriven) –Web sites on last slides

10. Evaluation Process 1.Define evaluand and connection to course goals 2.Methodology & data collection 3.Data analysis 4.Conclusion & implications

11. 1. Define Evaluand and its Fit with Course Boundaries What, how, & why…logic model Intended and unintended consequences

12. 2. Methodology & Data Collection Institutional Review Board (IRB) Qualitative –Interviews –Observations/VT –Focus Groups –Quick writes Quantitative –Surveys –Tests –Participation, attendance, server hits, etc Pilot test & timing

13. Activity #2 Outline an evaluation plan –Evaluation questions –Tools or methods you’re going to use –Timeline –Expected outcomes Keep it simple –What can be accomplished in a semester? –What can be accomplished with limited resources and staff? Share in small groups

14. Clicker study: Evaluation Questions Will implementing this clicker program affect how students perform academically? Will implementing this clicker program affect the overall student feedback scores for my course?

15. Design Comparison groups –Two different lecture sections Independent variable –Use of clickers, along with instructional plan –Other section just uses the instructional plan Dependent variable –Some measure of academic performance, perhaps a cumulative final

16. Timeline 1.July– Select clicker technology, training 2.August-- TA training 3.Late August– Student training, pre-test 4.Sept.-December-- 16 weeks of treatment 5.Reliability observations-- October, December 6.December-- post-test 7.April-- follow-up reliability observation

17. 3. Data Analysis Results…what does it mean? –Level of analysis –Practical significance –Outliers/extremes –Baseline/benchmark data –Response rates

18. 4. Conclusion & Implications What do your results tell you? –Make changes –Sharing your results (IRB?) –Follow-up evaluation –Share with next generation of students

19. Evaluating Ed Tech: Common Issues Too much data Evaluating something that could be done by other means, or is impossible to do without the technology Technology is a constantly moving target so results will be quickly outdated Any other issues?

20. Evaluation Resources Teaching Center or Survey Center on your campus Western Michigan University Evaluation Center http://www.wmich.edu/evalctr/ Key Evaluation Checklist http://www.wmich.edu/evalctr/checklists/checklistmenu.htm#models An Educator's Guide to Evaluating The Use of Technology in Schools and Classrooms http://www.ed.gov/pubs/EdTechGuide/index.html Evaluation Springboard http://www.evaluationspringboard.org/ Survey Monkey http://www.surveymonkey.com

21. Teaching & Technology Resources Principles of Good Practice http://fc.byu.edu/tpages/tchlrn/7princip.htm http://www.tltgroup.org/programs/seven.html The Technology Source http://technologysource.org/article/toward_more_effectiv e_instructional_uses_of_technology/ ECAR www.educause.edu/ecar

22. Teaching & Technology Resources An analysis of technology enhancements in a large lecture course. http://www.educause.edu/ir/library/pdf/eqm0335.pdf Planning and Sustaining Evaluation of Instructional Technology Support Programs http://www.educause.edu/ir/library/powerpoint/E DU04110.pps

23. Questions? Brian McNurlen (217) 244-5724 mcnurlen@uiuc.edu Chris Migotsky (217) 333-3490 migotsky@uiuc.edu