Using Evaluation Tools Effectively ……………… Thomas C. Reeves, Ph.D., Professor Emeritus, Dept. of Educational Psychology and Instructional Technology College of Education, The University of Georgia

USING EVALUATION TOOLS EFFECTIVELY Thomas C. Reeves The University of Georgia Thursday, April 14, 2011 1:55 p.m. - 2:35 p.m.

HIGHER EDUCATION INNOVATION Online / Blended Courses Social Media Apps Serious Games Authentic Tasks Service Learning Podcasts E-Textbooks Online Assessment

EVALUATING INNOVATION What are student and faculty reaction to the innovation? What evidence supports student engagement? What evidence supports learning and transfer? Is it worth the time and costs?

EVALUATION PLANNING Establish the goals of the evaluation - What you want to learn? Determine your sample - Who will you ask? Choose methodology - How will you ask? Create your instrument - What will you ask? Pre-test the instrument - Are you getting what you need?

Questionnaires Interviews Observations DATA COLLECTION Paper Online Individual Focus Group Observations Classroom

TYPES OF QUESTIONNAIRES Paper distributed in class Mail E-mail Online, e.g., SurveyMonkey Twitter poll Facebook

Face-to-face Telephone or Skype Teleconference Synchronous Chat TYPES OF INTERVIEWS Face-to-face Telephone or Skype Teleconference Synchronous Chat Asynchronous forum Focus group Twitter poll Facebook

Onlooker Overt Fully-explained Once or limited Narrow focus TYPES OF OBSERVATIONS Onlooker Overt Fully-explained Once or limited Narrow focus Participant Covert Unexplained On-going Broad focus

TRIANGULATE TO TELL THE FULL STORY Q I Full Story O

FACTORS IN SELECTION Speed Cost Sensitivity Literacy Security

QUESTIONNAIRES PROS CONS Less expensive Distribution challenges Familiarity Pictures and graphics Answer at their leisure Less intrusive Distribution challenges Response rates low Literacy Misinterpretations Can’t be explained

INTERVIEWS PROS CONS Immediate feedback More costly Personalization Probing questions can be asked Higher response rate are possible More costly Can be threatening Undesirable bias can be introduced Subjective interpretation

OBSERVATIONS PROS CONS Can see interactions You can’t see it all Can be unobtrusive Video observations can be reviewed Intended and unintended You can’t see it all May change behavior Subjective interpretation Costly

CASE STUDY – ONLINE COURSE WITH AUTHENTIC TASKS DESIGN

CASE STUDY – ONLINE COURSE WITH AUTHENTIC TASKS DESIGN

CASE STUDY – ONLINE COURSE WITH AUTHENTIC TASKS DESIGN Objectives Content Instructional Design Instructor Roles Learner Roles Technology Roles Assessment Competent Evaluator E-Learning Evaluation Authentic Tasks Formative Coach/Mentor Engaged Group Work Collaboration Conduct Evaluation

CASE STUDY – ONLINE COURSE WITH AUTHENTIC TASKS DESIGN Q I Full Story O Data Collection Online Questionnaires Student interviews in-person & Skype Observations by graduate research assistants

REPORTING RESULTS Podcasts increased learning 233%. Discussion: blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah The data: blah, blah, blah, blah, blah, blah Bottom Line: We should …..

QUESTIONS

Professor Emeritus Tom Reeves The University of Georgia Learning, Design, and Technology 604 Aderhold Hall Athens, GA 30602-7144 treeves@uga.edu http://it.coe.uga.edu/~treeves