1. Using Evaluation Tools Effectively
……………… Thomas C. Reeves, Ph.D., Professor Emeritus, Dept. of Educational Psychology and Instructional Technology College of Education, The University of Georgia

2. USING EVALUATION TOOLS EFFECTIVELY
Thomas C. Reeves
The University of Georgia
Thursday, April 14, 2011
1:55 p.m. - 2:35 p.m.

3. HIGHER EDUCATION INNOVATION
Online / Blended Courses
Social Media Apps
Serious Games
Authentic Tasks
Service Learning
Podcasts
E-Textbooks
Online Assessment

4. EVALUATING INNOVATION
What are student and faculty reaction to the innovation?
What evidence supports student engagement?
What evidence supports learning and transfer?
Is it worth the time and costs?

5. EVALUATION PLANNING
Establish the goals of the evaluation - What you want to learn?
Determine your sample - Who will you ask?
Choose methodology - How will you ask?
Create your instrument - What will you ask?
Pre-test the instrument
- Are you getting what you need?

6. Questionnaires Interviews Observations DATA COLLECTION Paper Online
Individual
Focus Group
Observations
Classroom

7. TYPES OF QUESTIONNAIRES
Paper distributed in class
Mail
Online, e.g., SurveyMonkey
Twitter poll
Facebook

8. Face-to-face Telephone or Skype Teleconference Synchronous Chat
TYPES OF INTERVIEWS
Face-to-face
Telephone or Skype
Teleconference
Synchronous Chat
Asynchronous forum
Focus group
Twitter poll
Facebook

9. Onlooker Overt Fully-explained Once or limited Narrow focus
TYPES OF OBSERVATIONS
Onlooker
Overt
Fully-explained
Once or limited
Narrow focus
Participant
Covert
Unexplained
On-going
Broad focus

10. TRIANGULATE TO TELL THE FULL STORYQ I
Full
Story O

11. FACTORS IN SELECTION
Speed
Cost
Sensitivity
Literacy
Security

12. QUESTIONNAIRES PROS CONS Less expensive Distribution challenges
Familiarity
Pictures and graphics
Answer at their leisure
Less intrusive
Distribution challenges
Response rates low
Literacy
Misinterpretations
Can’t be explained

13. INTERVIEWS PROS CONS Immediate feedback More costly Personalization
Probing questions can be asked
Higher response rate are possible
More costly
Can be threatening
Undesirable bias can be introduced
Subjective interpretation

14. OBSERVATIONS PROS CONS Can see interactions You can’t see it all
Can be unobtrusive
Video observations can be reviewed
Intended and unintended
You can’t see it all
May change behavior
Subjective interpretation
Costly

15. CASE STUDY – ONLINE COURSE WITH AUTHENTIC TASKS DESIGN

16. CASE STUDY – ONLINE COURSE WITH AUTHENTIC TASKS DESIGN

17. CASE STUDY – ONLINE COURSE WITH AUTHENTIC TASKS DESIGN
Objectives
Content
Instructional Design
Instructor Roles
Learner Roles
Technology Roles
Assessment
Competent Evaluator
E-Learning Evaluation
Authentic Tasks
Formative Coach/Mentor
Engaged Group Work
Collaboration
Conduct Evaluation

18. CASE STUDY – ONLINE COURSE WITH AUTHENTIC TASKS DESIGNQ I
Full
Story O
Data Collection
Online Questionnaires
Student interviews in-person & Skype
Observations by graduate research assistants

19. REPORTING RESULTS Podcasts increased learning 233%.
Discussion: blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah, blah
The data: blah, blah, blah, blah, blah, blah
Bottom Line: We should …..

20. QUESTIONS

21. Professor Emeritus Tom Reeves
The University of Georgia
Learning, Design, and Technology
604 Aderhold Hall
Athens, GA