2. Online Survey and Assessment Techniques Dr. Curtis J. Bonk Associate Professor, Indiana University President, CourseShare.com http://php.indiana.edu/~cjbonk, cjbonk@indiana.edu

3. Pause… What tools do you use or know about?

4. Collecting Evaluation Data & Online Evaluation Tools

5. Types of Data Collection Direct Observation in Work Setting –By supervisor, co-workers, subordinates, clients Collect Data By Surveys, Interviews, Focus Groups –Supervisors, Co-workers, Subordinates, Clients Self-Report by learners or teams Email and Chat Online Tests E-Portfolios or Learning Logs

6. I. Online Testing Tools (see: http://www.indiana.edu/~best/)

8. Choice: Select companies that specialize in online assessment.

9. Or: Use what the courseware package gives ya…

15. Some things in life are free…!

17. Test Selection Criteria (Hezel, 1999) Easy to Configure Items and Test Handle Symbols Scheduling of Feedback (immediate?) Provides Clear Input of Exam Dates Easy to Pick Items for Randomizing Randomize Answers Within a Question Weighting of Answer Options

18. More Test Selection Criteria Recording of Multiple Submissions Timed Tests Comprehensive Statistics Summarize in Portfolio and/or Gradebook Confirmation of Test Submission

19. More Test Selection Criteria (Perry & Colon, 2001) Supports multiple items types—multiple choice, true-false, essay, keyword Can easily modify or delete items Incorporate graphic or audio elements? Control over number of times students can submit an activity or test Provides feedback for each response

20. Flexible scoring—score first, last, or average submission Flexible reporting—by individual or by item and cross tabulations. Outputs data for further analysis Provides item analysis statistics (e.g., Test Item Frequency Distributions). More Test Selection Criteria (Perry & Colon, 2001) Web Resource: http://www.indiana.edu/~best/

21. II. Electronic Gradebooks (Vockell & Fiore, 1993) Calculate scores, store info Weight scores Flag students with certain characteristics Print reports by individual or group Provide prompt feedback But inflexible, impersonal, & can be incorrect

23. III. Online Survey Tools for Assessment

24. Perhaps you have heard of my survey techniques?

26. Survey Tool Features Maintain email lists and email invitations Conduct polls Adaptive branching and cross tabulations Modifiable templates Maintain library of past surveys Publish reports Technical support, chat advice Different types of accounts—hosted, corporate, professional, etc.

27. Still More Online Survey Features Online help, FAQs, demos, tutorials, white papers, glossaries, site maps, tips, overviews Website evaluation tools or options Supports different types of items (Likert, multiple choice, forced ranking, paired comparisons, etc.) Customizable audience lists Launch and close surveys

28. More Online Survey Features Online viewing of assessment results and question analysis (including various graphical displays) Easy to author surveys Instant feedback to participants Custom reporting, editable images and background, personalized greetings Storage of results in Access or SQL

29. Still More Features Ability to start and stop a survey Reports on subgroups Randomize q’s to reduce bias Easy email reminders Personalized banner or logo Expanded question options Ability to copy surveys

31. Web-Based Survey Advantages Faster collection of data Standardized collection format Computer controlled branching and skip sections Graphics can reduce fatigue Easy to answer clicking Wider distribution of respondents

32. Web-Based Survey Problems: Why Lower Response Rates? Low response rate Lack of time Unclear instructions Too lengthy Too many steps Can’t find URL Perceived as aggressive

33. Web-Based Survey Solutions: Some Tips… Send second request Make URL link prominent Offer incentives near top of request Shorten survey, make attractive, easy to read Credible sponsorship—e.g., university Disclose purpose, use, and privacy E-mail cover letters Prenotify of intent to survey

34. Tips on Authentification Check e-mail access against list Use password access Provide keycode, PIN, or ID # (Futuristic Other: Palm Print, fingerprint, voice recognition, iris scanning, facial scanning, handwriting recognition, picture ID)

35. Sample Survey Tools Zoomerang (http://www.zoomerang.com) IOTA Solutions (http://www.iotasolutions.com) QuestionMark (http://www.questionmark.com/home.ht ml) Survey Solutions from Perseus (http://www.perseusdevelopment.com/fromsurv. htm) Infopoll (http://www.infopoll.com)

36. Sample Survey Tools Active Feedback (http://www.activefeedback.com/af) Majon International (http://www.coolsurveys.com) EZSurvey from Raosoft (http://www.raosoft.com/) SurveyShare (http://SurveyShare.com; from Courseshare.com)

60. Zoomerang Steps 1.Register: Sign Up for Basic Membership (SKIP FOR NOW!) 2.Click on new survey. 3.Click on survey template (e.g., “Education” and Professor) 4. Delete, modify, or add 1-2 items. 5.Submit survey.

61. Zoomerang Steps 6.Modify Web greeting and submit. 7.Pick option #1 for submission. 8.Type Survey invitation & email message 9.Enter email addresses for potential survey respondents 10.Decide on how share results

62. Zoomerang Steps 11.Launch survey 12.Click on “My Surveys” and browse 13.Click on “Email List” and browse 14.Click on “Account Info” 15.Other explorations…

80. SurveyShare Steps 1.Register: Sign Up (perhaps take Quick Tour, Flash version) 2.Login 3.Click on Create Survey 4.Check out existing templates 5.Select one of the templates 6.Add one item, delete one item, modify one item

81. SurveyShare Steps 7.Edit Survey Introduction and Title 8.Edit Closing and Submit 9.Select Survey Invitation Via Email 10.Type emails to complete the survey 11.Click add (to move emails to right) 12.Give email subject and message

82. SurveyShare Steps 13.Enter email subject line and text 14.Preview survey 15.Activate survey 16.Go to MySurveys 17.Click on Survey Info 18.Go to mailing lists

83. SurveyShare Steps 19.Add a person 20.Add list 21.Check out resources (books) 22.Check out help, quick tour, and sitemap 23.Perhaps create a survey from scratch 24.Perhaps create a survey from “My Previous Surveys”

84. Any Reactions to Online Surveys???

85. Fitting Surveys into Evaluations Plans

86. Bonk’s Evaluation Plan…

87. What to Evaluate? 1.Learner—attitudes, learning, use, performance. 2.Instructor—popularity, course enrollments. 3.Training—internal and external components. 4.Task--relevance, interactivity, collaborative. 5.Tool--usable, learner-centered, friendly, supportive. 6.Course—interactivity, participation, completion. 7.Program—growth, long-range plans. 8.Organization—cost-benefit, policies, vision.

88. The 4 Levels Reaction Learning Behavior Results

90. Ziegler, April 2002, e- Learning “…the key is not to measure every possible angle, but rather to focus on metrics that are pragmatic and relevant to both human and business performance at the same time.”

91. Some Final Advice… Or Maybe Some Questions???