1. Kresge Business Administration Library 1 EVALUATING VIDEO TUTORIALS: Measuring Excellence And Outcomes Sally Ziph Instruction Coordinator Kresge Business Administration Library Stephen M. Ross School of Business University of Michigan Elizabeth Beers Web Developer, MPublishing University of Michigan Library University of Michigan

2. Kresge Business Administration Library 2 OUR VIDEOS

3. Kresge Business Administration Library 3 Rationale New MBA Consortium students needed company and industry information to prepare for interviews with recruiters. Videos were intended to help complete a specific task: build lists in OneSource, find career information in Vault, etc.

4. Kresge Business Administration Library 4 The Process Librarians created scripts based on frequently asked questions. Technologies included Audacity for audio and Camtasia for screen capture. Videos were integrated into the library website and advertised through email.

5. Kresge Business Administration Library 5 A Challenge How do we evaluate the effectiveness of non- interactive task-based point-of-use video tutorials?

6. Kresge Business Administration Library 6 EVALUATING OUR VIDEOS

7. Kresge Business Administration Library 7 Best Practices Pre/Post tests A/B tests Focus groups and user interviews Surveys Statistics

8. Kresge Business Administration Library 8 Pre/Post Tests From the literature: Used to measure learning that takes place when students view tutorial Most suitable for assessing tutorials that seek to teach a specific skill Can be difficult to write questions that will give good comparison data

9. Kresge Business Administration Library 9 Pre/Post Tests For our purposes: Tests intended to help with a task, rather than teach a skill

10. Kresge Business Administration Library 10 A/B Tests From the literature: Used to compare outcomes from versions of the same tutorial: –In person vs. online –Different instructional techniques Also useful for gathering subjective feedback Difficult to tease out whether variance in outcomes is due to content or presentation

11. Kresge Business Administration Library 11 A/B Tests For our purposes: Abbreviated project timeline made comparison testing impossible. Useful for refining tutorials

12. Kresge Business Administration Library 12 Focus Groups and User Interviews From the literature: Easy ways to get feedback from a variety of viewpoints Can be conducted iteratively and at any stage of the development cycle Require an experienced facilitator

13. Kresge Business Administration Library 13 Focus Groups and User Interviews For our purposes: Abbreviated project timeline made iterative assessment impossible Useful for future assessment

14. Kresge Business Administration Library 14 Surveys From the literature: Easy method for data collection Easy to formulate questions – but can be difficult to formulate good questions Results often reflect subjective preferences

15. Kresge Business Administration Library 15 Surveys For our purposes: Quick method for getting feedback from the target user group Not the best data, but better than none.

16. Kresge Business Administration Library 16 Statistics From the literature: Raw numbers from Google Analytics or built in utilities measure use. Can be used to validate data from other methodologies Does not measure usefulness or quality of tutorial content

17. Kresge Business Administration Library 17 Statistics For our purposes: Data we were collecting anyway may be useful for future videos. Most popular: Career resources Company information Industry reports

18. Kresge Business Administration Library 18 OUR EVALUATION

19. Kresge Business Administration Library 19 Our Survey Managed using Qualtrics Distributed by targeted email to 45 students Questions included: –Did you watch one or more of the Kresge database videos? –How useful were the instructional videos for the following situations? –What changes would you make to the Kresge videos to make them more useful?

20. Kresge Business Administration Library 20 Results 13 out of 45 students responded (28%) 72% of respondents did not use the videos: –Videos were perceived as inconvenient or irrelevant –Students wanted other topics

21. Kresge Business Administration Library 21 Conclusion We need to clarify: –What the Kresge Library can do to help Consortium students –Why these videos are worth their time We need to send reminders at strategic times throughout the program.

22. Kresge Business Administration Library 22 NEXT TIME?

23. Kresge Business Administration Library 23 In the Future Continue gathering statistics Use focus groups to refine tutorials and identify new topics Consider testing methodologies to measure learning and usefulness

24. Kresge Business Administration Library 24 QUESTIONS? 24