1. talks! How to Evaluate Your Digital Literacy Program June 30, 2011 Audio is only available by calling this number: Conference Call: 866-740-1260; Access Code: 6339392 Sponsored by

2. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Using ReadyTalk Chat & raise hand All lines are muted If you lose your internet connection, reconnect using the link emailed to you. If you lose your phone connection, re-dial the phone number and re-join. ReadyTalk support: 800-843-9166

3. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 You are being recorded… This seminar will be available on the TechSoup website along with past webinar presentations: –www.techsoup.org/go/webinars You will receive a link to this presentation, material and links. After the webinar, you can ask follow-up questions in our Accessible Technology and Public Computing Community Forum: –http://bit.ly/tsg-pcc Twitter hashtags: #techsoup

4. talks! How to Evaluate Your Digital Literacy Program Presenters: Linda Hofschire & Erick Ruiz Sponsored by This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License

5. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Community Technology Network

6. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Today’s Speakers Linda Hofschire Colorado State Library Facilitator: Kami Griffiths, Community Technology Network Assisting with chat questions: Sarah Washburn, TechSoup Global Stephanie Gerding, TechSoup Global Susan Burkholder, Colorado State Library Erick Ruiz Instructional Design Consultant

7. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Agenda Overview of evaluation fundamentals Colorado State Library’s BTOP program Systems used to track impact Long term benefit of tracking data Tools used to track Q&A

8. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Poll Are you a BTOP grantee? –Yes –No

9. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 To measure or NOT to measure Before you begin Program objectives must be developed Stakeholders must agree Realities ROI is not for every organization or program Are you prepared to handle a negative ROI? ROI measurement can add to your program costs Phillips, P. (2002). The Bottomline on ROI. Atlanta, GA: CEP Press

10. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 What is ROI? ROI is a measure of accountability ROI is a mathematical explanation, comparing program costs to program benefits. –Benefit Cost Ration (BCR) BCR = Program Benefits/Program Costs –Return on Investment (ROI) ROI (%) = (Net Program Benefits/Program Costs) X 10

11. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 What is ROI? Tangible program benefits: expressed quantitatively –BCR and/or ROI Intangible program benefits: expressed qualitatively –Benefits that can not be quantified –Benefits that can not be isolated to your program

12. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 ROI Levels First... –Program objectives must be developed Then... –Level 1: Reaction & Satisfaction –Level 2: Learning –Level 3: Application –Level 4: Business Impact –Level 5: ROI

13. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Level 1: Reaction & Satisfaction Key Questions Was the program relevant? Was the program important? Did the program provide new information? Do participants intend to use what was learned? Would Participants recommend the program to others? Method Online Surveys Face-to-face interviews Focus Groups Random Sampling Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press

14. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Level 2: Learning Key Questions Did participants acquire the knowledge and skills presented in the program? Do participants know how to apply what they learned? Are participants confident to apply what they learned? Method Testing –Norm Referenced Testing –Criterion Referenced Testing Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press

15. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Level 3: Application Key Questions How effective are participants at applying what they learned? How frequently are participants applying what they learned? What support are participants receiving? Method Participant and “other” Online Surveys Face-to-face interviews Focus Groups Random Sampling Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press

16. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Level 4: Business Impact Key Questions To what extent did application of learning improve the measures the program was intended to improve? How do you know it was your program that produced these measures? Method Survey partners Survey participants –Online Surveys –Interviews –Focus Groups Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press

17. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Level 5: Return On Investment Key Questions Do the monetary benefits of the improvement in business impact measures out-weigh the cost of the program? Method Financial Measures Tangible Benefits Data Intangible Benefits Data ROI (%) = (Net Program Benefits/Program Costs) x 100 Phillips, P. & Phillips, J. (2005). Return On Investment Basics. Alexandria, VA: ASTD Press

18. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Colorado Computer Centers: Bridging Colorado’s Great Digital Divide

19. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 81 Public Computer Centers (PCCs)

20. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 PCCs in Action

21. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 450 desktops 700 laptops 70 ADA compliant workstations 10,000 trainings By the Numbers…

22. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Areas of Focus Basic computer skills Workforce/employment Health Resources for adult education ESL Partnerships 9 partners, including the Gates Foundation, College in Colorado, the Workforce Development Council, and the Colorado Department of Labor

23. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Three Components of Our BTOP Evaluation Outputs Class evaluations Immediate outcomes

24. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Planning the Evaluation

25. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Data Map Data element ReportCollection Point Quarterly Report Annual Report Grant Application Compliance OfficerWebsite Personnel $XX Broadband speed XXXX Type of classXXX Equipment $XX Scheduled one-to-one tutoring sessions X Hours open per 48-hr weekend XXXX

26. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Laying the Groundwork Pretest Establish baseline

27. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Outputs

28. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Class Evaluation

29. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Outcomes For classes: –Basic technology skills –Employment –Business –Health For open lab users: –General

30. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Class Outcomes—Employment Please check all that apply: After taking this class, I am better able to…  Create or work on a resume  Search for a job online  Learn about a potential employer  Learn skills for current or new job

31. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 General Outcomes Please check all that apply: During my recent visit to the computer center, I…  Did schoolwork  Looked for employment  Communicated with someone (ex., email, online chat, instant messaging, Facebook)

32. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Assessing Outcomes Over Time Select a sub-sample Pre-/post-test Case study Tell the story behind the numbers

33. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Benefits Beyond the Grant Period

34. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Benefits Beyond the Grant Period

35. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Benefits Beyond the Grant Period Use results for budgeting, program planning, advocacy, etc. Ongoing evaluation

36. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Tools/Resources Impact survey http://impactsurvey.org/ http://impactsurvey.org/ Colorado State Library’s BTOP site http://coloradovirtuallibrary.org/btop/ http://coloradovirtuallibrary.org/btop/ Computer reservations system http://code.google.com/p/powerline/ http://code.google.com/p/powerline/

37. talks! Questions? Submit your questions using the chat box.

38. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Continue the Discussion… Additional questions can be answered by posting in in our Community Forum: –http://bit.ly/tsg-pcc

39. How to Evaluate Your Digital Literacy Program Conference Call: 866-740-1260 Access Code: 6339392 Thank you to our Webinar Sponsor! ReadyTalk offers dedicated product demos for TechSoup organizations 4 times per week. For more information: pages.readytalk.com/techsoup.html

40. talks! Thank you! Please complete the post event survey! Kami Griffiths, kami@ctnbayarea.org