1. Qualtrics Survey Research Suite: A Case Study of Innovating Business Processes and Systems at UCSF Orlando Leon, Applications Leader - Education, ITS

3. Overview Introduction to Qualtrics Statistics about Qualtrics at UCSF Use Cases Future Roadmap Resources Q&A

4. Introduction: Cloud-based Surveys “Easy enough for an intern, sophisticated enough for a PhD.”

5. Introduction: Key Features Able to collect ePHI SSO integration with MyAccess Mobile versions of surveys “out of the box” Integration with Salesforce objects Supports 48 languages ADA compliance-checking tool Panels Triggers Robust user/group/organization/library features Programmatic interface to administration tools Training and Support

6. Introduction: Who May Use It Staff Faculty Students/Learners Affiliates Others

7. Introduction: Scope of License The term of the license for the Qualtrics Software is one year, beginning January 16, 2012 and includes an unlimited number of surveys. The Qualtrics license may be used by academic areas within University of California San Francisco for projects that are academic in nature. Academic projects may serve research (instructional or academic) and administrative purposes. The license may not be used for purely corporate or "for- profit" purposes. Such corporate projects would include purely clinical and patient related surveys that are not related to academic research. Affiliated hospital and clinics may purchase Qualtrics access according to Qualtrics corporate pricing model. A 10% discount will be applied to any license purchased for corporate purposes. 7

8. Statistics about Qualtrics at UCSF 45+ Divisions 1,643 Users 7,596 Surveys 331,338 Auditable Responses 69,957 Responses on currently most active survey * Campus License since January 2012 – Officially Announced to UCSF Summer 2012

9. Use Cases Education –Student Learner Survey, Evaluation, Feedback, Registration, etc Research –Radiology Orthopedics Study, mOst Survey Tool –ATHENA, Clinical Trial Qualification Survey, Research Questionnaires, etc Administrative –Polls, Voting, Usability Satisfaction/Feedback Surveys, Interview Candidate Feedback, etc Clinical –Nursing PI Surveys, Patient Satisfaction, etc

10. Innovative Use Cases (1) Shared Asset Libraries Among Departments Customizable Reports with Live Updates [link]link Accreditation Review Objectives Collection of Data (ie pictures, homework, quizzes, research materials) Student Elections [link]link Quizzes/Scoring 360 Evaluations + Follow-up Evaluations Online Training [link]link Access Requests [link]link Event Registration

11. Innovative Use Cases (2) Online Form Integration (ie Drupal) Dynamic Classroom or Meeting Polling with Live Results with Mobile Devices (ie Clickr) [link]link Service Checklist (ie onboarding, change control, surgery, etc) [link] [link2]linklink2 Shopping List with Pictures [link]link Service Tracking (ie quick questionnaire for front- line staff) [link]link Decommissioning of FileMaker Pro Web Forms Application Forms [link]link Interview Evaluations [link]link Reservation Requests [link]link

12. Innovative Use Cases (3) Service Requests (ie. computer/supply purchase requests, etc) [link]link Branded Customer Service Feedback forms [link]link Integration within Web Applications using Custom REST API Calls (ie SLR) Collaboration without Sharing Passwords Collaboration Outside of UCSF Panels and SSO-based Contextual Information [link]link Quizzes and Learning Evaluation Annual Evaluations (ToC) ID Badge Scanning [link]link

13. Lessons Learned Limitations Growing Pains New Features and Enhancements –Some free, some not

14. Future Roadmap Funding Offline Response Gathering 360 Reviews Evaluations Crowdsourcing of Survey Innovations Deeper Integrations with Salesforce

15. Resources 15 Sharecase Booth A2 Recorded Training Sessions UCSF Informational Website –http://tiny.ucsf.edu/surveysoftwarehttp://tiny.ucsf.edu/surveysoftware Qualtrics Website –http://www.qualtrics.comhttp://www.qualtrics.com Qualtrics User Conference - February 2014 –http://www.qualtrics.com/summithttp://www.qualtrics.com/summit Other Survey Tools at UCSF –REDCap, Salesforce Qualtrics Chatter Group

16. Questions? 16

18. Demo https://ucsf.qualtrics.com http://tiny.ucsf.edu/qualtrics_kitchensink http://tiny.ucsf.edu/qualtrics_sharecase https://ucsf.us.qualtrics.com/SE/?SID=SV_3C0A0QVPbEsHOv2 https://ucsf.us.qualtrics.com/SE/?SID=SV_50fnPOSxU2srWAI https://ucsf.us.qualtrics.com/SE/?SID=SV_4ZcwwpjdoQMbfW4 https://ucsf.us.qualtrics.com/SE/?SID=SV_6VxQn3fGl6x6UMk https://ucsf.us.qualtrics.com/SE/?SID=SV_brUenOV1gE4wXyc https://ucsf.us.qualtrics.com/SE/?SID=SV_bJm24deQNGh1V08 https://ucsf.us.qualtrics.com/SE/?SID=SV_5o8SkNlPBmB7VJi 18

19. Business Case for Enterprise Survey Tool Should we purchase and support Qualtrics as an enterprise solution for the Campus? Should UCSF decide on a single solution for all survey needs? Should UCSF rationalize availability and support of existing survey solutions currently within the Campus?

20. Business Case for Enterprise Survey Tool: Considerations In conversations with stakeholders (likely survey users—SON, SOM MedEd, Nursing PI, SOM TAC), people asked us to consider: –Product Features –External/Vendor Support –Internal Product Ownership –Integration with Existing Systems –Total Cost of Ownership –Expertise in Survey Design vs Knowledge of the Technology

21. Introduction: Cost 21 It is FREE (funded by SAA and ITS)

22. Analysis: Cost Projection 2011-2012 – est $48,000 –$20,000 - Annual license cost, based on “Carnegie Classification” –$20,000 - 0.25 FTE @ $80k/year for brand administration, organization configuration and custom template creation –$5,000 – 0.05 FTE @ $100k/year for MyAccess integration –$6,000 – 0.10 FTE @ $60k/year for marketing and communications 2012-2013+ est $24,000* –$20,000 - Annual license cost –$4,000 - 0.05 FTE @ $80k/year for brand administration * The assumption is that each functional unit using Qualtrics will administer their own users and organization configuration, while the overall brand administrator administers the overall Campus/global configuration and setup of new functional units

23. Analysis: Implementation Details BAA reviewed and signed License negotiated, reviewed, and signed –License terms –Features Advisory and Oversight Committee –SON, SOM/UME/GME, SOD, SOP, Graduate Division, Academic Senate, Center for Health Professions, CTSI, Library, Memory and Aging, SAA, ITS Configuration and Deployment –MyAccess/SSO, Wiki/FAQ/Listserv/Chatter, Branding, Marketing/Communications, Administration 23

24. Analysis: Implementation Details Review and Consultation with University Relations and UCSF Legal Division and general use policies Marketing and Communication to UCSF community Group Training Sessions On-site Visit and Training 24

25. Introduction: Who May Use It Staff Faculty Students/Learners Affiliates …

26. Resources 26 Sharecase Booth #2 Qualtrics UCSF Site Visit –Wednesday, October 17, 2012 (9AM-12PM) – Parnassus HSW301 – http://QualtricsAtUcsf2012.eventbrite.com Introduction to Qualtrics Show Me the Cool Stuff How Do I… UCSF Informational Website –http://tiny.ucsf.edu/surveysoftware

27. Sample Surveys 27

28. Sample Surveys 28

29. Sample Surveys 29

30. Demo https://ucsf.qualtrics.com http://tiny.ucsf.edu/qualtrics_kitchensink http://tiny.ucsf.edu/qualtrics_sharecase https://ucsf.us.qualtrics.com/SE/?SID=SV_3C0A0QVPbEsHOv2 https://ucsf.us.qualtrics.com/SE/?SID=SV_50fnPOSxU2srWAI https://ucsf.us.qualtrics.com/SE/?SID=SV_4ZcwwpjdoQMbfW4 https://ucsf.us.qualtrics.com/SE/?SID=SV_6VxQn3fGl6x6UMk https://ucsf.us.qualtrics.com/SE/?SID=SV_brUenOV1gE4wXyc https://ucsf.us.qualtrics.com/SE/?SID=SV_bJm24deQNGh1V08 https://ucsf.us.qualtrics.com/SE/?SID=SV_5o8SkNlPBmB7VJi 30