2016 Faculty Technology Survey Stephen Burd (burd@unm.edu) Insights on UNM Learn from the 2016 Faculty Technology Survey    Stephen Burd (burd@unm.edu) Academic Technology Liaison Further information available at: http://averia.unm.edu Last revised: 2/4/2019 10:47 PM

Overview UNM Learn is a system that many faculty love to hate: Is that a problem? Why is it a problem? How can UNM fix the problem? Gathering faculty and student perspectives: Faculty technology survey (completed in April) Student technology survey (completed but results not yet analyzed)

LMS Faculty Survey Questions LMS defined as: A learning management system (LMS) is a web application that provides an instructor with a way to create and deliver educational content/experiences, support and monitor student participation, post and receive student assignments. I use an LMS for class sizes of _____ students Agree or disagree - The LMS is critical to my teaching is very useful as a tool to enhance my teaching is very useful as a tool to enhance student learning creates efficiencies for my instruction Years of experience using LMS technology? Do you currently use UNM Learn?

Survey Questions - Continued For UNM Learn Non-Users What factors influence your decision not to use UNM Learn? For UNM Learn Users Describe how you use an LMS to support your instructional activities Indicate your satisfaction with the following aspects of UNM Learn Do you have experience with an LMS that you prefer over UNM Learn?

Survey Questions - Continued For UNM Learn Users who’ve used and prefer another LMS What is your preferred LMS? Why is do you prefer that LMS? If my preferred LMS was available for use at UNM, I would invest time in moving my content to that LMS from UNM Learn Provide additional comments comparing your experience with the other LMS to UNM Learn

Survey Stats Distributed to 1828 full-time faculty Main campus but not north campus Branches 387 responses (21%) – 328 complete – typical of past surveys Individual URLs make it possible to slice/dice by Banner info such as campus, school, department, rank, and years of service

Respondent Characteristics 43% claim 5 or more years of LMS experience Respondents are probably better informed about and more experienced with the LMS than the faculty as a whole 77% currently use UNM Learn Large but possibly biased sample from which to gather responses to the detailed UNM Learn questions No way to know how (un)representative of the faculty as a whole are those respondents 16% have experience with an LMS that they prefer to UNM Learn Large enough sample to gather some useful information about comparisons between UNM Learn and other LMSs Too small for high confidence in the accuracy of those comparisons nor whether they would reflect the attitudes of the entire faculty should UNM switch to a different LMS

What UNM Learn Features are Used? Frequently-used: Distributing documents and learning materials (95%) Gradebook (74%) Online assignment/paper submission (63%) Quizzes/exams (52%) Class-related communication between the instructor and students (50%) Seldom-used: Graded discussions (22%) Group collaboration (20%) Analytics (17%) Student-to-student messaging (14%) Web conferencing (8%)

Class Characteristics No clear pattern based on class size or delivery method/modality

Overall Satisfaction Overall satisfaction – current users only Satisfied or very satisfied (59%) Neutral (21%) Dissatisfied or very dissatisfied (19%) If users of other LMSs are presumed to be non-users of and dissatisfied with UNM Learn them satisfaction level is 49%

Reasons For Not Using UNM Learn Canned responses (select all that apply) Ease of use for faculty (18.6%) Ease of use for students (10%) Usefulness (lack thereof) of LMS tools (10%) Ease of maintaining personal websites (9%) Availability of training (9%) Flexibility of the system (9%) Open-ended responses were largely consistent with the canned responses Some responses praised the utility and ease-of-use of alternatives such as: Dropbox Google Classroom Traditional web sites

Faculty Experience With Other LMSs 49 responses – top alternative was Canvas (16) – all others in single digits Respondents were asked why they preferred another LMS to UNM Learn. Ease-of-use (lack thereof) was the most frequently cited reason Other responses included LMS tool usefulness, flexibility, and reliability/stability. Open-ended responses gave many anecdotes describing UNM Learn’s poor ease-of-use and other shortcomings and praising the relative strengths of a chosen alternative 80% said they would invest the time to transfer content from UNM Learn to their preferred LMS

Survey Take-Aways If you’re looking for a mandate for change it’s not in the survey results If you’re looking for a vote of confidence in UNM Learn its not there either What is there: UNM Learn is underutilized Problem areas (inter-related) Ease-of-use Complexity Training

UNM Learn Under-Utilization 74% of survey respondents use UNM Learn but that probably over-represents use among the entire faculty – we don’t know by how much Users are using a subset of the features Greatest use is for the “convenience” features Underuse of features with the greatest promise for improved learning outcomes Non-users have voted with their feet for other LMSs and combinations of simpler tools Number and distribution of chosen alternatives provides little clarity about a “better alternative” to UNM Learn

Costs of Under-Utilization To students: A labyrinth of alternate tools and alternate ways of using them Negative learning outcomes? (not a proven result) To UNM Inability to tightly and uniformly integrate other systems with the LMS (e.g., LoboAchieve, UNM-wide analytics) Loss of learning outcomes that might or could be achieved through better LMS use (e.g., group collaboration, programmed or self-paced learning, within-course analytics) More complex support for faculty and students

Ease-Of-Use, Training, and Support A Gordian Knot? Poor ease-of-use combined with high complexity creates a greater and more complex need for training and support Users, especially faculty, resist using tools that require up-front training Avoiding up-front training creates more failures when faculty do try to use the LMS – those stories feed continued underuse and resistance to use How can this cycle be broken? Replacing UNM Learn with another LMS or a tool set that’s less complex and easier to use? Something else?

Where To From Here? Possibilities for discussion A drastically revised or completely new approach to training and motivating LMS use? Providing and supporting a simpler tool set in addition to the LMS? Controlled experiments with other LMSs or tool sets? Start an RFP process for the next generation of UNM Learn? All/most of the above? Others?