David Ackerman, Associate VP Crystal Butler, Research Associate

1 – MAKE IT COUNT 2 – MAIN METHOD 3 – GETTING RESULTS what did we find out? 4 – LESSONS LEARNED...and what does it mean? AGENDA SAKAI MARKET SHARE MAKE IT COUNT MAIN METHOD LESSONS LEARNED what determines market share? how do we get good data? GETTING RESULTS

1 – COUNT METHOD 2 – RESPONSE RATE one institution, one vote: enrollment rates are not reflected since enrollments vary, sample data can't be generalized commercial LMS providers won't share client lists; open source registries are poorly managed THE ISSUES MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT GETTING RESULTS 3 – ABSENT PROVIDER DATA SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED Counting installations rather than users could present a skewed perspective of the changing LMS landscape. COURSE MENU MAKE IT COUNT THE COUNT GETTING RESULTS SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED LMS market share as calculated by The CCP, based on a one system per institution measure for all higher ed respondents, including Community Colleges. COURSE MENU MAKE IT COUNT THE COUNT GETTING RESULTS SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED LMS market share as calculated by The CCP, based on a one system per institution measure for all higher ed respondents, with Community Colleges removed. COURSE MENU MAKE IT COUNT THE COUNT GETTING RESULTS SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED Four-year+ schools had an average response rate of 28.4%....making it impossible to generalize about student usage across colleges and universities with widely varying enrollment rates. COURSE MENU MAKE IT COUNT RESPONDENTS GETTING RESULTS SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED Commercial vendors do not share their complete customer lists. COURSE MENU MAKE IT COUNT LMS PROVIDERS GETTING RESULTS SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED Open source providers may try to track system installations... COURSE MENU MAKE IT COUNT LMS PROVIDERS GETTING RESULTS SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED...but have incomplete control over implementation and client downloads, so cannot provide accurate usage statistics. COURSE MENU MAKE IT COUNT LMS PROVIDERS SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED As the end users, learners should be tallied to determine market share. By completion of their fourth year in college, the ratio of students who will have used a school's primary LMS approaches 100%. COURSE MENU MAKE IT COUNT THE USERS GETTING RESULTS SAKAI MARKET SHARE

1 – IDENTIFY INSTITUTIONS 2 – DETERMINE A DATA GATHERING METHOD select schools and get enrollment numbers how to count 100% of the student population of interest? 3 – ESTABLISH A PROCESS the plan, pitfalls, and (striving for) perfection THE STEPS MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT MAIN METHOD GETTING RESULTS SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED Schools were selected using the U.S. Department of Education's Institute of Education Sciences database... COURSE MENU MAKE IT COUNT IPEDS GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED with the following criteria: 4-year+ not-for- profit degree granting based in the U.S. 1,000 or more students have a physical campus COURSE MENU MAKE IT COUNT IPEDS GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED We have a lot of work to do: how to determine the LMS in use by over 1,500 schools? COURSE MENU MAKE IT COUNT IPEDS RESULTS GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED Web search. With a large data set requiring repetitive search operations and intelligent evaluation of results, we had an ideal use case for Amazon's Mechanical Turk. COURSE MENU MAKE IT COUNT MECHANICAL TURK GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED MTurk works best when the instructions are standardized, so we had to establish a web search that would return optimal results. We tested the term 'learning management system' and related terms suggested by Google and other source s... COURSE MENU MAKE IT COUNT THE SEARCH GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED...but querying specific LMS names returned the best links. COURSE MENU MAKE IT COUNT THE SEARCH GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED COURSE MENU MAKE IT COUNT THE SEARCH GETTING RESULTS MAIN METHOD 1 – MTurk TESTING 2 – WORKER QUALIFICATION first run: 100 items, 2 Master Classification Workers each (200 HITs) second run: 50 items, 3 Master Workers each (150 HITs) third run: 400 items, 3 Master Workers each (1200 HITs) goal: create workforce with 95% accuracy or greater reality: standards had to be set at 85% accuracy to generate a large enough worker base SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED COURSE MENU MAKE IT COUNT GETTING RESULTS MAIN METHOD use 3 workers per item; verify LMS choice for all worker disagreements do 5% audit on all worker agreements compare final LMS list to provider information found on web 4 – DATA VERIFICATION 3 Custom Qualification Workers per item remaining institution list split in half and run sequentially in two sets 3 – DATA COLLECTION THE SEARCH SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED Mechanical Turk provides analytics for completion velocity, average hourly pay rate, and percent HITs submitted. COURSE MENU MAKE IT COUNT MTurk TESTING GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED Of the 49 workers who attempted our HITs, only 10 performed well enough to meet our qualification standards. COURSE MENU MAKE IT COUNT WORKER QUALIFICATION GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE

MAKE IT COUNT MAIN METHOD LESSONS LEARNED Audits consistently found a 95% accuracy rate for worker agreements. Workers:.95 x 3621 = 3440 In-house search: 1.0 x 1263 = 1263 Accuracy: 4703/4884 = 96.29% We are confident that our results are not off more than +/- 5% COURSE MENU MAKE IT COUNT DATA VERIFICATION GETTING RESULTS MAIN METHOD SAKAI MARKET SHARE

1 – OUR FINDINGS 2 – A COMPARISON LMS market share based on year headcount how do our results differ from the CCS count? 3 – THE SPREADSHEET our data by LMS THE RESULTS MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT MAIN METHOD GETTING RESULTS SAKAI MARKET SHARE

THE NUMBERS MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT MAIN METHOD GETTING RESULTS Based on a one-year enrollment of 13,723,024 students. SAKAI MARKET SHARE

THE COMPARISON MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT MAIN METHOD GETTING RESULTS Campus Computing Survey Count by Institution New York University Count by Enrollment SAKAI MARKET SHARE

ENROLLMENTS MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT MAIN METHOD GETTING RESULTS SAKAI MARKET SHARE

OUR DATA MAKE IT COUNT MAIN METHOD LESSONS LEARNED MAKE IT COUNT MAIN METHOD GETTING RESULTS SAKAI MARKET SHARE

1 – GROWTH OF CANVAS 2 – SAKAI 6.2% market share since founding in 2008, with many recent migrations 8.3% market share since deployment in 2005, with few recent migrations and many previous Sakai installations having been replaced by other systems 3 – OPEN SOURCE 26.9% of primary LMSs by enrollment are open source, less than the 33.6% reported by the CCS count; Blackboard is often the primary LMS at schools with smaller parallel or test installations of other systems CONCLUSIONS MAKE IT COUNT MAIN METHOD MAKE IT COUNT MAIN METHOD GETTING RESULTS LESSONS LEARNED SAKAI MARKET SHARE

1 – PUBLISH PAPER 2 – FIND HOME FOR SPREADSHEET OF DATA Include analysis as to statistical significance of comparison survey This will allow schools to correct any errors as well as report a change in LMS 3 – SHARE DATA FUTURE WORK MAKE IT COUNT MAIN METHOD MAKE IT COUNT MAIN METHOD GETTING RESULTS LESSONS LEARNED SAKAI MARKET SHARE

QUESTIONS MAKE IT COUNT MAIN METHOD MAKE IT COUNT MAIN METHOD GETTING RESULTS LESSONS LEARNED SAKAI MARKET SHARE