1. LEMMA: Learning Environment in Multilevel Modelling and Applications Fiona Steele London School of Economics & Political Science Director NCRM LEMMA node, University of Bristol http://www.bristol.ac.uk/cmm/learning/online-course/index.html

2. LEMMA Training Capacity building in the analysis of data with complex structure –Ultimate goal is to move learners to “take-off”, i.e. conducting and publishing multilevel analyses Different modes of delivery –Face-to-face workshops –Web-based materials in a virtual learning environment

3. LEMMA e-learning: First steps Originally planned to host searchable repository of materials BUT … –Written by different authors using different styles and notation –Not clear how to tag and organise materials Concluded that we needed new materials organised as a course –Coherent structure – provide ladder for novices –Ensure consistent terminology and notation → LEMMA online course launched April 2008

4. Lessons from Face-to-Face Training Learners need motivation and time –Best motivated are those with multilevel data and research questions Learners often do not possess prerequisites for multilevel modelling In practical sessions, learners tend to focus on mechanics of using software rather than interpretation

5. LEMMA online course: principles Accessible to anyone with basic statistics training Modules to have two integrated components: concepts and practical Facility for learner’s self-evaluation (quizzes) Collect data to evaluate materials Design materials so they can be modified by other trainers

6. A typical module Concepts –Approx 40-50 pages, split into lessons –Illustrative examples based on a single dataset –Draw links between fitted model equations, graphs of predictions and verbal interpretation –No reference to software Practical –MLwiN, Stata and R (SPSS coming soon) –Detailed analysis of one dataset (different to that used in Concepts) Quizzes

7. 7

8. Current modules (1) 1.Using quantitative data in research 2.Introduction to quantitative data analysis 3.Multiple regression 4.Multilevel structures and classifications 5.Introduction to multilevel modelling 6.Regression models for binary responses 7.Multilevel models for binary responses 8.Multilevel modelling in practice Note: Modules 1-3 and 6 not on multilevel modelling

9. Current modules (2) 9.Single-level and multilevel models for ordinal responses 10.Single-level and multilevel models for nominal responses 11.Three-level multilevel models 12.Cross-classified multilevel models 13.Multiple membership multilevel models 14.Missing data 15.(coming soon) Multilevel models for repeated measures

10. Choice of e-learning application In 2005 choice was between Moodle and Blackboard Moodle chosen because: –Free and open-source –Excellent on-line community support –More easily customised than Blackboard

11. Growth in number of registered users Launched April 2008 Now ~14k users

12. Intended use of materials* n% Learning Self learning1196787.7 Recommended reading for a course7265.3 Preparation for LEMMA workshop3632.7 Teaching5954.3 Total13651 *All figures correct as of 16/6/14

13. Users by sector n% Student808558.6 Academic380327.6 Non-academic190713.8 Total13795

14. Users by country Countryn n UK3861Spain316 US2701Italy310 Germany604India287 Netherlands514Belgium284 Australia504Switzerland242 Canada474France216 72% users from outside the UK

15. Users by primary discipline n% Medical sciences196214.2 Statistics, methods and computing188613.7 Psychology177812.9 Economics12439.0 Sociology11848.6 Education9326.8 Human geography and demography7845.7 Management and business studies6835.0 Political science and management studies6634.8 Biological sciences5043.7 Other217615.8

16. User familiarity with statistical methods

17. User familiarity with mathematical statistics

18. Evaluation of LEMMA online course Survey of UK users conducted by NCRM in 2013 264 responses received Full report at http://eprints.ncrm.ac.uk/3261/1/MOLEY_NCRM_ Impact_2011-2013.pdf http://eprints.ncrm.ac.uk/3261/1/MOLEY_NCRM_ Impact_2011-2013.pdf

19. Summary statistics on usage Median time spent on course is 10 hours (SD = 14.8) ~ 65% had at least partially completed modules on Introduction to Quantitative Research and Multiple Regression (Modules 1-3) –So important to provide introductory materials as preparation for multilevel modelling modules Source: ESRC NCRM survey

20. Reasons for registration

21. Combining with face-to-face training “The course enabled me to get better value from face-to-face training, by providing in-depth preparatory reading. Together with the face-to-face training, I improved my knowledge of MLM to a sufficient degree to engage in detailed discussions about analytical methods with academic collaborators.” “…I’d like to take further classes in this, but I need the online course first so I’m prepared for the further courses.” Respondents to NCRM survey. Online course modules are recommended reading for LEMMA face-to-face courses.

22. Impact of LEMMA online course Source: ESRC NCRM survey

23. Resource considerations Writing online materials is resource intensive –Materials unsupported so need to be clear and thorough –Ensure consistency of style and notation across modules and cross-reference previous modules –Writing quizzes and feedback especially time-consuming Wide range of technical expertise required –Installation and maintenance of Moodle –Programming skills for customising Moodle –Web design –Database management –Ongoing technical support and trouble-shooting

24. Future developments New materials in pipeline –SPSS practicals for selected modules –Module on longitudinal data analysis Convert materials to interactive e-books –New ESRC project led by William Browne

25. Thanks Funding for 3 phases of the NCRM LEMMA node, 2005- 2015 Technical team –Sacha Brostoff –Hilary Browne –Christopher Charlton –Hugh Garner Authors