1. LEARNING ANALYTICS & THE LEARNING SCIENCES Zachary Stein, Ed.M, Ed.D (c) Lectica ®, Inc. Harvard University

2. OVERVIEW  Learning about learning  What is learning?  What are the sciences of learning?  Analytics: definitions and approaches  Narrowing in on learning analytics and its relatives  Lessons from history: testing and technology in US education  Putting the learning in learning analytics  Lectical ™ Assessments and the future of educational assessment © 2012 Lectica, Inc. all rights reserved

3. THE SCIENCES OF LEARNING

4. WHAT IS LEARNING?  There are over a dozen definitions currently in use by researchers and educators  From learning as “changing connections between neurons” to learning as “targeted behavior change” to learning as “the active construction of knowledge that is transferable, applicable, and reflectively held”  There is no one “right” definition, but we should set our terms so we know we are talking about the same thing © 2012 Lectica, Inc. all rights reserved

5. WHAT IS LEARNING?  Individual researchers and innovators should define what they mean  A look at nearly a dozen papers on learning analytics revealed no attempts at defining learning  Yet taking pedagogical or institutional actions based on learning analytics data assumes a theory of learning, which is often implicit and inarticulate  How do you know what to do to promote learning if you don’t know how learning works?  We should look to the learning sciences for help… © 2012 Lectica, Inc. all rights reserved

6. WHAT ARE THE SCIENCES OF LEARNING?  They go way back and they are diverse and complex  Philosophy, biology, psychology, neuroscience, etc.  They involves a wide variety of models, methods, and motives  The emerging interdisciplinary field of Mind, Brain, and Education (MBE) exemplifies the new science of learning  Here I will look at one of the central orienting generalization in this field, stemming from the cognitive developmental tradition © 2012 Lectica, Inc. all rights reserved

7. LEARNING AS A VIRTUOUS CYCLE  An idea that goes back to the first biologically oriented psychologist  A generalization covering learning processes at multiple levels, from neural-networks, to cognitive development, organizational and group learning (dynamic steering), and scientific investigation.  A process that explains the stage-like development of individual learners through learning sequences of increasing complexity © 2012 Lectica, Inc. all rights reserved

8. 8 THE CIRCULAR REACTION: LEARNING AT ITS CORE  Baldwin, Piaget, and a baby in a highchair  think you got it...  try it...  get feedback....  try again (hopefully at a higher level)....  Virtuous cycles always include tests © 2012 Lectica, Inc. all rights reserved

9. VIRTUOUS CYCLES OF LEARNING  Can occur at multiple levels in educational institutions:  Students  Teachers  Administrators  Researchers  Consider the example of a classroom  A process we refer to as instructional dynamic steering™  Note the role of embedded assessment… © 2012 Lectica, Inc. all rights reserved

11. LEARNING ANALYTICS REDUX  Let’s think through learning analytics with learning defined as a multi-level virtuous cycle… © 2012 Lectica, Inc. all rights reserved

12. ANALYTICS: DEFINITIONS AND APPROACHES

13. DATA-DRIVEN DECISION MAKING  In a wide variety of contexts modern organizations have become “centers of calculation and measurement”  Scientific management  Dynamic steering  In educational contexts this has mostly meant the application of business models and an increase in testing  Learning analytics steps in as information technology opens new opportunities for data generation and analysis (e.g., so- called “Big Data”) © 2012 Lectica, Inc. all rights reserved

14. LESSONS FROM THE HISTORY OF TESTING  Preferences for automation and economies of scale  Anti-theoretical orientation leaves us satisfied with weak, under-theorized measures  Measurement structures the system  Preferences for objectivity increasingly marginalize the judgment of those on the ground  The informational needs of institutional decision- makers override the learning needs of the students © 2012 Lectica, Inc. all rights reserved

15. THE VARIETIES OF LEARNING ANALYTICS  Learner-centric analytics  Learning-centric analytics  Classroom/cohort analytics  Program evaluation analytics  Institutional analytics  These can work at cross-purposes  As in the history of testing  These can also work in tandem  As is the case with Lectical Assessments © 2012 Lectica, Inc. all rights reserved

16. PUTTING THE LEARNING IN LEARNING ANALYTICS

17. CALL FOR RESEARCH BASED ASSESSMENTS  National Research Council Committee on the Foundations of Assessment (1999; 2001)  Assessment design requires research  Tests should be built around a theory of learning, not just psychometrics and expert opinion  Goals and standards should be grounded in evidence about student capabilities  However, high-stakes tests continue to be built around psychometrics and expert opinion, with little input from the learning sciences © 2012 Lectica, Inc. all rights reserved

18. LECTICAL ASSESSMENTS R&D  Decide on key subject areas and topics in collaboration with educators and researchers  Conduct basic research in order to build empirically- grounded learning sequences that can be translated into low-inference scoring rubrics  Compile and “developmentally curate” related learning materials  The result: an integrated set of assessments and learning materials that support virtuous cycles of learning © 2012 Lectica, Inc. all rights reserved

20. MEASURING LEARNING  The Lectical Assessment System  Based on Fischer’s Skill Theory (1980; 2006)  A domain general, content independent, developmental assessment system (Dawson, 2004; 2011)  Excellent psychometric properties (Dawson-Tunik, Commons, Wilson, & Fischer, 2005)  The common core metric behind all Lectcial Assessments  And the tool used to build them © 2012 Lectica, Inc. all rights reserved © 2011 Developmental Testing Service, all rights reserved

21. A LOOK AT THE LDMA  One of 7 existing Lectical assessments for use in higher-education  Currently being used as part of large-scale leadership education program for a major Canadian city.  Like all Lectical assessments, the LDMA asks students to construct short essays, which are then coded using empirically grounded rubrics based on research into a set of relevant learning sequences.  Feedback fosters learning for students, teachers, decision-makers, and researchers © 2012 Lectica, Inc. all rights reserved

22. STUDENT LEARN  Lectical Assessments support student learning by…  requiring them to operate on concepts and apply skills  encouraging reflection on personal performance (see growth over time)  providing rich, targeted, real-time feedback based on test results, including suggestions for what comes next  delivering targeted (and vetted) learning activities © 2012 Lectica, Inc. all rights reserved

29. TEACHERS LEARN  Lectical Assessments support teacher learning by…  encouraging deep engagement with student performances (teachers read and score tests, not computers)  providing insight into student learning (through online student and teacher reports)  offering specific information, in real time, about what individual students have learned and what they are most likely to benefit from learning next  providing learning resources that show how to target instruction to individual students (learning resources) © 2012 Lectica, Inc. all rights reserved

31. DECISION-MAKERS LEARN  Lectical assessments support decision-maker learning by providing…  complex multi-focal learning profiles, at the cohort, classroom, and organization (in real time)  multiple, repeated measures that reveal trends and trajectories (growth models) © 2012 Lectica, Inc. all rights reserved

33. RESEARCHERS LEARN  DiscoTest support educational research by...  contributing to basic research about learning during their construction  generating large relational databases of detailed individual learning profiles  trends and trajectories contribute to our understanding of learning in general  generating overviews of learning at the level of the cohort, classroom, and organization  having data standardized to a common core metric

34. © 2012 Lectica, Inc. all rights reserved

36. SUMMING UP  The emerging field of learning analytics can benefit from taking the learning sciences seriously  Otherwise, we run the risk of repeating the mistakes made during the first wave of data-driven decision making in schools  Recent advances in educational assessments technology can be used to catalyze multi-level learning in education intuitions  Leveraging true measures of learning to help students, teachers, administrators and researchers

37. QUESTIONS?

38. FOR MORE INFORMATION  www.lectica.org www.lectica.org  zak@devtestservice.org