chai: : Computer human adapted interaction research group Interfaces for Learning Data Visualizations Judy Kay CHAI: Computer Human Adapted Interaction Research Group School of Information Technologies, University of Sydney President of the AIED Society ( ), Programme co-Chair ITS2010, General Chair AIED2011 Advisory Board User Modeling Programme co-Chair Pervasive 2012, Chair of the Joint Ubicomp and Pervasive Steering Committee

chai: : Computer human adapted interaction research group Learning analytics is making it possible to transform learner's and teacher's digital footprints and their digital artefacts into models of learners and learning processes. To make such information really valuable, we need interfaces that enable people to make effective use of that data. A current question for the LA community is: "how do we create such interfaces?" This workshop will present two classes of answers. The first is to take classic HCI user-centred design methods, adapting them to the particular needs of the various stakeholders and users for LA. The second is to identify a set of principles for creating a set of core tools that have the promise of being applicable to many learning contexts. To illustrate approaches to both of these, this workshop will present cases studies from diverse contexts at two major levels. The first operates at the class level, serving learners, their peers, their mentors and teachers to gain insights about individual, group and classroom learning. The second is at an institutional level for curriculum custodians and administrators. Participants will actively engage in each aspect, identifying links to core problems they wish to address.

chai: : Computer human adapted interaction research group About me Inventing future technology to tackle important problems, notably in learning Personalisation Personal data and its management Putting people in control Open Learner Models (OLMs) Metacognition and OLMs Interactive surfaces… walls, tables…

chai: : Computer human adapted interaction research group Learning analytics as a form of Learner/User Modelling with interfaces

chai: : Computer human adapted interaction research group How to create interfaces for LA? User-centred approaches – Stake-holders – Mental models – The problem? Core tools and principles Case studies – Institution – Class – Individual learner

chai: : Computer human adapted interaction research group Interfaces… visualisations

chai: : Computer human adapted interaction research group Why visualisations?

Fekete, J. D., Van Wijk, J. J., Stasko, J. T., & North, C. (2008). The value of information visualization. In Information Visualization (pp. 1-18). Springer Berlin Heidelberg.

“…easy and fast to see that there is no red circle, or to evaluate the relative quantity of red and blue circles. Color is one type of feature that can be processed preattentively, but only for some tasks and within some limits. [eg] if there were more than seven colors …, answering the question could not be done with preattentive processing and would require sequential scanning, a much longer process.

chai: : Computer human adapted interaction research group But how to create the right visualisations? Are there simple rules? Simple principles? Simple and constant solutions?

chai: : Computer human adapted interaction research group Principle: individual data takes on more meaning…. When comparisons are supported: Others Temporal Contextual

chai: : Computer human adapted interaction research group Patina: Dynamic Heatmaps for Visualizing Application Usage (CHI2013) Justin Matejka, Tovi Grossman, and George Fitzmaurice This user’s footprints Overall population footprints

chai: : Computer human adapted interaction research group Extended Case Study Concrete example of my work to underpin the activities

chai: : Computer human adapted interaction research group Defining features The problem: – Group work is hard but it is important – Group work in learning context has many problems that cause great anguish, inefficiency Target stakeholders: – Learner as individual – Team leaders (manager, tracker) – Facilitators (tutor, lecturer)

chai: : Computer human adapted interaction research group Trac: Tool supporting long term group work Used by team members, facilitators, teachers, some clients

chai: : Computer human adapted interaction research group TRAC open source tool for supporting software development projects Wiki page editor Ticket Manager SVN source repository Not a learning system but used in a learning context.

chai: : Computer human adapted interaction research group  Huge amounts of data about the group members and their interactions

chai: : Computer human adapted interaction research group Narcissus Upton, K., and J. Kay. (2009) Narcissus: interactive activity mirror for small groups. In UMAP09, User Modeling, Adaptation and Personalisation, Springer-Verlag, 54-65

chai: : Computer human adapted interaction research group 19 Integrated of mirror tool Narcissus tab

chai: : Computer human adapted interaction research group ITS2008 Lifelong learning, learner models and sugmented cognition Lifelong modelling – mirrors and mining

chai: : Computer human adapted interaction research group 21 Header – Group view Display for one user Time – activity on that day is shown for each user, on each medium

chai: : Computer human adapted interaction research group 22 Wiki contribitions svn contribitions ticket contribitions

chai: : Computer human adapted interaction research group 23 Click on cell … …to see details

chai: : Computer human adapted interaction research group 24 Explains scoring

chai: : Computer human adapted interaction research group 25 Individual summary Group average

26 Click on ticket activity for a day Associated details Click on ticket label

27 Details of that ticket

ITS2008 Lifelong learning, learner models and sugmented cognition Sequence mining Managers DevelopersLoafersOthers Group 1*1311 Group 2*1013 Group 3012**3 Group 4*1320 Group 53*103 Group 6*1131 Group 7*1024 Group 1 – 1 person had sequences characteristic of managers. * That person had the manager role Group 1 – 3 members had developer activity sequences Group 3 – dysfunctional and here we might see why Group 5 – another way to be dysfunctional

chai: : Computer human adapted interaction research group Activity 1 Your Stakeholders?

chai: : Computer human adapted interaction research group Activity 1 Stakeholders? – Learners – Parents, Mentors, Facilitators – Teachers – Supervisors – Institutions – Quality assessors – Researchers

chai: : Computer human adapted interaction research group Activity 2 Problems you would like to tackle?

chai: : Computer human adapted interaction research group Activity 2 Current problems we aim to tackle? – Teacher: Early identification of at-risk individuals – Learner: Decision support Am I doing well enough? Am I doing what is expected of me? – Institution: Effectiveness of teaching and learning?

chai: : Computer human adapted interaction research group Building from SMILI Bull, S., & Kay, J. (2007). Student Models that Invite the Learner In: The SMILI:() Open Learner Modelling Framework. International Journal of Artificial Intelligence in Education, 17(2),

chai: : Computer human adapted interaction research group What is an Open Learner Model? Any interface to data that a system keeps about the learner Came from AI + personalisation where learner model drives personalisation OLM has become a first-class citizen! Link to Learning Analytics….

chai: : Computer human adapted interaction research group SMILI questions How does the open learner model fit into the overall interaction? – What problem does it aim to address? WHAT is open? HOW is it presented? WHO controls access?

chai: : Computer human adapted interaction research group The purposes for opening the learner model are: Improving accuracy Promoting learner Helping learners to plan and/or monitor their Facilitating collaboration and/or competition Facilitating navigation of the learning system Assessment Complex of issues of managing personal data: – right of access to data about themselves – Right of control over their learner model – increasing trust

chai: : Computer human adapted interaction research group Scrutable user models and personalised systems Research systems only, so far But hints of their being ready to emerge in mainstream software

chai: : Computer human adapted interaction research group Interfaces to substantial learner models Core concepts in a whole semester long subject

chai: : Computer human adapted interaction research group HCI subject with online lectures Exploit data from: – logs of interaction with lecture “slides” – class assessments Lightweight ontology for tagging – automatic analysis of online dictionary – augmented with class-specific concepts (as class glossary) – enabling combination of multiple data sources about each concept – and inference up/down ontology

chai: : Computer human adapted interaction research group ITS2008 Lifelong learning, learner models and sugmented cognition SIV Lots of green means learner doing well Weak aspects visible as red Overview visualisation

chai: : Computer human adapted interaction research group ITS2008 Lifelong learning, learner models and sugmented cognition SIV Kay, J and A Lum. "Exploiting readily available web data for scrutable student models.” Proceedings of the conference on Artificial Intelligence in Education 2005.

Little detail

chai: : Computer human adapted interaction research group Mental models

chai: : Computer human adapted interaction research group Mental models A set of beliefs that the user holds

chai: : Computer human adapted interaction research group Mental models A set of beliefs that the user holds eg. A whale is a fish The subject requires rote learning I expect to perform at about the median in this class

chai: : Computer human adapted interaction research group Mental models come from: Formal education And so much else – Experience – Cultural expectations – Context – Emotional state – …. Determining what the user – Believes to be true – Trusts – Feels permitted to consider and do – Feeling of competence

chai: : Computer human adapted interaction research group Why do mental models matter for interface designers?

chai: : Computer human adapted interaction research group Why do mental models matter for interface designers? They define what a user can “see” and “hear” How they interpret information Clashes between user, programmer, expert MMs

chai: : Computer human adapted interaction research group Activity Mental models What are key elements for your LA needs?

chai: : Computer human adapted interaction research group Pervasive technologies Case study Lots of embedded interaction devices, ready for interaction Where things may be headed….

chai: : Computer human adapted interaction research group User models in real classrooms For orchestration For in-class monitoring to inform teacher actions For post-hoc reflection by the teacher

The collaborative task (concept mapping and problem solving) [6] Novak, J. and A. Cañas, The Theory Underlying Concept Maps and How to Construct and Use Them T.R.I.C , Editor. 2006, Florida Institute for Human and Machine Cognition. Concept mapping is: – A tool for externalising knowledge – Applied in different domains – Promotes meaningful learning – Has been used by organisations such as NASA, Navy, and universities around the world.

chai: : Computer human adapted interaction research group

chai: : Computer human adapted interaction research group

chai: : Computer human adapted interaction research group Architecture

chai: : Computer human adapted interaction research group The big picture CSCL Computer Supported Collaborative learning HCI Human Computer Interactions EDM Educational Data-Mining Interactive tabletops in the classroom Interactive Tabletops and Surfaces 2010, 2011 Work In Progress, CHI 2012 Int. Conf. in Learning Sciences, ICLS 2012 Intelligent Tutoring Systems, ITS 2012 Computer Supported Collaborative Learning CSCL 2011 Educational Data Mining 2011 Interactive Tabletops and Surfaces 2012 Workshop on Orchestration, ICLS 2012 orchestration

chai: : Computer human adapted interaction research group Architecture

chai: : Computer human adapted interaction research group Collaid Our gear Learner’s physical differentiation

chai: : Computer human adapted interaction research group Our enriched interactive tabletop Kinect sensor Multi-touch tabletop R. Martinez, A. Collins, J. Kay, and K. Yacef. Who did what? who said that? Collaid: an environment for capturing traces of collaborative learning at the tabletop. In ACM International Conference on Interactive Tabletops and Surfaces, ITS 2011, pages , Logs: Differentiated tabletop actions Snapshots of the artefact

chai: : Computer human adapted interaction research group Our gear 2 Cmate Concept Mapping at the Tabletop

chai: : Computer human adapted interaction research group Learning outcomes for activities Concentric layout Significant correlated with higher levels of equity of participation (>0.4). Concentric Oriented towards a Learner “next time I would ask students to use a circular layout” Teacher:

chai: : Computer human adapted interaction research group From Design to Enactment and Reflection

chai: : Computer human adapted interaction research group Classroom activity design

chai: : Computer human adapted interaction research group Adherence to the class script (14 tutorials) Implications This was the most important activity from the learning perspective It forced the teacher to use more time than the 50 minutes There was not enough time for activity 2 as planned

chai: : Computer human adapted interaction research group Adherence to the class script “This is a very good reminder... maybe the structure for the next tutorials should be changed to give more time for Activity 2” Teacher: Effectiveness of the script Post-hoc teacher’s reflection Standardisation if multiple tutors. Activity re-design

chai: : Computer human adapted interaction research group Learning outcomes for activities * high achieving groups had more than 50% of these crucial propositions This analysis suggests the low achieving groups took longer to get started. “It would be more valuable to get this information per each group during the tutorials”. Teacher:

chai: : Computer human adapted interaction research group Can tabletops automatically alert teacher which group may need attention? Adding real-time learner model What’s the impact of showing information to the teacher?

chai: : Computer human adapted interaction research group The Orchestration Dashboard To help teachers to control multiple classroom tutorial sessions

chai: : Computer human adapted interaction research group Awareness and Control

chai: : Computer human adapted interaction research group The Awareness Dashboard To help teachers to determine whether groups or individual learners need attention Multiplatform!

chai: : Computer human adapted interaction research group Group 1 Group 2 Group 3 Class level Dashboard Martinez Maldonado, R., Kay, J., Yacef, K. and Schwendimann, B. An Interactive Teacher’s Dashboard for Monitoring Groups in a Multi-tabletop Learning Environment. Intelligent Tutoring Systems (2012),

chai: : Computer human adapted interaction research group Group 1 Group 2 Group 3 A Best-First tree model trained in another dataset classifies each block 30 seconds of activity Features: # of active participants in verbal discussions, amount of speech, number of touches symmetry of activity (Gini coefficient). Labels: Collaborative, Non-collaborative, or Average. The visualisation shows the accumulation of these. Martinez R, Wallace J, Kay J, Yacef K Modelling and identifying collaborative situations in a collocated multi-display groupware setting. In: AIED pp (2011) Class level: Indicator of detected collaboration.

chai: : Computer human adapted interaction research group Class level: Graph of interaction with others’ objects Group 1 Group 2 Group 3 The Circles indicate the number of touches The Lines represent the number of actions that each learner performed on others’ links and concepts

chai: : Computer human adapted interaction research group But how to create the right visualisations? Are there simple rules? Simple principles? Simple and constant solutions?

chai: : Computer human adapted interaction research group State of the art For learners….

Kahn Academy, what a student sees after the pre-test Model of learner Gamification element

chai: : Computer human adapted interaction research group State of the art Skill meters Game elements Good match to mental models

Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, 46(5),

chai: : Computer human adapted interaction research group State of the art Teachers

chai: : Computer human adapted interaction research group Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012, May). The student activity meter for awareness and self-reflection. In CHI'12 Extended Abstracts on Human Factors in Computing Systems (pp ). ACM. Data about many students in an online learning environment. Current focus is red student

chai: : Computer human adapted interaction research group Acknowledgements

chai: : Computer human adapted interaction research group Interactive surfaces Software infrastructure user control, scrutability Interfaces to user model Acknowledgements Data mining

Teacher assessment of usefulness (20 participants, most Computer Science)

“Visualization is much more effective at showing the differences between these datasets than statistics. Although the datasets are synthetic, Anscombe’s Quartet demonstrates that looking at the shape of the data is sometimes better than relying on statistical characterizations alone.

“Spence and Garrison …describe a simple plot called the Hertzsprung Russell … [shows] the temperature of stars on the X axis and their magnitude on the Y axis. … It turns out that no automatic analysis method has been able to find the same summarization, [as graphs at right] due to the noise and artifacts on the data such as the vertical bands.

chai: : Computer human adapted interaction research group The eye... the window of the soul, is the principal means by which the central sense can most completely and abundantly appreciate the infinite works of nature. Leonardo da Vinci ( )