1. Effective Methods for Educational Research The value of prototyping, observing, engaging users Diana Laurillard

2. Methods for educational software design An iterative design-test-analyse-redesign approach The focus is achievement of learning outcomes Align outcomes – activities – assessment Discover requirements – design – design tests Prototype Observe Test Engage Evaluate

3. Prototyping Sketch the design idea Design user tasks – Use materials to represent choices and actions – Allow users to construct the choices and actions Data collection – 6 target users (min) – User constructions (materials) – Users’ commentary (audio, observation notes) – Video (usually unnecessary)

4. A paper prototype Proposed screen design Proposed actions

5. A paper prototype User defined actions

6. A paper prototype

7. Observing Individual – Observation notes – ‘Stimulated recall’ (Bloom1953) – video or playback to act as a memory aide – Exercise: ‘what were you thinking at this point?’ – Audio + notes Pairs – listen to them talking, audio + notes Recording of screen activities Video

8. Data capture and analysis tools To capture screen activities To capture eye movements, logged interactions, and for eye- tracking data analysis To coordinate and analyse the multiple streams of data

9. User engagement Workshops – with frequent discussions Focus-group - question then facilitated discussion Survey – Closed questions + open comment Construct elicitation interviews to discover users’ own categories of description

10. Learning analytics Measures that correlate with final grade Total discussion messages posted (+ve) Total number of online sessions (+ve) Total time online (+ve) Time spent on assignments (+ve) Time spent on assessments (-ve) (MacFadyen and Dawson, 2010)

11. LMS tracking those at risk Ave grade = 72% Ave grade = 63% Ave grade = 57% Ave grade = 44% (MacFadyen and Dawson, 2010) Predicting those at risk Week of course Average hours online per week

12. Learning analytics on outcomes Pedagogically-driven analytics Define the ecology of experiences and resources in which the software is embedded Design appropriate assessment of outcomes Use the software tasks to generate tests Formulate the pre-post tests that will capture the differences the software should make Interpret outcomes for re-design

13. Working groups 1. What kinds of problems do you see with methods of prototyping, observing and engaging? 2. What might be the advantages of using such methods? 3. How might an educational software developer can draw on evidence-based research?