Helping students find what they need: an intuitive ‘sliding scale’ design for formative CMA question sequences Ingrid Nix Lecturer in Teaching and Learning (FHSC) CETL COLMSCT Teaching Fellow (co-author of the LINA project with Ali Wyllie) EATING talk, 15 November 2007

2 Outline Overview of LINA (Learning with Interactive Assessment) aims Some existing CMA examples – the context What is the LINA ‘sliding scale continuum’? Selection of tentative early data analysis Next steps

3 LINA aims – areas of interest How to make Qs more motivating (e.g. using audio, animations)? How to optimise student choice (e.g. more effective use of study time)? How to encourage reflection and evidence-based practice? How to share system information on choices made & provide a tool to reflect? e.g. How do students want Feedback displayed?

4 Today’s questions Can using a continuum bring any benefits to students? Can you see it being of relevance in your own subject area?

5 A typical journey from formative to summative using OpenMark A K216 student is being assessed on ICT skills They try out formative CMAs When ready, they access summative CMAs Examples of standard OpenMark interface follow:

6

7

8

9

10

11

12

13

14 Current characteristics of ad hoc Qs. Surprise Challenge Unknown elements Inability to gauge whether need to access

15 Inserting sequences Can create links between questions to build narrative and aid students –Create impact and memorability –Scenario on which to hang learning experiences –Build pathway supporting learning acquisition

16 Example sequence: Summative K113 CMA

17

18

19 Standard OpenMark menu of options If we’re going to use sequences, can we improve students’ knowledge or expectation of what lies behind a question?

20 What is the sliding scale continuum? Description to students: Scenario-based, linked questions From left to right moves from ‘easier’ to ‘more difficult’ Can travel along it in any direction, revisit, skip Qs. Preparation for summative test

21 PPP continuum approach: from theory to applying a skill (EFL language teaching)

22 ‘Sliding scale’ continuum

23 Mapping PPP against other teaching approaches e.g. Bloom’s revised taxonomy P PresentationP PracticeP Production Present scenario + check general comprehension (Passive) Detailed comprehension Practice (Active) Further practice/transferable situations Getting ready for real world (Generative application) Anderson and Krathwohl (2001)

24 Interface compromise – v.1

25 Interface compromise – v.2

26 Demo of LINA project (from K113 course website)

27 Scenario introduced with choice of media Audio/ animation and transcripts Ability to select based on preference of learning approach

28 Confidence indicator tool Set tool before submit answer Self-assess their confidence that the answer will be correct Marks vary according to degree of confidence vs whether correct or not

29 Learning log for reflections At any point can open learning log to refer to Prompted at end of each question, after feedback, to enter current reflections

30 Preparing for summative At end of formative sequence, question types encourage transferring knowledge into other contexts

31 Preparing for summative At end of formative sequence, question types encourage transferring knowledge into other contexts

32 Example of a Learning log

33 Online feedback: additional information At end of summative sequence, invite Feedback on project areas of interest e.g. use of media, continuum, confidence indicator tool, etc.

34 Feedback form responses recorded in Learning Log (for our analysis later)

35 The process of gathering data LINA website made available online to K216 (and later K113) currently using standard OpenMark iCMAs Optional invitation to ‘have a go’ and feedback online (5 responses) If interested, invited to come & be videoed (IET userlab) (2 students respond from those above, 1 later cancelled) To increase data collection: 2 colleagues (course managers) invited to IET userlab (no previous experience of iCMAs) Total respondents providing data: 7 people

36 Feedback: quantitative & qualitative 1. Learning Log: While working online the system records student pathway and actions; presents these in a log (for students to refer to (and for us to refer to afterwards) Learning Log text input: students can type in additional text reflections during activity (saved in the log) 2. Feedback form: After each person completes a topic online we ask their views in an attached online Feedback form (one per topic) Responses are based on 7 people who completed one or both topics (1-2 repetitions of a topic each = 1 to 4 feedback opportunities)

37 Feedback: quantitative & qualitative 3. IET Userlab: Collected data from 3 recorded sessions Learning Log/report tracking pathway, selections made; + text input for reflections Video of screen actions, + audio recording of commentary speaking aloud their actions; + additional reflections Feedback forms completed after each topic Audio interviews on completion of the 2 topics covering issues emerging from observations

38 Quantitative: feedback form analysis Data collected is from: – some people who returned 1 Feedback form – or some people who returned 4 Feedback forms, (sometimes identical responses, sometimes changing their responses) For the following analysis the last response to each question was taken as reflecting their current view The following results therefore show a current snapshot, not the change in perception of respondents over time as they revisited the system

39 Learning log: pathways through continuum Qs\ Tester ID W3CSRAD L1  L2    L3 22  2  L4  2    L5   L6   T1 22  33 T2  22 T3   T4    

40 Use of Continuum

41 Use of Confidence Indicator tool

42 Features affecting motivation

43 Use of the Learning log

44 Various quotes [4.27] For me that’s just a good way of checking [SB:transc] [7.54]Now [Recalls Qs from previous visit.] If I remember right, this one, when I did the original test I got wrong. [11.45]Instead of ‘You want to insert..’ she says ‘I want to insert an extra row..’ reads second sentence as is on screen including ‘do you need to place your cursor’. [17.55] Right, now I’ve only noted 7 days across the top and 3 events down the side. So making my column up I’m just going to look at the audio transcript because I’m more of a visual person rather than doing that in my head. So I can check it here.

45 Various quotes [7.26]So I’ll do L1 to get an idea what level everything’s at. [17.10]Ok I’m confident that I’ll be able to do 6 correctly now. …..Because I think I know sort of how much difficult it’s getting all the time. [9.12]Because I never remember things I’ve heard I’m going to use the transcript as a crib sheet. [15.10] I’m going to number 4 next because I still don’t feel confident that I know how the level of difficulty is increasing. What kind of scale it’s increasing at. [15.53] Depending on how difficult 4 is, I might just go for 6. Or 5. [21.00]I used the transcript to check answers, therefore high confidence.

46 Identifying issues from transcripts -+ Usability: audio button too small+Learning strategy: scoping degree of difficulty of qs +learning: using techniques to support reflection, to increase confidence [opening transcripts] + learning: places self within the scenario (“..I want”) +learning: has scoped degree of difficulty based on confidence in getting all correct so far

47 Next steps Analysis of userlab video data (6 hours work to transcribe per topic) Transcribe and analyse audio interviews Explore other areas within project: –Use of media –Learning log –Confidence indicator tool –Feedback ? Do we need to extend our data collection

48 References Anderson, L. & Krathwohl, D. (2001) A Taxonomy of Learning, Teaching and Assessing: A revision of Bloom’s Taxonomy of Educational Objectives. New York, Addison Wesley Longman

49 Try-out LINA Please complete both topics + the Feedback form at the end of each topic Insert your name & faculty (to indicate you are not a student tester) Many thanks, Ingrid and Ali